Back to projects
Public demo using fictional English data without internal tables, SQL, or records

Internal · In development

NL2SQL Quality

Quality improvement for an internal service that turns natural-language questions into SQL for security and asset data.

An internal service that turns everyday questions into SQL for a business PostgreSQL database. Starting with a single product domain, OMNIGuard, and later expanding to another product domain, I recovered answer criteria from screens, logs, and SQL across roughly 600 tables, then designed question-SQL worked examples, base instructions, and evaluation questions.

Period
2026.06 - 2026.08
Ownership
Business SQL analysis · prompt design · evaluation
Project
Internal LSware project
Repository
Private repository

Recovering the product's definition of correct before adding examples

ClaronX lets people ask questions in everyday language about security products and company assets. It should return the current installation state or latest history without requiring them to understand the database behind it.

My first assignment at LSware was to add worked examples for a single product domain, OMNIGuard: pairs of user questions and the SQL the AI should produce. After the initial results were recognized, I applied the same improvement process to another product domain. The target database covered roughly 600 tables with little documentation and no ready reference for which tables and conditions formed the correct answer.

At first, it was difficult to know where correctness should come from. A query could run and still return deleted data or an old record, so I had to recover the product's answer criteria before expanding the examples.

How an answer is checked

From an ambiguous question to a query that can be checked

  1. 01

    User question

    Different ways to ask the same thing

  2. 02

    Business rules

    Checked from logs, DDL, and existing SQL

  3. 03

    Instructions and examples

    System prompt and few-shot

  4. 04

    Generated SQL

    Checked for syntax and business meaning

  5. 05

    Execution and review

    Reviewed through PostgreSQL results

Two layers for correct SQL: shared rules and question-specific patterns

Generated SQL could execute successfully and still be wrong for the business if it selected deleted data or an old history row.

Across roughly 600 tables, rules such as excluding deleted rows, choosing the latest history, and staying read-only applied broadly, while joins and aggregations changed by question.

I moved reusable conditions into the system prompt and kept question-specific joins and aggregations in few-shot examples. AI generated candidate questions, while people verified expected SQL against screens, logs, and query results.

  • Recovered deletion, date-range, and latest-state rules from the running product.
  • Prepared roughly 176 question-SQL examples versus 30–40 in comparable internal work, plus an 80-question holdout set.
  • Compared a setup without shared rules against one using both rules and examples.

The setup without shared rules met roughly 30–40% of the criteria; using rules and examples together met the expected SQL criteria on all 80 defined questions.

The boundary between reusable rules and question-specific patterns mattered more than the raw number of examples.

Finding business rules beyond the DDL in runtime logs

Across roughly 600 tables, similarly named current and history tables and undeclared relationships made intent hard to infer. DDL alone could not establish the conditions used by the running product.

The required logs were in an internal Linux environment that my account could not initially access. Internal table names and SQL text also could not be exposed outside the company.

After explaining the investigation scope and receiving access from my supervisor, I mapped screen actions to the database logs they produced and cross-checked those queries against DDL and existing SQL.

  • Recorded which queries followed specific screen actions.
  • Extracted current/history, logical-deletion, and product-installation rules.
  • Organized the evidence into a data-model guide and handoff notes.

The recovered rules became evidence for the system prompt, few-shot examples, and evaluation criteria. The internal logs remain private; the portfolio describes only the method and rule categories.

When documentation is incomplete, I learned to gather evidence from running behavior before inventing an ideal interpretation of the system.

I prepared roughly 176 few-shot examples and a separate 80-question holdout set; all 80 met the criteria defined for their expected SQL.

Repository, SQL, prompts, and evaluation data are private because this was an internal company project.

When documentation is missing, I now collect evidence from logs and running behavior before inventing an ideal model. I also define error classes and evaluation criteria before adding examples.

80/80 is limited to a defined internal product scope and evaluation set. It is not general accuracy across other customer schemas or all natural-language questions.

  • Audit semantic similarity between examples and holdout questions, then repeat under stricter conditions
  • Test whether the workflow transfers to another domain
  • Measure execution success, semantic error classes, latency, and cost when available