How to check AI-generated SQL (and Python) before you trust the number

Treat AI-written SQL like a colleague's first draft. Read it before you run it, then check the result against a number you already know. In practice that comes down to five checks: the right tables, the right grain after every join, date boundaries, how NULLs are treated, and whether the business definition matches the one your stakeholders use. Most wrong answers fail at least one of these.

Why AI-generated SQL fails quietly

A syntax error is easy, because the query doesn't run. The dangerous case is a query that runs, returns a plausible number and is wrong. Nothing warns you. The number goes into a deck, someone asks "how did you get that?", and the real answer is a join you never read.

Language models usually write well-formed SQL. They slip on context they can't see: which table your team trusts, what "active" means at your company, and how your data behaves at the edges. That is what the checks below are for.

Check 1: Is it using the right tables and columns?

Read the FROM and JOIN clauses first.

Is it the table your team treats as the source of truth, or a copy? Look for names like _old, _bkp, _tmp and _v2, and for staging schemas.
Does each column mean what its name suggests? amount might be gross or net, and created_at might be the order time or the load time.
If two tables could answer the question, ask which one finance or your BI layer uses, and make the query use the same one.

Check 2: Is the grain still right after every join?

Join fan-out is the most common silent error. Joining orders to order items, or customers to addresses, multiplies rows, so a SUM over the result counts the same money more than once.

After each join, compare:

SELECT COUNT(*)                   AS row_count,
       COUNT(DISTINCT o.order_id) AS distinct_orders
FROM orders o
JOIN order_items i ON i.order_id = o.order_id;

If row_count is larger than distinct_orders and you are summing an order-level column, aggregate the child table first and then join.

Check 3: Are the date boundaries right?

On a timestamp column, BETWEEN '2026-09-01' AND '2026-09-30' stops at midnight at the start of 30 September and drops the rest of that day. Use >= '2026-09-01' AND < '2026-10-01' instead.
Check the time zone. If timestamps are stored in UTC and your business day runs on IST, anything after 6:30 PM UTC belongs to the next day in India.
"Last month" and "last 30 days" are different questions. Make sure the query answers the one that was asked.

Check 4: What happens to NULLs?

WHERE status <> 'cancelled' also drops every row where status is NULL. If those rows matter, add OR status IS NULL.
AVG(column) ignores NULLs, so the average is taken over fewer rows than you might think.
COUNT(column) skips NULLs and COUNT(*) doesn't. Check which one the question needs.

Check 5: Does the definition match the business?

"Active customer", "revenue" and "churned" each have several reasonable definitions. A query can be correct SQL and still answer a different question from the one finance asked.

Write the agreed definition down once, in a glossary, a dbt metric or a reviewed view, and check that the WHERE clause matches it. When three teams get three different numbers, this is usually why.

If the AI wrote Python instead of SQL

The same traps show up in pandas under different names:

merge can duplicate rows just as a join does. Check len(df) before and after, or pass validate="one_to_one" (or "many_to_one") so pandas raises an error when the keys aren't unique.
groupby drops rows whose key is NaN unless you pass dropna=False.
mean() skips NaN values, just as AVG skips NULLs.
Dates read from CSV or Excel can arrive as strings, or with the day and month swapped. Parse them explicitly (pd.to_datetime(..., dayfirst=True) for Indian date formats) and check the minimum and maximum.
A filter applied after aggregation answers a different question from the same filter applied before it.

The rule is the same as for SQL: read the code before you trust the chart.

The known-answer test

Before you trust a new answer, ask one question whose answer you already know. That could be last month's revenue as reported by finance, or the row count of a table you loaded yourself. If that number doesn't match, stop and find out why before you ask anything new. It takes two minutes, and it catches wrong tables, wrong definitions and broken joins in one go.

A 2-minute checklist

[ ] Tables: the trusted source, not a copy or a staging table
[ ] Grain: COUNT(*) vs COUNT(DISTINCT key) after every join
[ ] Dates: half-open ranges (>= start AND < end), right time zone
[ ] NULLs: filters and averages behave the way you expect
[ ] Definition: matches the agreed business definition
[ ] Known answer: one question you can check against a number you trust

How Riyu helps

Riyu, our AI data analyst, shows the SQL or Python it ran next to its answers, so you can run these checks instead of taking the number on trust. Teams can also write their definitions once in a business glossary, and Riyu uses them when it answers. Try it on your own data.

FAQ

Is ChatGPT's SQL accurate? It is usually valid SQL, but valid isn't the same as correct. The usual failures are wrong tables, join fan-out, date boundaries, NULL handling and definitions it has no way of knowing. Read the query and run a known-answer test before you trust the result.

Can I let an AI tool query my production database? Give it a read-only database user limited to the schemas it needs, and set row limits or timeouts. A read replica is safer still.

How do I test text-to-SQL on my own schema? Write 10 to 20 questions whose answers you already know, covering joins, date ranges and your key definitions. Run them after every change and compare the answers with the expected values.