For engineering and procurement teams / Evaluation guide
Evaluating AI-to-SQL software with source access
Use a small, repeatable trial to decide whether a natural-language database tool fits your team.The decision
Can your team trust the path from question to result?
A business user asks for a number. An AI system produces a plausible query. The important question is whether the result answers the intended business question using data that user is allowed to access. A successful demo is the beginning of that investigation.
Source access gives your engineers a way to inspect schema discovery, model requests, query validation, execution, and error handling. It also gives them the option to adapt those parts within the license. The value depends on the actual workload and the work required to integrate the system.
A concrete starting point
What DataChat currently shows
DataChat's public evidence describes a demonstrated Claude-to-SQL Server path. Its code includes schema-aware query generation, query validation, streaming execution, cancellation, timeouts, history, and audit-related instrumentation. These are useful mechanisms for a buyer to inspect and exercise.
The current application uses an external model provider in that demonstrated path. That is not evidence of an offline or air-gapped system. Different model providers, database adapters, credentials, schemas, and deployment environments must be tested separately. Confirm the capabilities of the exact member release you intend to evaluate.
An example trial
Start with questions whose answers you already know
Create a small synthetic order database with customers, orders, refunds, and dates. Write expected answers before involving the model. Include a straightforward total, a join, a date boundary, an ambiguous request, and a question the available schema cannot answer.
For example, 'revenue last month' needs a definition: gross or net of refunds, order date or settlement date, and which timezone? The trial should reveal whether that ambiguity is surfaced. Compare the returned values with independently written reference queries. Valid SQL alone is not a correct business answer.
- Record the original question, schema version, generated query, returned values, and reference answer.
- Repeat the questions to expose variation rather than keeping only the best output.
- Keep some representative questions out of prompt tuning and use them for the final review.
- Report correct answers and failures separately from latency and provider cost.
Boundaries and failures
Test the controls outside the prompt
Give the evaluation account only the database permissions needed for the trial. Test a request to modify data, a restricted table, an expensive query, an unavailable provider, cancellation, and a timeout. Confirm what the database actually executed and what the user saw.
Query filters and model instructions are not substitutes for database authorization. Review what schema or data is sent to the model provider, what is recorded in logs, and whether canceled work actually stops. These questions are especially relevant when a company or public-sector team has restricted data handling requirements.
Your next step
Evaluate one workflow before expanding
A useful evaluation ends with a clear account of the workload that passed, the conditions that failed, and the integration work still required. That evidence helps an engineering lead decide whether to continue, adapt the implementation, or choose another approach.
If your team is considering DataChat, share the database type, deployment setting, and a non-sensitive example of the question you want to support. We can discuss relevant source access and the scope of an evaluation.