Automated Testing for Fabric Data Agents: From Spreadsheet to Pipeline
The manual eval loop works until the agent matters. Then you need what every other production system already has: a test suite that runs itself. The SDK's evaluation module turns the spreadsheet into ground truth in a lakehouse, an LLM critic judging every answer, and accuracy history you can chart.
Two posts ago I told you to build a spreadsheet: fifteen questions, expected answers, re-run by hand after every change. I stand by it as the starting discipline. I also admit what it becomes once an agent matters: the test suite nobody runs. Manual evals get skipped on busy weeks, and busy weeks are exactly when someone changes an instruction and quietly breaks last month's fix.
The Fabric Data Agent SDK has an evaluation module that automates the whole loop: ground truth in a DataFrame or CSV, one function that runs every question against the agent, an LLM critic that judges each answer, and results landing in Delta tables you can query, chart, and alert on. This post is how I moved my eval sets from spreadsheet to pipeline, and what changed once agent accuracy became a number on a schedule instead of a vibe on a good day.
- 1 function call per eval run.
evaluate_data_agenttakes your question-and-answer set, runs every question through the agent, and judges the responses. The whole manual ritual, one call. - 3 verdicts per answer. True, false, or unclear. The unclear pile is its own signal: answers the critic could not confidently judge usually mean a vague question or a drifting expected answer.
- 2 delta tables per run. A summary table with accuracy, and a steps table with the detailed execution behind every answer. Your eval history becomes data you already know how to query.
Ground Truth Is the Spreadsheet, Graduated
The input format is exactly what the manual loop already produced: questions paired with verified answers. Two columns, question and expected_answer. Define it inline for a quick run, or keep it as a CSV in the lakehouse Files area, which is where mine live so the eval set is versioned, shared, and editable without touching the notebook:
Everything from the manual loop still applies to what goes in this file. Answers verified independently in SQL, never copied from the agent itself. A mix of easy, ambiguous, and should-refuse questions. And one new rule that only bites once you automate: pin questions to stable time periods. "Revenue last month" has a different correct answer every month, which means a scheduled eval fails eventually with nobody having broken anything. "Revenue for March 2026" stays true forever.
One Call Runs the Loop
The evaluate_data_agent function takes the ground truth and does what you used to do by hand: sends every question to the agent, collects the answers, judges each one against the expected answer, and writes the results out.
The parameter that changes your workflow is data_agent_stage. The agent has two lives: the draft configuration you are editing, and the published version users hit. Setting the stage to sandbox evaluates the draft, which means you can score a configuration change before publishing it. The workflow becomes: edit instructions, eval the sandbox, compare accuracy against the last production run, publish only if it improved. That is a deployment gate, the same shape as running tests before merging, and it is the single biggest upgrade over the manual loop.
Each run writes two Delta tables: the summary table with per-run accuracy, and a companion steps table with the detailed execution behind each answer. The returned evaluation_id tags every row, so runs stay separable.
Reading the Results
Two helper functions cover the two questions you actually ask after a run. "How did it do?" is get_evaluation_summary, which returns counts of true, false, and unclear verdicts plus overall accuracy. "What broke?" is get_evaluation_details, which by default returns only the failures:
The details DataFrame has the question, the expected answer, what the agent actually said, the verdict, and a thread_url linking to the full conversation thread for that question, visible to whoever ran the eval. That link is the automated equivalent of reading the generated query in the editor, and the habit transfers: the verdict tells you a question failed, the thread tells you why. Triaging failures still routes to the same fixes as always. Wrong routing or wrong table goes to instructions, wrong join or wrong filter goes to example queries.
The Critic Is a Model Too
The judging step deserves scrutiny, because it is an LLM comparing the agent's answer to your expected answer. That is what makes automation possible at all: "$4.21M", "4,210,000 dollars", and "Net revenue was $4.21 million" are the same answer in three formats, and string comparison would fail all of them. The default critic handles ordinary formatting variance. When your domain needs different judgment, you supply your own critic prompt:
The placeholders {query}, {expected_answer}, and {actual_answer} get substituted per question. Tightness cuts both ways: a lenient critic passes answers a human would reject, a strict one fails harmless rephrasings and buries you in false alarms. My calibration habit is to spot-check a handful of true verdicts, not just the failures, every few runs. A critic that never disagrees with the agent is not judging, it is nodding along, and you find that out by auditing agreements.
Put It on a Schedule, Chart the Trend
Everything above still describes a person running a notebook. The finish line is removing the person: schedule the eval notebook in Fabric, weekly or nightly depending on how often the agent and its underlying schema change. Because every run appends to the same Delta tables with its own evaluation_id, the summary table quietly becomes a time series of agent accuracy.
Which means the natural end state, in a stack that is all about reporting anyway: the agent gets its own report. Accuracy over time as a line chart, failures by question as a table, unclear verdicts flagged for review. Schema drift shows up as a cliff in the line, instruction regressions show up the week they happen, and the "who owns quality" question from the five tips post gets a concrete artifact: someone owns keeping that line flat or rising.
Worth keeping straight: this pipeline and the example query validator test different things. evaluate_few_shots checks that your teaching materials are clear, correct, and conflict-free. evaluate_data_agent checks that the agent actually answers real questions correctly end to end. Library quality and answer quality, both automated, and neither one substitutes for the other.
Failure Modes I Keep Seeing
Expected answers that drift
Ground truth written against "last month" or live tables fails as data moves, and the eval cries wolf until nobody reads it. Fix: pin questions to fixed periods, and when a data correction legitimately changes an answer, update the CSV in the same commit.
Trusting the critic blindly
The judge is a model with its own failure modes, and a too-lenient critic inflates accuracy right up until a user finds the wrong answer it waved through. Fix: audit a sample of passing verdicts periodically, and tighten the critic prompt when you catch it nodding.
Ignoring the unclear pile
Unclear verdicts get mentally rounded to "probably fine" and accumulate. Each one is a question whose answer could not be judged, which usually means a vague question or a malformed expected answer. Fix: treat a growing unclear count as a failing test.
Evaluating only production
If evals only run against the published agent, you learn about regressions after users can hit them. Fix: eval the sandbox stage before every publish and make "accuracy did not drop" the gate, exactly like tests before a merge.
Where This Fits in the Series
This post closes the loop that the instructions post opened. The spreadsheet taught the discipline: known questions, verified answers, re-run everything after every change. The SDK keeps the discipline and deletes the toil, and adds the two things a spreadsheet never gave you: a pre-publish gate and an accuracy history. Together with the example query work, the accuracy story is now fully testable, which is what lets you touch a production agent without holding your breath.
And that is the last prerequisite cleared. The agent is authenticated, instructed, taught by example, and continuously tested. Next post, it finally ships: embedding the agent as a chat surface inside your own application.
Further Reading
- Example Queries: Teaching Fabric Data Agents to Write Correct SQL: the previous post, including the example query validator this pipeline complements
- Evaluate your data agent: Microsoft Learn documentation for the evaluation module, including all function parameters
- Fabric Data Agent Python SDK: SDK overview, installation, and authentication for running evals outside a Fabric notebook