Back to posts
Part 5 of 5 · Building Accurate Fabric Data Agents
Nov 09, 2026•11 min read

Agents as Code: Testing Fabric Data Agents with the SDK Before They Ship

Once a data agent matters, every change to its instructions is a production change. Keep the agent in files, generate ground truth from SQL, evaluate the draft three times per question, and let a gate compare it with production before anything is published.

In July I wrote about graduating the evaluation spreadsheet into a scheduled pipeline. That answers is the agent still right? It does not answer the question that comes next, the one every team asks once an agent has real users: can I change this agent without breaking it?

The answer is the same one software teams reached decades ago. Keep the thing in files, review changes as diffs, test every change before it ships, and refuse to ship anything that makes a known-good case worse. This post shows how to do that for Fabric Data Agents with the SDK and the Fabric REST API, using the setup from the team of agents I built for my Day of Data session.

  • 4 files per agent. Description and tables, agent instructions, data source instructions, example queries. Everything the agent knows, in git.
  • 3 repeats per question. Agents are not deterministic. A question passes only when it passes every time.
  • 0 regressions allowed. The gate compares question by question with production. A better average does not excuse a broken answer.

The Pipeline at a Glance

Here is the whole loop before the detail: one change, five checks, and a gate that decides whether the change is published.

Changeinstructions.mdExample queriesrun + scoreUpdate draftupdateDefinitionEvaluate ×3sandbox stageGatevs productionOne pull request, five checkscommit6 / 6 run ✓draft ✓scoredpublish ✓blocked ✗2nd changeQuestionProductionDraft: repeat 1 · 2 · 3Net sales this year3/3✓✓✓✓✓✓Net sales by region3/3✓✓✓✗✗✗Accounts behind budget2/3✓✗✓✓✗✓Open backlog by status3/3✓✓✓✓✓✓Forecast vs budget3/3✓✓✓✓✓✓Out of scope: HR question3/3✓✓✓✓✓✓Regression: passed 3/3 in production, 0/3 in the draftNo question got worse (one was flaky before, too): publish
Agents as code

A change starts as a pull request

Someone edits the agent's instructions, adds an example query, or changes its tables. All of it lives in files, so the change is a diff anyone can review.

Check the example queries first

Every example query runs against the SQL endpoint and must return rows. Then the SDK's evaluate_few_shots scores each one and flags conflicts. A broken example quietly teaches the agent wrong SQL, so it fails the build here.

Update the draft, not production

The pipeline writes the new definition to the agent's draft through the Fabric REST API. Callers keep getting the published version while the draft is tested.

Ask every question three times

evaluate_data_agent runs the ground truth against the sandbox stage, three repeats per question. Agents are not deterministic, so one pass proves little.

Gate on regressions, not averages

Compare question by question with production. One question was flaky before and is still flaky, so nothing got worse: the pipeline publishes.

The second change is blocked

Another edit drops a question that passed 3 out of 3 in production to 0 out of 3. The average barely moves, but the gate catches it, and the run steps show the SQL that broke.

1. Put the Agent in Files

Everything that makes an agent answer correctly can live in a folder next to your other code:

# One folder per agent
agents/sales-analyst/
agent.json # name, publish description, selected tables
instructions.md # agent instructions
datasource-instructions.md # joins, units, dates
fewshots.json # example queries
agents/operations/ ...
agents/planning/ ...
verify_answers.sql # the ground truth, one query per question

A short script turns a folder into an agent. The Fabric REST API accepts a data agent definition: a set of base64 parts with paths such as Files/Config/draft/stage_config.json for the instructions and Files/Config/draft/<data source>/fewshots.json for the example queries. The script builds those parts from the files, then calls updateDefinition on the agent. Publishing is a separate call, staging/publish, with the publish description from agent.json.

# Two calls: change the draft, then (only if the gate passes) publish it
POST /v1/workspaces/{ws}/dataAgents/{id}/updateDefinition
{ "definition": { "parts": [ ...instructions, data source, example queries... ] } }
POST /v1/workspaces/{ws}/dataAgents/{id}/staging/publish
{ "publishedDescription": "Answers questions about net sales and the budget..." }

That split is what makes testing possible. updateDefinition changes the draft. Everyone calling the agent, over MCP or in Copilot, keeps getting the published version until you decide otherwise. The same split exists in the portal; here it becomes something a pipeline can control.

The payoff shows up in review. A change to "net sales" now arrives as a two-line diff in instructions.md, with a test run attached, instead of an edit someone made in the portal on a Tuesday.

2. Generate Ground Truth from SQL

In July the ground truth was a spreadsheet of questions and answers. The weakness of a spreadsheet is that its answers go stale the moment the data changes, and nobody remembers how a number was worked out. So I moved the answers upstream into SQL: one query per test question, tagged with an id and the question text.

-- verify_answers.sql: the house rules at the top, then one block per question
-- Net sales: real invoices only, converted to one currency at monthly rates
-- S2 · Sales · What were net sales by region this year?
SELECT r.region,
CAST(SUM(i.net_invoice_value / br.rate * er.rate) AS decimal(18,0)) AS net_sales
FROM invoices i
JOIN invoice_doc_type d ON d.code = i.doc_type_code AND d.[group] = 'Invoice'
JOIN budget_rate br ON br.from_currency = i.local_currency
JOIN exchange_rate er ON er.from_currency = i.local_currency AND er.date = i.billing_date
GROUP BY r.region
GO

A small script runs every block against the SQL endpoint and writes one CSV per agent with the two columns the SDK expects, question and expected_answer, formatted the way the agent is told to answer (currency symbol, thousands separators, one decimal for percentages). Three good things follow:

  • Expected answers refresh with the data. Rerun the script and the test set is current.
  • The rules are written down twice, on purpose. The house rules at the top of the SQL file are the same rules the agent is given. If they disagree, one of them is wrong, and you find out in review.
  • One source feeds everything. The same script also wrote the recorded answers for my app's offline mode, so the tests, the demo fallback, and the charts all agreed by construction.

3. Check the Example Queries Before the Agent Does

Example queries are the most powerful layer of an agent, which makes a broken one the most dangerous. The agent copies their patterns. So before the draft is touched, the pipeline runs two checks:

Does it run?

Execute every example query against the SQL endpoint. Any error, or zero rows where rows are expected, fails the build. This catches renamed columns and typos that the agent would otherwise learn from.

Is it a good example?

The SDK's evaluate_few_shots scores each pair for Clarity, Relatedness, and Mapping, and flags pairs that conflict. I covered it in depth in the example queries post; here it simply becomes a build step.

4. Evaluate the Draft, Three Times

The SDK's evaluate_data_agent takes a data_agent_stage: production for the published version, or sandbox for the draft. Scoring the sandbox is the whole trick: you test exactly what you are about to publish, without publishing it.

# In a Fabric notebook attached to the lakehouse
import pandas as pd
from fabric.dataagent.evaluation import evaluate_data_agent, get_evaluation_details
truth = pd.read_csv("/lakehouse/default/Files/evaluation/ground_truth_sales.csv")
run_id = evaluate_data_agent(
truth, "Sales Analyst",
data_agent_stage="sandbox", # the draft the pipeline just wrote
table_name="eval_sales", # results land in eval_sales and eval_sales_steps
critic_prompt=critic_prompt,
num_query_repeats=3, # ask every question three times
)
details = get_evaluation_details(run_id, "eval_sales", get_all_rows=True)

Why three repeats. The same question can produce slightly different SQL on different runs. One pass tells you the agent can answer; three passes tell you it reliably does. My rule for the session was blunt: a question counts as passing only if it passes every repeat, and only those questions went on stage as suggestion chips. Questions that pass two out of three are not failures to ignore. They are the first place to add an example query.

The judge. A second model compares each answer with the expected one, following your critic prompt. Mine is lenient about wording and strict about numbers: every figure in the expected answer must appear, rounded sensibly, with the same period and names. Microsoft's documentation shows the prompt with {query}, {expected_answer}, and {actual_answer} placeholders; check the SDK version you have, because the placeholder handling has changed across preview releases.

5. Gate on Regressions, Not Averages

The tempting gate is a threshold: publish if accuracy is above 90%. It is the wrong gate. An instruction change can fix two questions and break one that users rely on every day, and the average goes up. Compare question by question instead:

Production Draft Verdict
3/33/3Fine
2/33/3Improved
2/32/3Still flaky: allowed, but logged
3/3anything lessRegression: block the publish

The production numbers come from the scheduled run you already have from the July setup, so the gate costs one extra evaluation per change. When it blocks, the details table has the SQL the draft wrote for the failing question, which is usually all you need to see which layer to fix.

6. Use the Same Suite for Bigger Decisions

Once the suite exists, it answers questions beyond "is this edit safe":

Standard or preview runtime?

Keep a copy of the agent on the preview runtime (update_configuration(enable_preview_features=True)) and run the same suite on both. Move production when the preview copy scores better and stays better, as I suggested in the what's new post.

Does it stay in its lane?

Add out-of-scope questions with an expected answer like "declines, and points to the Operations agent". For a team of agents, wrong-lane answers are a worse failure than wrong numbers, because they look authoritative.

Is the answer still usable by code?

If an app parses the answers, run the parser over the detail rows: a table must exist, headers must carry units, the Chart and Notes lines must be there. A correct number in the wrong shape still breaks the app.

Did the data break it?

Run the suite after schema changes upstream, not only after agent changes. A renamed column fails the example query check first, which is exactly where you want to find out.

Wiring It into CI

One constraint shapes the wiring: the evaluation runs in a Fabric notebook and writes its results to lakehouse tables. So the pipeline in GitHub Actions or Azure DevOps does not run the evaluation itself. It orchestrates:

# On every pull request that touches agents/
1. validate_fewshots.py # every example query runs (SQL endpoint)
2. update the draft # POST .../dataAgents/{id}/updateDefinition
3. run the evaluation notebook # POST .../items/{notebook}/jobs/instances?jobType=RunNotebook
4. poll the job until it finishes # GET the job instance from the Location header
5. read the results tables # SQL endpoint: draft vs production, per question
6. publish only if nothing regressed # POST .../dataAgents/{id}/staging/publish

Run the pipeline as a service principal with access to the workspace, the agent, and its data sources, and keep the evaluation notebook itself in git too, deployed through the same REST API. Two caveats: the evaluation SDK is in preview, and its result tables and parameters have shifted between releases, so pin the SDK version in the notebook. And the links to each evaluation's conversation thread are visible only to the identity that ran it, so a person investigating a failure should rerun that question as themselves.

Failure Modes This Catches

What happened Where the pipeline stops it
An upstream column was renamedStep 1: the example query no longer runs
A new example contradicts an old oneStep 1: conflict detection
A wording tweak changed what "this year" meansStep 5: date questions regress
A new table gave the agent a tempting wrong joinStep 5: a 3/3 question drops
The answer format driftedShape checks on the detail rows

The Point

None of this is exotic. It is version control, tests, and a release gate, applied to a new kind of artifact. What makes it work for data agents is three platform details that are easy to miss: the draft and published stages are separate, the SDK can score the draft, and the agent's whole definition is available through the REST API. Put those together and "we changed the instructions" stops being a risk you take and becomes a change you can prove.

Next in this thread: what an app should do with the answers once they are right. Turning an agent's markdown into charts people can read at a glance.

Further Reading

Tags

Microsoft FabricAIData AgentsTestingPythonCI/CD