Five Things That Separate Good Fabric Data Agents from Demos
Anyone can stand up a Fabric Data Agent in an afternoon. Whether it survives contact with real users comes down to five decisions you make long before anyone types a question: scope, schema, instructions, testing, and a plan for production.
I have now built enough Fabric Data Agents, for clients and for myself, to notice a pattern. The ones that get quietly abandoned all failed the same way: not because the model was weak, but because the data and the process around it were. The ones people actually trust share five habits, and none of them involve clever prompting.
This is the overview post. Each tip below is a decision you make once, early, and it pays off for the life of the agent. Where a topic deserves its own deep dive, I link to it rather than compress it here.
- 1 domain per agent. One subject area, a handful of tables. Every source you add is another way for the agent to be confidently wrong.
- 0 agents that fix bad schemas. The agent reads your metadata the way a new hire reads documentation. If the metadata lies, the agent lies.
- 100% of changes need re-testing. A fix for one question can break another. Without an evaluation set you are tuning by vibe, not by evidence.
Tip 1: Start Narrow, One Domain, Few Tables
The most common failure I see is scope. Someone attaches the whole lakehouse, thirty tables across sales, inventory, HR, and web analytics, and expects the agent to be an oracle. What they get is an agent that joins order headers to daily snapshots, sums across grains, and answers finance questions from a marketing table. Fewer sources beats more sources, every time, because every extra table adds candidate join paths and near-duplicate columns for the agent to choose between.
My approach is to start from the questions, not the schema. Before attaching anything, I write down the questions the agent exists to answer. For a recent retail client that list looked like this:
Four tables answered every question on the list. When the client later wanted supply chain questions, we built a second agent rather than widening the first. Two focused agents each stayed accurate; one omniscient agent would have been mediocre at both jobs.
Tip 2: Fix the Schema Before You Touch the Agent
A Data Agent learns your data the way a new hire reads documentation: it takes table names, column names, and descriptions at face value. A human analyst eventually learns that amt2 is the net amount and dt_ld_utc is the load timestamp, not the order date. The agent gets one look at the metadata and believes it. Schema hygiene is the highest-leverage work in the whole build, and it happens before you open the agent configuration at all.
Concretely: give tables and columns friendly names, write descriptions for anything ambiguous, and keep staging tables, backups, and duplicate columns out of the agent's sight entirely. If your only option is a raw bronze layer, put a curated view or a semantic model in front of it first. A gold layer built for human report authors is almost always a better agent source than the tables underneath it, for exactly the same reasons it is better for humans.
Every landmine defused in the view is a paragraph of corrective instructions you never have to write, and a class of wrong answer that becomes impossible instead of merely discouraged.
Tip 3: Instructions and Example Queries Are the Product
Once the scope is narrow and the schema is clean, the remaining accuracy work lives in the agent configuration: agent-level instructions, per-source instructions, and example queries. This is where tribal knowledge gets written down. Business definitions like what "revenue" means, the fiscal calendar, which table wins when two could answer, and the join patterns the agent keeps getting wrong, taught by demonstration through stored question and query pairs.
The rule of thumb I work by: prose instructions change what the agent decides, example queries change what it writes. Wrong table picked, fix it with instructions. Right table, wrong join, fix it with an example query. This topic deserves more than a paragraph, so I give it the full treatment in Writing Instructions That Make Fabric Data Agents Answer Accurately, including the three instruction layers, a real instruction block, and the failure modes each layer fixes. If you read one linked post from this article, read that one.
Tip 4: Test Like It Will Lie to You
The dangerous failure mode of a Data Agent is not an error message. It is a plausible number produced by a wrong query. Nothing crashes, the answer looks reasonable, and it flows straight into a decision. So I treat every agent as if it will lie to me, and I build the lie detector before I start tuning: a small evaluation set of real business questions with answers I have verified independently, in SQL or an existing report.
Ten to twenty questions is enough. Include easy ones, ambiguous ones, and at least one the agent should refuse. After every change, re-run the whole set, not just the question you were fixing, and track the pass rate over time. Mine is a spreadsheet, and a slice of it looks like this:
| Question | Expected | Agent answer | Pass |
|---|---|---|---|
| Total net revenue last complete month | $4.21M | $4.21M | Yes |
| Store count in the Southeast region | 47 (active only) | 52 (included closed) | No |
| Average order value, last quarter | $86.40 | $86.40 | Yes |
| Which employee had the most refunds? | Should refuse, HR scope | Refused, pointed to HR reporting | Yes |
That second row is the whole argument for this discipline. The wrong answer was not absurd, it was five stores off, and nobody eyeballing the output would have caught it. The eval set caught it in two minutes. While testing, also read the generated query the agent shows alongside its answer: "the number looks right" and "the number is right" are different claims, and only the query tells you which one you have. And once the eval set proves its worth, it graduates from spreadsheet to pipeline: the automated testing post covers running it on a schedule with the SDK.
Tip 5: Plan for Production from Day One
A demo runs under your own sign-in, on your screen, with you interpreting the answers. Production is none of those things, and the gap is easier to cross if you plan for it before the first line of configuration. Three questions to answer on day one:
How will it authenticate?
Anything headless, a backend service, a scheduled job, a multi-user chatbot, needs application-level identity rather than a user token. I covered the setup end to end in the service principal post, including the tenant settings that silently 403 you if missed.
Where will it live?
Users will not come to a Fabric workspace to chat with your agent. Teams, an internal portal, a customer-facing app: the destination shapes auth, capacity, and UX decisions. Embedding the agent in your own app is where this goes next.
Who owns quality?
Schemas drift, measures get renamed, and an untended agent decays. Someone must own the eval set, re-run it on a schedule, and triage failures. If nobody is named, the answer is nobody, and users find the decay before you do.
None of these are hard individually. What is hard is retrofitting them onto an agent that grew up as a demo, with a personal login baked into every integration and no record of what "correct" ever looked like.
The Thread Running Through All Five
Look back at the list: scope, schema, instructions, evaluation, operations. Not one of them is about the model. That is the point. Fabric Data Agents fail on data and process, not on the model. The language model behind the agent is a fixed quantity you do not control, and in my experience it is rarely the weakest link. What you control is what the agent can see, what it has been told, how you verify it, and who keeps it honest after launch.
The good news hiding in that: every one of these five habits is ordinary data work. Curating scope, cleaning metadata, writing definitions down, testing against known answers, and assigning ownership are things good BI teams already do. An agent that people trust is mostly a well-run semantic layer with a conversational interface on top. Build it in that order and the demo-to-production gap gets a lot smaller.
Further Reading
- Writing Instructions That Make Fabric Data Agents Answer Accurately: the deep dive behind Tip 3, covering instruction layers, example queries, and the evaluation loop in detail
- Service Principal Authentication for Fabric Data Agents: the production auth setup behind Tip 5
- Embedding a Fabric Data Agent in Your Own App: putting the finished agent in front of real users
- Fabric Data Agent concepts: Microsoft Learn documentation on building and configuring agents