A post from @mardehaym on October 16 has been read more than 178,000 times, and it is one of the most practical accounts of building a business AI agent shared on X this month. A private equity operating partner asked his team to build AI agents inside a portfolio company's healthcare billing platform, on real claims data, under HIPAA. The company's CPTO had already tried AI and "summed it up: it 'did a bunch of silly crap.'"
Six weeks later, that same CPTO was testing the first agent with real analytical questions. The first deploy scored 59 out of 60 on the team's test set, and "the flagship denials question reconciles to the penny." Then the client asked for more agents.
Nothing in the post depends on healthcare or private equity. The method is a template for any owner who wants an agent that answers questions about the business without inventing things. Here it is, translated for a service business with 5 to 75 people.
TL;DR
- The failed first attempt pointed a model at the data and let it guess. The working build started with domain work: trace one transaction end to end and write down how experts reason about it.
- Keep each agent to one job, check every answer type twice (is the query right, is the answer right), and let nothing an agent suggests be applied on its own.
- Build a test set of real questions with known answers before deploying. Theirs grew from 10 to 60.
- Put one data layer between agents and your systems so the model can change without a rebuild. AI drafts, your team approves.
Why the first attempt failed
The post is blunt about the usual pattern: "Most teams skip this. They point a model at the product, watch it make things up, and decide AI isn't ready for their industry. The domain work is what makes it ready."
That will sound familiar to anyone who has connected a chatbot to a shared drive and asked it about last quarter. The model does not know which report is current, which customer codes mean what, or that the "Completed" status in your job software sometimes means "Completed, not yet invoiced." It fills the gaps confidently. In the claims company's case, every question about its own data had previously meant "an expert hand-writing SQL against a 20-year-old schema." The agent had no expert in the loop, so it guessed.
Step 1: Trace one transaction end to end
Before the engagement started, the team "traced one claim from charge to cash through their data." That single trace is what turns a database into something an agent can reason about.
For a service business, pick one ordinary job and follow it:
- Lead. Where did it come in, and where is that recorded?
- Quote. Which system holds the estimate, and how is it linked to the customer?
- Job. How is the work scheduled, completed and marked done?
- Invoice. Where is it created, and what ties it back to the job?
- Cash. How do you know it was paid, and where do partial payments and write-offs live?
Write down every system, field and status the job passes through. You will find at least one place where the link is a person's memory. That is exactly where an agent would make things up.
Step 2: Write down how your experts reason
Next, the team "sat with their billing experts and wrote down how they reason about it, so the agents work from their logic instead of guessing."
In your business, the experts are the office manager who knows which jobs are really overdue, the estimator who knows which quotes are padded, and the bookkeeper who knows which customers always pay late but always pay. Interview them with real examples:
- "How do you decide this invoice is a problem?"
- "Which numbers do you never trust, and why?"
- "When two systems disagree, which one wins?"
Their answers become the agent's instructions. This is the part no vendor can do for you, and it is where most of the value is.
Step 3: One job per agent, two checks per answer
The team kept "every agent to one job." The first one handles the research questions the analysts used to answer by hand. And "every answer type passes two checks. Their data experts confirm the query is right, and the business confirms the answer is. Nothing an agent suggests gets applied on its own."
Those are two different failures. An agent can run a correct query that answers the wrong question, or the right question with a broken query. For a small business, the equivalent is simple: whoever knows the systems confirms the agent looked in the right place, and whoever owns the decision confirms the answer makes sense. Often that is two people. Sometimes it is one person wearing both hats, as long as they check both things.
Step 4: Build the test set before you trust it
Then "we built the eval set." It "started with 10 real questions their experts already knew the answers to. By first deploy it held 60. Automated judges grade every answer, and each one gets checked against the source database."
You do not need automated judges to borrow this. You need a spreadsheet:
| Column | Example |
|---|---|
| Question | Which jobs completed in September are not yet invoiced? |
| Known answer | 7 jobs, listed by job number |
| Where the answer lives | Job software status plus accounting invoices |
| Agent's answer | Filled in on each test run |
| Pass or fail | Exact match on the list |
Start with ten questions your team already answers every month. Add a question every time the agent gets something wrong in daily use. Rerun the whole set whenever you change the instructions or the model. The post's own line: "Most teams demo their agents and hope they hold up. We prove ours work before anyone relies on them."
Step 5: Put one layer between agents and your systems
The last design choice is easy to skip: "We put one data layer between the agents and their databases, so every agent reads the claims data the same way and the model on top can change without a rebuild. Connecting the model was the smallest part of the work."
In small-business terms, this means agents should not each have their own login to your job software, accounting system and CRM. Give them one sanctioned path, such as an official connector, an export that refreshes on a schedule, or a small reporting database, with read-only access by default. When a better model comes out next quarter, you swap the model and keep the path, the instructions and the test set.
What "making things up" looks like in practice
Even in a good build, agents get things wrong. In testing, the CPTO pushed back on one answer and "the agent admitted the mistake and corrected it." That is a healthy sign, and it only happens when a person who knows the data is reviewing answers rather than accepting them.
That is the rule SpiAI builds every client workflow around: AI drafts, your team approves. In an analytics agent, the draft is the answer and the query behind it. A person who knows the business decides whether to act on it.
How long this takes
The claims build reached its first real test in six weeks, on a 20-year-old schema under HIPAA. A small service business with a few modern systems can usually trace a job, interview its experts and assemble a ten-question test set in a couple of weeks. The work is mostly conversations and spreadsheets, not code.
FAQ
What is the first step in building an AI agent for my business?
Trace one ordinary transaction, such as a job from lead to cash, through every system it touches, and write down where each piece of data lives. That map is what lets an agent find answers instead of guessing.
Why do AI agents make things up about business data?
They are usually pointed at data without the context an expert carries, such as which status really means done or which system wins when two disagree. Writing that reasoning into the agent's instructions removes most of the guessing.
Should one AI agent handle everything?
No. Keeping each agent to one job, such as answering reporting questions or drafting follow-ups, makes it easier to test and easier to trust.
What is an eval set and do I need one?
It is a list of real questions with known answers that you rerun whenever the agent changes. A spreadsheet with ten questions your team already answers every month is enough to start.
How accurate should an agent be before we rely on it?
Accurate on your full test set, with a person still reviewing answers that lead to decisions. The healthcare team's first deploy scored 59 out of 60 and still kept humans checking.
Can an AI agent change records or apply its suggestions on its own?
In the build described here, nothing an agent suggested was applied on its own. For a small business, keep that rule until the agent has a long, clean track record on low-risk tasks.
How long does it take to build a business AI agent?
The healthcare team reached real testing in six weeks on a 20-year-old system. A small business with a few modern tools can often trace a job and build a first test set in about two weeks.