How to tell whether your AI project actually worked

Many AI projects in small businesses end without anyone being able to say whether they worked. The tool went live, people said it was "pretty good", and six months later the licence renews automatically. Sometimes the project helped. Sometimes it quietly added a checking step that cost more time than it saved.

You do not need a data team to find out. You need one number recorded before the change, the same number recorded after, and a few checks on quality and use. This article sets out a method a 10 to 50 person business can run with a spreadsheet.

Pick one before-and-after number

Before the project starts, choose the single measure that should change if it works. It should be something you can count without much effort and that matters to the business. Good examples:

• Time to first reply on customer enquiries, if the project is about follow-up.
• Staff hours per task, such as minutes to prepare a quote, a report or a renewal summary.
• Error or rework rate, such as the share of invoices or orders that need correcting.
• Volume handled by the same team, such as applications processed per week.
• Conversion, such as the share of enquiries that become a booked call or quote.

Avoid measures like "productivity" or "efficiency" with no definition. If you cannot write down how you will count it, it is not a measure yet.

Measure a baseline for 2 to 4 weeks

Record your chosen number for two to four normal weeks before anything changes. Two weeks is the minimum to smooth out one unusual day. Four is better if your work is uneven or seasonal. Avoid baselines that fall over holidays, year-end or a known busy period unless the "after" period will be the same.

Keep the method simple enough that people actually do it. For task time, ask the two or three people who do the task to log start and finish times for each instance in a shared sheet. For reply time, most inboxes and CRMs already record when a message arrived and when it was answered.

If you skip the baseline, you will end up comparing the new system with people's memory of the old one, and memory is a poor measuring tool.

Measure quality, not only speed

A tool that halves the time on a task but introduces errors may be a net loss. Track at least one quality measure alongside your main number:

• Accuracy against a test set: collect 20 to 50 real past examples with known correct answers, and run the AI against them before launch and after every change.
• Edit rate: how often staff have to change the AI's draft, and how much. Heavy edits on most drafts means the time saving is smaller than it looks.
• Escaped errors: mistakes that reached a customer, such as a wrong price, a wrong date or a complaint.

If you already have an assistant producing wrong answers, our article on why AI chatbots give wrong answers explains how to build a test set.

Check adoption after week six

Usage in the first two weeks tells you about curiosity. Usage around week six tells you whether the tool fits the work. Check how many of the intended users are still using it, and for what share of the task. If people have drifted back to the old way, ask them why. The answer is usually more useful than any metric: the drafts take too long to fix, the tool is in the wrong place, or it does not handle the common cases.

Leading and lagging indicators

Some measures move quickly and some take months. Leading indicators, such as usage, edit rate and time per task, show up within weeks and tell you whether the project is on track. Lagging indicators, such as revenue, customer retention or staff overtime, take longer and are affected by many other things. Judge the project mainly on leading indicators at 30 and 60 days, and look for the lagging ones at 90 days and beyond, without claiming the AI caused every change.

How do you calculate the return on an AI project?

The basic calculation is hours saved per month, multiplied by the loaded hourly cost of the people saving them, minus the monthly cost of the licences and upkeep. Spread any one-time setup cost over the period you expect to use the system, such as 12 months.

Example: an insurance brokerage with 18 staff

This is an illustrative example with assumed numbers, not a client case or a benchmark. A brokerage has six account managers who each write renewal summaries for commercial clients. It is testing an AI assistant that drafts each summary from the policy documents for the account manager to check.

Item	Baseline (4 weeks)	After (weeks 5 to 8 of use)
Renewal summaries per month	120	120
Average minutes per summary	25	12, including checking
Hours per month	50	24
Summaries needing correction after review by a senior broker	6	5

That is 26 hours saved a month. At an assumed loaded cost of $45 an hour (wage plus benefits and overhead; use your own figure), the time is worth about $1,170 a month. Against that, six Microsoft 365 Copilot Business seats at the list price of $21 USD per user per month, as of November 2026, cost $126 USD a month, plus whatever setup and upkeep cost spread over a year. Quality held steady, so the saving is real.

Two cautions. Hours saved only turn into value if they go to useful work, such as more client contact, rather than disappearing. And if quality had dropped, the true saving would be smaller, because someone would be fixing errors later.

When to stop an AI project

Agree a stop rule before you start. For example: "If, after 60 days, time per summary has not dropped by at least a quarter, or more than one in ten drafts contain a factual error, we stop or redesign." Choose thresholds that make sense for your business; these are examples, not industry standards.

Serious projects have stop rules. A stop rule protects you from the slow drift where a tool that is not working keeps its licence because nobody wants to be the one to cancel it. Stopping is also not always the answer. Often the result is a narrower scope: keep the AI for the simple cases and send the complex ones back to people.

A one-page review for 30, 60 and 90 days

At each review, answer the same questions in a single page:

1. Main measure: baseline, current value, change.
2. Quality: test set score, edit rate, escaped errors since last review.
3. Adoption: how many intended users are using it, and for what share of the work.
4. Cost: licences, usage charges and staff time spent maintaining it.
5. Problems: what went wrong, and what was changed.
6. Decision: continue, adjust scope, or stop, checked against the stop rule.

Keep the reviews short and keep them on the calendar. A 20-minute meeting at day 30 catches problems while they are cheap to fix.

What to do next

• Write down the one number your next or current AI project should change.
• Start recording the baseline this week, before any further changes.
• Collect 20 real past examples to use as a test set.
• Write your stop rule and put the 30, 60 and 90 day reviews in the calendar.

If the project is about enquiries, our guide on responding to enquiries faster covers which numbers to track. If you have an AI setup already running and are not sure it is working, our workflow rescue and quality care service starts with this kind of measurement. For enquiry projects, see enquiry and follow-up automation.