How to Measure AI Search Visibility Without False Precision

Why this is hard to measure

Traditional search reporting had a comforting shape. A keyword, a position, a number of clicks. It was never as precise as it looked, but it was stable enough to trend.

AI answers break that shape in three ways:

• There is no fixed position. The same question can produce different businesses depending on how it is phrased, who asks, where they are, what they asked before, and which model version is running that week. There is no “rank” to record, only a probability of appearing.
• Much of the value never becomes a click. Someone asks an assistant for three firms, notes your name, and searches for you directly later. Your analytics record a branded search or a direct visit. The assistant gets no credit.
• The platforms share little data. There is no equivalent of Search Console for most assistants. Google has, so far, folded its AI features into its overall Search Console reporting rather than breaking them out, so you see the combined effect, not the AI share.

None of this means you should not measure. It means you should measure several imperfect things, look at them together, and be explicit about what each one cannot tell you. If you want the background on why answers vary so much, how AI search engines find and recommend businesses covers the mechanism.

Signal one: referral traffic from assistants

When someone clicks a link in an AI answer, the visit sometimes arrives with a referrer identifying the assistant. In GA4 those visits appear among referral sources, and you can group them into a single “AI assistants” channel so they are easy to follow over time.

What it tells you: which assistants send visitors, which pages they land on, and whether those visitors go on to inquire. Landing pages are especially useful. They show which of your pages assistants consider worth citing.

What it does not tell you: anything about visits that arrive without a referrer (some apps and browsers strip it, so those visits are counted as direct), anything about people who saw your name and did not click, or anything about Google’s AI features, which arrive as ordinary Google organic traffic. Treat AI referral traffic as a floor, not a total.

Keep the channel definition in a written document and update it as new assistants appear. A channel that silently misses a new source will make a trend look worse than it is. The GA4 reporting guide covers custom channel groups.

Signal two: sampling a fixed set of questions

The closest thing to a rankings report is a structured sample: a fixed list of questions, asked in the same way, on a schedule, with the results recorded consistently.

How to build the sample

1. Write the questions buyers actually ask. Pull them from sales calls and inquiry forms. Mix recommendation questions (“who are the best employment lawyers in Saint John”), comparison questions and informational questions in your field. Twenty to forty is a workable size for an established organization.
2. Fix the conditions. Which assistants, signed in or not, which location setting, which language. In New Brunswick, include French versions of the questions that francophone buyers would ask.
3. Run each question more than once. One run per question is an anecdote. Several runs give you a rough frequency.
4. Record the same fields every time: were you mentioned, were you linked or cited as a source, which competitors appeared, and was what the assistant said about you accurate.
5. Repeat on a schedule, monthly or quarterly, and keep the question list stable so the trend means something. Add questions in a separate section rather than swapping them in.

The accuracy field is the one most often skipped and often the most valuable. An assistant that mentions you with the wrong services, an old address or a merged competitor’s details is a problem to fix at the source, and sampling is how you find it.

How to report it

Report it as a share with a plain caveat. An illustrative version: “Across our question set, we were mentioned in roughly a third of answers this quarter, up from roughly a quarter last quarter. Results vary run to run, so treat changes of a few points as noise.” That sentence is less impressive than a score out of 100. It is also true.

The mistakes that skew it

The most common is the executive who asks an assistant one question from a personal account, does not see the company, and concludes the program is failing (or sees it and concludes the job is done). Personal accounts carry history and preferences that shape answers. Run the sample from clean sessions, and resist reacting to any single answer, flattering or not. The second mistake is changing the question list every month, which guarantees a trend line that means nothing.

Signal three: brand demand and mentions

If assistants are recommending you, some of that shows up as people looking for you by name. Watch:

• Branded search impressions and clicks in Search Console, kept separate from non-branded searches. The Search Console guide shows how to filter them.
• Direct traffic to deep pages, such as a specific service page, which people rarely type from memory but often reach from an app or copied link.
• Mentions on other sites: directories, associations, news and forums, since these are among the sources assistants read.

Brand signals are confounded by everything else you do (advertising, events, word of mouth, a news story), so they cannot prove AI visibility on their own. They are useful as a cross-check: if sampling says you appear far more often and branded demand has not moved at all, look harder at both.

Signal four: ask the people who inquire

The simplest and most underused signal is asking new inquiries how they found you, as an open question on the form or in the first call, with the answer recorded in the CRM. People increasingly say things like “I asked ChatGPT” or “Google’s AI summary mentioned you”. Self-reported attribution is imperfect (people misremember, and they often name the last thing they did), but it captures exactly the influence that analytics cannot see.

Make the field free text, or include an AI assistant option in the list, and review the answers monthly. Connecting this to sales outcomes is covered in connecting marketing reporting to sales.

A word on tools and visibility scores

A growing number of tools offer AI visibility tracking. Some are useful for running a large question set consistently. Before trusting any score, ask:

• Which questions does it ask, and can you see and change them?
• How many times is each question run, and under what conditions?
• Does it use the same interfaces your buyers use, or an approximation through an API?
• How does it handle answers that vary between runs?
• Does it show raw answers, so you can check accuracy yourself?

A tool that cannot answer these is selling a number, not a measurement. A single “AI visibility score” with no published method is the definition of false precision.

An honest monthly scorecard

Put the four signals on one page and keep it short. An illustrative layout for a professional services firm:

1. AI assistant referrals: sessions and inquiries, with the top landing pages.
2. Question sample: mention share and citation share across the fixed set (quarterly), plus any inaccuracies found and what was done about them.
3. Brand demand: branded search trend and direct visits to service pages.
4. Inquiries citing AI: count and a few verbatim answers.
5. Caveats: one or two plain sentences on what changed in the method or the platforms.

Then judge the program by the line that matters: qualified inquiries for the services you are trying to grow. Visibility that does not show up there eventually is not worth paying for. This fits inside a broader executive marketing dashboard, and it is how we report the AI side of our SEO and AI visibility work, with the caveats left in. For what to actually do about the results, see SEO and AI search visibility as one strategy.