Which tools track AI search share of voice and citations? (Episode 2)

Welcome to episode two of Question.Marketing podcast, Will My Brand Be Visible in AI Search, and particularly how we work with answer engine optimisation or AEO.

Now, being in that field, I am offered tools upon tools. Every single day I get people contacting me, as will you guys if you are listening to this and you are in market for wanting your business to appear in the AI search, LLM citation lists. If you want to be listed, there are so many tools coming out. One of the estimates puts it between 50 and 150 current tools that you will have put in front of you, and the thing is those tools are unfortunately not tracking, as far as I am concerned, correctly. We have got everybody marketing these tools, and are they working how they should?

Now, we all know if you want to appear in ChatGPT, Gemini, Perplexity, Copilot, Google AI Overviews, AI Mode, et cetera, it is a coin flip at the moment. So you could be sat in Manchester and let's say you are a recruitment business. I am working with a recruitment business in Manchester and I ask you to search for your business in one of the LLMs and you come up and everyone is happy. That is great. We are shown, we are cited.

Well, the problem is unless you have the correct tracking tools in place, that reporting is absolutely not going to give you a result that is at all credible. So if you are going to go and search, or you are using a tool to go and search or cite or find out if you are mentioned for particular prompts, how many times is that tool going and running a search? And a lot of these tools are only running one or two searches across different LLMs at a time.

So if, for example, a different result is shown every 20 results, right? So we are getting 20 different results for 20 searches, you might come up 5% of the time, i.e. one in 20. Now, is that result at all accurate? And my argument would be no, it cannot be accurate. It cannot be even close to accurate because to get a realistic understanding of how much share of voice, that is the term that a lot of people are using, share of voice. How much share of voice does your brand have across a particular set of large language models? You need to run a lot of queries. You need to run hundreds and hundreds of searches and you cannot do that manually. We are not asking you to do that, but can you trust a tool that is tracking, searching once or twice a day across a set of models? And that is going to lead to a very thin set of credible results. They are not credible.

And a lot of these tools, this is what I am saying, you need to be careful that a lot of these tools are only running a handful of searches giving you a supposed share of voice, and hopefully marketing directors or teams are not reliant on a lot of these tools because the results: garbage in, garbage out. It is giving you a completely false sense of security, maybe.

Right? But the tools that are going out there and doing, for example, tracking 100 prompts across 9,000 responses, that is going to cost you a lot of money to do that because the amount of AI credit that you are going to burn to generate those interactions with the models is going to cost hundreds of dollars. Now, if you are paying for a tool that is supposedly costing you 49 bucks a month, and you are hoping that that is going to give you decent tracking, it honestly absolutely is not.

So, what are these tools doing? All of these AI visibility tools, they solve or try to solve the same problem in the same way. So there is no alternative. You cannot go out and find a different way of tracking. In the good old days of SEO, we would have used Google AdWords to go in, run a quick Google Keyword Planner, search for how many times a particular search is made per month. And this is one thing that we are not necessarily able to do at the moment with large language models: take a prompt and get a response on how many times a particular question is getting asked of an LLM or each of the individual LLMs, because we are finding that at the moment a lot of them are trying to ring-fence their own databases.

So you will find you could search something on Google AI Overviews or Gemini and you will find a certain set of results and then a completely different set of results on ChatGPT because it is serving from a different database. And there does not seem to be a way necessarily yet to be able to say, right, okay, we are getting a prompt set that is tracked and evidenced. So we have no idea necessarily how many times that prompt is being searched across the different models.

You could have done that with Google. Google has the tools to measure, in traditional search engine land, how many times a particular search is being made, and then you can target your marketing towards that. That search is being searched, let's say, 5,000 times a month in the UK. It is valuable to us, and therefore, if we rank at the top of Google or in the top three, we can expect X number of clicks through to our landing page. We can measure that. We can look at if we rank number one.

And this is not about ranking. This is about can we even track the number of times that a particular prompt is mentioned or asked? Are we looking at the value of that prompt? I would hope so, because different prompts will have different levels of intent within them. So that is important, as would be intent within keyword searches.

So OpenAI, Anthropic, Google, Perplexity, they do not publish this. They do not publish what people are asking, or how often they ask it. What they are doing is saying, here is a set of sources, and these are the respected answers and the trusted answers. And they are not looking like they are going to share the prompts or that prompt frequency. There is no supposed visibility score on the market that is constructed from prompts or that the prompt vendor, if you want to call it that, has inferred somewhere else. We do not find that each of these engines is publishing that data, which is going to be important at some point in the future.

[Personal observation / verify product detail, not a confirmed product fact. Simon (via CoS) approved this softening for the published transcript page. Preferred public framing: expanding ChatGPT ads tooling / Ads Manager plugin; verify user-facing agent behaviour. Video remains as filmed.]

We have seen, I think yesterday or today, what looks to me like OpenAI expanding ChatGPT ads tooling (Ads Manager style / plugin direction). My reading of that announcement, and this is my observation to verify against the live product, is that intent-aware ads in ChatGPT may surface more clearly when you type certain prompts, because the system can see intent level and whether a product relates to the prompt. Whether that shows up as full AI agents managing adverts for end users is the detail to check. But you do not have the ability now. There is not the ability to see the prompt list or the prompt volume, so that is going to be really important. Does that come? I do not know. I would hope so because as an advertiser, we would like to be optimising content to the most valuable prompts. Of course we would.

But this particular piece of content today is mainly around those search prompts. If you are using a tool to measure that, which of the tools is actually valuable? And the majority of them are not at the moment. So if you are looking for a tool, and you are trying to find something that is going to measure this properly for you, you need to find the most stable tool. And unfortunately at the moment, you need to probably pay a significant amount for it if you are looking to track. Probably even if you are tracking 30 prompts, you might be into hundreds of dollars if you want a robust share of voice. And to see whether your brand is actually being cited across all of those different platforms is really, really difficult because it has got to go out and talk to those models.

There will be an element of this in the future where we are looking at local prompting. This is not something that I am seeing people necessarily talking about at the moment. And if you are a local business, particularly B2B, and people are searching locally for a service provider that is able to satisfy that, I think I am testing this with some clients at the moment. So I am thinking that if we are including localised search within prompting, I think that will find its way into being really, really valuable, especially the recruitment example, where I am looking to track, for example, who are the B Corp certified recruitment agencies or partners in Manchester. That is a lot more targeted. We do not know how many times that is being searched, but my presumption would be that when you search around that phrase...

Now, this is where it is going to get really complicated. I do not want to get too complicated in an episode too early. But if we start searching around a phrase, the AI will be able to use content that is interpretive or close to or feeling the sentiment of what you are looking to find out and include prompts that you have not necessarily searched. I do not know whether I have explained myself clearly enough here. But in the B Corp certified recruitment partner example in Manchester, if somebody then had searched, "I am looking for a recruitment partner," or, "Which recruitment agencies in Manchester are B Corp certified and able to find me software engineers for AWS?" Now, that question fits the previous prompt closely in terms of AI citation and would probably cite the optimised pages or content, be that blogs, videos, et cetera, that are published. As long as they have got unique information gain in them, they would then probably be cited.

But I think what we are saying today is be careful which tools you choose. Come onto my site and I am going to list a whole bunch of them. I will give you some ideas which ones I am using, which ones I think work, which ones do not, and which ones. I think more than ever, the phrase you get what you pay for is relevant here, and this is an area we have not had to pay for before. If you were running traditional search, you could run these type of tracking of keywords and phrases really, really cheaply because at the end of the day, you did not need to go and run those queries thousands of times, which we are looking at having to do now to get an accurate share of voice measurement.

If we are looking to test 10,000 keywords through Google AI Mode three times on the same day, that is a huge amount of search. The traditional search tools like SEMrush or Ahrefs would allow you to just track all this sort of stuff. But now, if you are going to run a prompt a thousand times through different models to get a genuine share of voice, you are going to need to pay a significant amount in AI credits. So be careful. Look at the published information I am going to attach to this blog. It will be published on the Question.Marketing website. Hopefully you found this episode useful and I will see you on the next one.