On 1 October we asked ChatGPT for the "best plumber in Canberra", three times.
Each time it listed five plumbers, each with a star rating and a review count, such as 5.0 stars and 652 reviews. The smallest count on any list was 186. The links beside the names went to review and ranking sites. Not one went to a plumber's own website.
We are tyso, a one-person studio in Canberra that gets local businesses found online, and we run this count every month. What we found and what to do about it comes first. The working is below.
What we counted
We asked ChatGPT and Google the questions people ask when they need a tradie: 13 questions each for seven Canberra trades (electricians, plumbers, roofers, landscapers, painters, builders, heating and cooling) and for plumbers in Sydney, Melbourne and Brisbane. Every question was asked three times: 390 ChatGPT answers. We then checked every business it named: is it real, and is it local.
Named means the AI says the business's name in its answer. Linked means the answer has a link you can tap to the business's own website. A business can be named without being linked. That happened a lot.
ChatGPT names local businesses most of the time
Of the 390 answers, 268 named at least one real local business, about seven in ten. Electricians were lowest, on 19 of their 39 answers, and roofers highest, on 33 of 39. Sydney, Melbourne and Brisbane plumbers were named about as often as Canberra's.
In real terms: a customer who asks ChatGPT who to hire usually gets business names back.
What to do: ask it your customers' questions and write down who comes back.
ChatGPT mostly does not link the business's website
ChatGPT mentioned those businesses 1,144 times. About four times in five (923), the business's own website was nowhere in the answer's links.
Among the answers that named a business, the outside site linked most often was hipages, a tradie marketplace, on 52 answers, about one in five. Starworks, a ranking site, was next on 20. Who To Hire, Word of Mouth, Product Review, Region (local news) and Localsearch were on the list too.
In real terms: a customer can be given your name and never be sent to your website. The link beside it may go to a marketplace or review site.
What to do: search your business name on those sites and fix what is wrong or missing. Looking is free. We did not measure whether paying any of them helps.
The businesses it names have about twice the Google reviews
For many of these searches Google shows a map with a short list of businesses under it. We split those businesses in two: the ones ChatGPT also named and the ones it skipped. The typical business it named had about 75 Google reviews. The typical one it skipped had 39. Star ratings were the same.
The plumbers we started with are the big end of that, on 186 reviews or more. In two of the ten trade and city groups (Melbourne plumbers and Canberra roofers) it went the other way.
In real terms: we could compare two things, stars and number of reviews. Only the number of reviews was different.
What to do: ask every customer for a Google review.
When someone is ready to book, Google shows the map
Sometimes Google puts an AI-written answer at the top of the results page. We call it Google's AI answer. On a bit over a third of our Google searches (157 of 443) it was not there, most often when the customer was closest to booking: on the "emergency plumber Canberra open now" kind of question it was missing 21 times out of 30. When it was missing, the map was there, 150 times out of 157.
On cost and "do I need to" questions it nearly always showed up, and in at least 7 of the 10 trades and cities it linked a tradie's own website.
In real terms: when someone is ready to ring, Google still shows them the map, which runs off your free Google Business Profile.
What to do: keep that listing complete, and put a plain page on your website for each cost and "do I need to" question customers ask.
Ask again and the answer changes
We asked most questions three times in one afternoon and checked each time whether a local business's website was among the links. More than one time in three (101 of 275), the result changed between asks.
In real terms: asking once tells you very little.
What to do: ask the same question several times, on different days, and count how often you turn up.
The short list
This is one day's count. We cannot prove these jobs get you named, and nobody can promise it. They fit what we saw:
- Ask every customer for a Google review.
- Keep your Google Business Profile complete.
- Look yourself up on the sites the AI links.
- Put plain cost and "do I need to" pages on your website.
Then check by asking the AI more than once.
If you want a hand
Those four jobs are simple and take hours. We do them.
First we check what ChatGPT and Google's map say about you now, more than once. Then we:
- Set up review requests, so Google reviews come in steadily.
- Keep your Google Business Profile complete.
- Check your listings on the sites the AI links and fix what is wrong or missing.
- Write your plain cost and "do I need to" pages.
Each month we check again and show you whether it moved. Nobody can promise a name in an answer, us included.
If you want to see what the AI says about your own business, the free audit asks it for you. It takes about ten seconds and shows your score before it asks for an email.
What we got wrong last month
Edition 2 said the AI named no local business for "best plumber in Canberra" and questions like it. That was wrong: our tool counted links and missed names. The full correction is in the data below.
The full data
Everything from here down is for anyone who wants to check the working. This is Edition 3 of the AU AI Search Watch. The two engines, ChatGPT and Google's AI answer, are reported separately, and named and linked are kept apart.
One thing to know before the tables. For ChatGPT, "linked" in these tables also counts links its search gathered without showing them in the answer, as Edition 2 did. Where it matters, the count of links shown in the answer is given beside it.
The limits
- One day, one operator, 168 questions. This is a pattern, not a market study, and not advice proven to work.
- Verification is a judgement. The evidence behind each call is kept, so a specific one can be disputed.
- API answers are not app answers. The ChatGPT app, with its memory and personalisation, can answer differently.
- The engine changed between months. ChatGPT's link lists roughly doubled in length, so link counts are not strictly like for like.
- A failed check is unknown, never zero, and is left out of every count.
- Three interests, declared: Sophisticated Manes is a client, ANT Painting is the family painting business that Tyson Farrington, who runs tyso, runs with his dad, and tyso.io is our own site. ANT Painting also counts as a verified local business in the Canberra painters set.
What was measured and how
| Item | Detail |
|---|---|
| Date | 1 October 2026, one sitting, 12:31 to 13:23 AEST for Google's AI answer and ChatGPT. The Google AI Mode pass finished at 13:28 |
| Questions | 168 in 13 sets: cat grooming Melbourne 8 (a client's set), house painting Canberra 22 (ANT Painting's set, family-run), SEO for tradies Canberra 8 (our own site's set), and ten trade and city sets of 13 |
| New sets | Six, frozen on 1 October before their first run: Canberra painters, builders, heating and cooling, and the Canberra plumber questions repeated word for word for Sydney, Melbourne and Brisbane |
| Unchanged | The original 90 questions, word for word |
| Engines | Google AI Overview (Google's AI answer) and ChatGPT search |
| Asks per question | 3 on eleven sets, 1 on two: 35 runs, 888 checks |
| Google failure rate | 51 of 444 first attempts failed, all with the provider error DataForSEO 40101. A retry recovered 50. Final: 1 of 444 |
| ChatGPT failure rate | 0 of 444 |
| Answers returned | 728 of 888. One check failed. Of the other 159 with no answer, 157 were Google showing no AI answer, which is a result and not a failure |
| Links captured | 7,762 across 1,348 websites |
| Morning attempt | A 07:00 attempt stopped after 9 of 17 runs, with Google failing 12 to 38% of checks. Kept untouched, used only as a same-day repeat |
| Data source and spend | DataForSEO, USD $5.74 across every pass |
The ten trade and city sets use the same 13 kinds of question, reworded for each trade and place.
How businesses were verified. Across the ten trade and city sets there were 1,380 distinct candidates, linked or named. 108 were excluded by rule (directories, social, government and so on). 105 were carried from September's records. The other 1,167 were checked this month against the business's own website, ABN record or licence register by AI agents working to a written rule, one evidence line each. Every business that would count was then re-checked independently by a second agent told to try to refute it: 721 re-checked, 36 knocked back to unverifiable. Of the 1,167, 911 were businesses. The rest were directories, government pages, cost guides and the like.
What changed since Edition 2
| Edition 2 | Edition 3 | |
|---|---|---|
| Question sets | 7 | 13 |
| Questions | 90 | 168 |
| What is counted | Linked only | Named and linked, separately |
| A failed check | Left out | Retried, left out only if it fails again |
| Answer wording | Not kept | Kept for every answer |
What Edition 2 got wrong, in full
Edition 2 said that on "best plumber in Canberra", and the same question for three other trades, the AI named a local business in none of eight cases: four trades, each asked of ChatGPT and of Google. That was the headline, and it was wrong.
Our tool counted links. It was meant to read the names too, but it read the wrong part of the response, so they always came back empty. We measured "nobody linked" and printed "nobody named". Four of those eight were Google showing no AI answer at all. In ChatGPT's four, the links went to review sites, directories and local news sites. Whether it named businesses in September cannot be recovered, because we did not keep the wording of the answers. This month, with the wording kept, it named between 3 and 9 businesses in every one of 30 such answers. The correct reading of September's result is "no local business was linked".
Four other things in Edition 2 were wrong.
- The ask-again table. It measured whether an answer came back with any link at all, a different thing from the result it claimed. Measured properly, 85 of 118 results agreed across repeat asks, where we printed 102. Electricians were 18 of 26, where we printed 25. Plumbers were 19 of 26, where we printed 22. Roofers were 16 of 25, where we printed 18, and landscapers 16 of 25, where we printed 21. Cat grooming was right at 16 of 16.
- The table of most-linked sites. The Google and ChatGPT columns were headed as link counts but held something else, and were wrong on 13 of the 14 rows even read as questions per engine. Three sites that belonged in it were left out: antpainting.com.au on 9 questions (the family business, so declared), leakyroof.com.au on 8 and service.com.au on 8. That makes the sentence "Not one business website appears in that list" false.
- The share of Google searches with no AI answer. We printed 41%. That counted the ten failed checks it said it had left out. With them left out it is 86 of 200, which is 43%.
- The count of businesses named without a link. Edition 2 said it could not be made. It could: the tool was reading the wrong part of the response. It was fixed on 1 October 2026, and the count is in this edition.
One reading also did not hold up. Edition 2 saw landscapers miss on the three urgent-sounding questions and took that to mean local businesses get named on emergencies. This month we added painters and builders, two more trades with no real emergency, to test it. On links, painters scored 5 of 6 on those questions and builders 5 of 6, while landscapers scored 1 of 6 (1 of 5 in September, with one check lost). Counted by name alone (the "Named, question by question" table below), ChatGPT named a landscaper on all 3. So emergencies do not explain it. Whatever does looks particular to landscaping, and it shows only on links.
The tool is fixed and the wording of every answer is now kept. Edition 2's page now carries a dated correction.
Named, question by question (ChatGPT)
Y: in at least two of three asks, ChatGPT said the name of a verified local business of that trade. A dot: not on this count. A link does not count here. Syd, Melb and Bris are plumbers. Google was scored on links only, so it is not in this table.
The total is 94 of 130, and it is a lower limit. The count reads the businesses ChatGPT shows as cards, so a name written only in the text is missed: checked against the text, painters row 13 is a Y, which makes 95. And 15 of the 36 dots include an ask where ChatGPT's search did not run and it replied with one holding line. Read a dot as "not counted here", not as "never named".
| # | Question | Elec | Plumb | Roof | Land | Paint | Build | Heat/cool | Syd | Melb | Bris |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Emergency or quotes | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 2 | Describes the problem | . | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 3 | After hours | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 4 | Big job, cost | . | Y | Y | . | Y | . | Y | . | . | . |
| 5 | Big job, who to hire | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 6 | Mid job, cost | . | . | . | . | Y | . | . | . | . | Y |
| 7 | Mid job, who to hire | . | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 8 | Rules question | . | . | . | . | . | . | . | . | . | . |
| 9 | Regulated job, who to hire | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 10 | Growth job, who does it | . | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 11 | Growth job, cost | . | . | Y | . | . | . | Y | . | . | . |
| 12 | "Best" in the city | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 13 | Near me, licensed | Y | Y | Y | Y | . | Y | Y | Y | Y | Y |
| Total, of 13 | 6 | 10 | 11 | 9 | 10 | 9 | 11 | 9 | 9 | 10 |
Named or linked, question by question (ChatGPT)
Y: in at least two of three asks, ChatGPT either named a verified local business of that trade or had one's website in its list of links, shown in the answer or not. A dot: neither. A link alone can earn a Y here, so this count is looser than the table above: 111 of 130 against 94.
| # | Question | Elec | Plumb | Roof | Land | Paint | Build | Heat/cool | Syd | Melb | Bris |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Emergency or quotes | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 2 | Describes the problem | . | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 3 | After hours | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 4 | Big job, cost | Y | Y | Y | . | Y | Y | Y | Y | Y | Y |
| 5 | Big job, who to hire | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 6 | Mid job, cost | Y | . | . | Y | Y | Y | Y | Y | Y | Y |
| 7 | Mid job, who to hire | . | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 8 | Rules question | . | . | . | . | . | . | . | . | . | . |
| 9 | Regulated job, who to hire | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 10 | Growth job, who does it | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 11 | Growth job, cost | Y | Y | Y | Y | Y | Y | Y | . | . | . |
| 12 | "Best" in the city | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 13 | Near me, licensed | Y | Y | Y | Y | . | Y | Y | Y | Y | Y |
| Total, of 13 | 10 | 11 | 11 | 11 | 11 | 12 | 12 | 11 | 11 | 11 |
Linked, question by question (Google / ChatGPT)
Each cell is Google's AI answer, then ChatGPT. Y: the engine linked a verified local business's own website in at least two of three asks. A question mark is a cell lost to a failed check. For ChatGPT a link counts whether it was shown in the answer or only gathered by its search, as in Edition 2. Counting links shown in the answer only, ChatGPT falls from 89 of 130 to 64 of 130.
| # | Question | Elec | Plumb | Roof | Land | Paint | Build | Heat/cool | Syd | Melb | Bris |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Emergency or quotes | . / Y | . / Y | Y / Y | . / . | Y / Y | Y / Y | . / Y | . / Y | . / Y | . / Y |
| 2 | Describes the problem | Y / . | Y / Y | Y / Y | . / . | Y / Y | Y / . | Y / Y | Y / Y | Y / Y | . / . |
| 3 | After hours | . / Y | . / Y | . / Y | . / Y | . / Y | Y / Y | Y / Y | . / Y | . / Y | . / Y |
| 4 | Big job, cost | Y / Y | . / . | . / Y | Y / . | Y / Y | Y / Y | . / Y | Y / Y | Y / . | Y / Y |
| 5 | Big job, who to hire | . / Y | . / Y | Y / Y | . / Y | . / Y | Y / Y | . / Y | . / Y | . / Y | . / Y |
| 6 | Mid job, cost | . / Y | Y / . | Y / . | Y / Y | Y / Y | ? / Y | Y / Y | Y / Y | Y / Y | Y / Y |
| 7 | Mid job, who to hire | . / . | . / Y | . / Y | . / Y | . / Y | . / Y | . / Y | . / Y | . / Y | . / Y |
| 8 | Rules question | . / . | Y / . | Y / . | Y / . | . / . | Y / . | Y / . | Y / . | . / . | Y / . |
| 9 | Regulated job, who to hire | . / Y | . / Y | Y / Y | Y / Y | . / Y | Y / Y | . / Y | . / Y | . / Y | . / Y |
| 10 | Growth job, who does it | . / Y | . / Y | . / Y | Y / Y | Y / Y | Y / Y | . / Y | Y / Y | . / Y | Y / Y |
| 11 | Growth job, cost | . / Y | Y / . | Y / Y | Y / Y | Y / Y | Y / Y | Y / Y | Y / . | Y / . | Y / . |
| 12 | "Best" in the city | . / . | . / . | Y / . | . / . | . / . | Y / Y | . / Y | . / . | . / . | . / . |
| 13 | Near me, licensed | . / . | . / . | . / . | . / . | . / . | . / . | . / Y | . / Y | . / . | . / . |
| Either engine, of 13 | 9 | 10 | 12 | 9 | 10 | 12 | 13 | 12 | 10 | 10 | |
| Google, of 13 | 2 | 4 | 8 | 6 | 6 | 10 of 12 | 5 | 6 | 4 | 5 | |
| ChatGPT, of 13 | 8 | 7 | 9 | 7 | 10 | 10 | 12 | 10 | 8 | 8 |
Row 8, the rules question, is the one row with no Y for ChatGPT in any of the three tables: 0 of 10 sets counted by name, 0 of 10 by name or link, and 0 of 10 by link. It pointed to the regulator instead. Google's AI answer linked a local business's site there in 7 of 10. For plumbers the question was "do I need a licensed gasfitter to install a gas cooktop in the ACT". Edition 2 found the same split on four trades.
Plumbers in four cities
The same thirteen questions, with only place names and the state changed.
| Measure | Canberra | Sydney | Melbourne | Brisbane |
|---|---|---|---|---|
| ChatGPT answers naming a verified local plumber | 29 of 39 | 27 of 39 | 26 of 39 | 27 of 39 |
| Different verified local businesses named | 33 | 63 | 51 | 43 |
| Top three's share of mentions | 28.6% | 14.4% | 14.7% | 25.5% |
| Linked by ChatGPT, questions (shown or gathered) | 7 of 13 | 10 of 13 | 8 of 13 | 8 of 13 |
| Linked by Google's AI answer, questions | 4 of 13 | 6 of 13 | 4 of 13 | 5 of 13 |
| Linked by either engine, questions | 10 of 13 | 12 of 13 | 10 of 13 | 10 of 13 |
| No Google AI answer | 17 of 39 | 17 of 39 | 20 of 39 | 15 of 39 |
| Map shown when no AI answer | 17 of 17 | 17 of 17 | 18 of 20 | 14 of 15 |
The cities differ in how many businesses share the mentions. In Canberra the three most-mentioned plumbers took more than a quarter of all mentions (28.6%). In Sydney and Melbourne the top three took about one in seven.
Where ChatGPT's names come from
Verified local businesses only, counted by name, in each of the ten trade and city sets. Each set is 39 ChatGPT answers.
| Set | Answers naming one | Different businesses | Mentions | Own website linked in the answer | Not linked | No website |
|---|---|---|---|---|---|---|
| Electricians, Canberra | 19 of 39 | 27 | 85 | 4 | 81 | 0 |
| Plumbers, Canberra | 29 of 39 | 33 | 119 | 19 | 97 | 3 |
| Roofers, Canberra | 33 of 39 | 29 | 125 | 29 | 93 | 3 |
| Landscapers, Canberra | 26 of 39 | 49 | 127 | 7 | 111 | 9 |
| Painters, Canberra | 29 of 39 | 33 | 127 | 33 | 91 | 3 |
| Builders, Canberra | 22 of 39 | 43 | 105 | 8 | 95 | 2 |
| Heating and cooling, Canberra | 30 of 39 | 30 | 126 | 29 | 97 | 0 |
| Plumbers, Sydney | 27 of 39 | 63 | 125 | 32 | 92 | 1 |
| Plumbers, Melbourne | 26 of 39 | 51 | 95 | 15 | 70 | 10 |
| Plumbers, Brisbane | 27 of 39 | 43 | 110 | 12 | 96 | 2 |
| All ten sets | 268 of 390 | 389 | 1,144 | 188 | 923 | 33 |
The pooled count of different businesses, 389, is lower than the column adds to because a business named in more than one Canberra trade is counted once. Before the local check, ChatGPT named 445 different businesses in 269 of the 390 answers.
The table below shows the outside sites most often linked in an answer that named a business, of those 269 answers. It shows which sites were linked, not where ChatGPT got the name. Business websites are in the full list too, and the most frequent of them was linked in 11 answers.
| Site | Answers | Kind |
|---|---|---|
| hipages | 52 | Marketplace |
| Starworks | 20 | Review and ranking site |
| planning.act.gov.au | 18 | Government |
| whatsthedamage.com.au | 13 | Cost guide |
| Access Canberra | 10 | Government |
| climatechoices.act.gov.au | 9 | Government |
| Who To Hire | 8 | Review and ranking site |
| Word of Mouth | 8 | Review site and directory |
| Product Review | 7 | Review site |
| Region | 7 | Local media |
| Localsearch | 6 | Directory |
On the "best" question, ChatGPT's links were mostly to review and ranking sites: Starworks, Word of Mouth, Who To Hire, Region, Product Review and hipages, with ThreeBestRated on a few answers. Google showed no AI answer on 22 of those 30 searches and showed the map on 21 of the 22.
Reviews, for businesses on Google's map for these questions, ten sets pooled. The 19 paid placements are left out. The middle value is the median. In this table a business counts as named when its name or website matches a listing on Google's map, without the local check used above. ChatGPT did not name 78 of the businesses on the map, and 74 of them showed a review count.
| Group | Businesses with a review count | Middle review count | Middle star rating |
|---|---|---|---|
| ChatGPT named | 126 | 75.5 | 5 |
| ChatGPT did not name | 74 | 39 | 5 |
Two of the ten sets ran the other way. Melbourne plumbers had a middle count of 194 for the 20 ChatGPT named and 245 for the 7 it did not. Canberra roofers had 28 for the 13 it named and 34.5 for the 4 it did not, which is a very small group.
Google's AI answer, by question type
Across all thirteen sets Google showed no AI answer on 157 of 443 usable checks (35.4%), and the map was there on 150 of those 157 (95.5%). By question, for the ten trade and city sets:
| # | Question | No AI answer | Map shown when no AI answer |
|---|---|---|---|
| 1 | Emergency or quotes | 21 of 30 | 20 of 21 |
| 2 | Describes the problem | 0 of 30 | n/a |
| 3 | After hours | 15 of 30 | 15 of 15 |
| 4 | Big job, cost | 3 of 30 | 3 of 3 |
| 5 | Big job, who to hire | 11 of 30 | 11 of 11 |
| 6 | Mid job, cost | 1 of 29 | 0 of 1 |
| 7 | Mid job, who to hire | 21 of 30 | 20 of 21 |
| 8 | Rules question | 0 of 30 | n/a |
| 9 | Regulated job, who to hire | 15 of 30 | 14 of 15 |
| 10 | Growth job, who does it | 12 of 30 | 11 of 12 |
| 11 | Growth job, cost | 0 of 30 | n/a |
| 12 | "Best" in the city | 22 of 30 | 21 of 22 |
| 13 | Near me, licensed | 22 of 30 | 22 of 22 |
On the questions where it nearly always showed, Google's AI answer linked a verified local business's own website in most sets: 7 of 10 on the rules question, and 7 of 10, 8 of 9 and 9 of 10 on the three cost questions (rows 8, 4, 6 and 11 of the linked table above).
Two engines, two different webs
Of all the websites the two engines linked across the original seven sets (the three we have an interest in and four Canberra trades), 14.1% were linked by both. That was 16.5% in September and 11.7% in Edition 1, which had three sets. Across all 13 sets it is 15.6%. Of the sources ChatGPT linked, 19.6% were in Google's top ten results for the same question. Of the times ChatGPT named a business with a website, 19.5% were of a business on Google's map for the same question.
| Kind of source | ChatGPT, links in the answer (850) | Google's AI answer (2,295) |
|---|---|---|
| Business websites | 45.2% | 32.3% |
| Google Business Profile cards | none | 37.3% |
| Government | 23.4% | 2.9% |
| Directories and marketplaces | 21.2% | 9.4% |
| Forums, social and video | 0.1% | 7.0% |
Month on month, four trades
Linked, counted by question, for the four trades with a September reading. ChatGPT's counts include links its search gathered, in both months.
| Trade | Either engine, Sep | Either engine, Oct | Google, Sep | Google, Oct | ChatGPT, Sep | ChatGPT, Oct |
|---|---|---|---|---|---|---|
| Electricians | 11 of 13 | 9 of 13 | 5 of 13 | 2 of 13 | 8 of 13 | 8 of 13 |
| Plumbers | 10 of 13 | 10 of 13 | 5 of 13 | 4 of 13 | 8 of 13 | 7 of 13 |
| Roofers | 11 of 13 | 12 of 13 | 6 of 12 | 8 of 13 | 9 of 13 | 9 of 13 |
| Landscapers | 9 of 13 | 9 of 13 | 6 of 12 | 6 of 13 | 6 of 13 | 7 of 13 |
Landscapers' September figure is 10 of 13 when re-scored against October's list of verified businesses. A caution on the whole table: ChatGPT's link lists roughly doubled in length between the two months, from about 5.7 links per answer to about 11. The engine changed, so these counts are not strictly like for like.
Ask again: how often the result held
The result is whether a verified local business was linked or, for the client set, whether the client was. For ChatGPT that includes links its search gathered. A case is one question on one engine with all three asks usable.
| Set | Same all three times |
|---|---|
| Cat grooming, Melbourne (client set) | 14 of 16 |
| Electricians, Canberra | 14 of 26 |
| Plumbers, Canberra | 18 of 26 |
| Roofers, Canberra | 16 of 26 |
| Landscapers, Canberra | 16 of 26 |
| Painters, Canberra | 16 of 26 |
| Builders, Canberra | 12 of 25 |
| Heating and cooling, Canberra | 16 of 26 |
| Plumbers, Sydney | 17 of 26 |
| Plumbers, Melbourne | 18 of 26 |
| Plumbers, Brisbane | 17 of 26 |
| All eleven sets | 174 of 275 |
The stopped morning attempt is a second check on the same day. Morning against afternoon, Google's result matched on 64 of 83 comparisons and ChatGPT's on 87 of 106.
The three sets we have an interest in
Questions on which the engine linked the business's own website. For ChatGPT the count includes links its search gathered, and the figure in brackets counts links shown in the answer only.
| Set | Engine | September | October |
|---|---|---|---|
| Sophisticated Manes (client) | Google's AI answer | 0 of 8 | 1 of 8 |
| Sophisticated Manes (client) | ChatGPT | 3 of 8 | 2 of 8 (1) |
| ANT Painting (family-run) | Google's AI answer | 5 of 22 | 2 of 22 |
| ANT Painting (family-run) | ChatGPT | 7 of 22 | 7 of 22 (6) |
| tyso.io (ours) | Google's AI answer | 0 of 8 | 1 of 8 |
| tyso.io (ours) | ChatGPT | 0 of 8 | 3 of 8 (2) |
On the Sophisticated Manes set, two ChatGPT results flipped between asks.
Google AI Mode: a first reading, not verified
AI Mode is the chat-style answer inside Google Search. We read it off Google's search results page, the same way we read Google's AI answer. Edition 1 ruled out collecting from Gemini's own API, and that still stands.
This is AI Mode's first reading here: 444 checks, none failed. We counted a business when AI Mode showed its Google Business Profile card or listed it in the body of the answer. Those are businesses Google lists for the location. They have not been through the checks described above, so these figures cannot sit beside the verified ones.
On that count, AI Mode showed a business in 227 of 390 answers, and on 75 of the 130 questions in at least two of three asks. By question type it runs the opposite way to Google's AI answer: 0 of 10 sets on each of the three cost questions and on the rules question, and 9 or 10 of 10 on the who-to-hire and "best" questions.
The reviews pattern showed here too, on the same unverified count. Among businesses on Google's map, the middle review count was 60.5 for the 164 that AI Mode showed and 32 for the 36 it did not, with a middle star rating of 5 in both groups.
Bing
We built the Bing measurement this month and ran all 168 questions once. 57 responses were for the question asked, 104 were the results page of some other search, and 7 were empty. No Bing figure is published. The sorting rule and every response are kept with the run files.
What we promised, and what Edition 4 will measure
| Promised | Outcome |
|---|---|
| The same 90 questions, re-run (Edition 2) | Delivered, untouched |
| A second reading for the four trade sets (Edition 2) | Delivered |
| A stated failure rate for each engine (Edition 2) | Delivered: Google 51 of 444 first attempts and 1 after retry, ChatGPT 0 of 444 |
| A count of businesses named without a link (promised in Edition 1, missed in Edition 2) | Delivered |
| Bing (promised in Edition 1, missed in Edition 2) | Built, but the provider was unreliable: 57 of 168 usable, no figure |
Edition 4 will measure the same 168 questions, with a second reading for the six new sets. Answer wording and raw responses will be kept every month, named and linked will both be reported, and Bing will be included only if the provider's pages match the question asked.
Verification
Every figure on this page comes from a dated file in our measurement archive. The question sets are frozen, one file per set, one question per line, and every answer and every list of links is stored raw, so the whole page can be recomputed from the files. The record of each business check is kept with the run files, one line of evidence per business. We hand the raw files over on request: tys@tyso.io.
Who made this. tyso is a one-person growth studio in Canberra. This page is one small operator's measurement, run on one client's questions, a family business's questions, its own site's questions and ten trade and city sets, published because we have not found an Australian equivalent. It is not a market study and should not be cited as one.
Corrections. If a number here is wrong, tell us and we will fix it and date the fix. tys@tyso.io