ChatGPT Search vs. Google vs. Bing: Search Results Review via @sejournal, @TaylorDanRW

The role of AI in SEO has kept rapidly evolving over the past few months.

Google formally launched AI Overviews in May, and Bing launched its generative search pilot in July, with a full launch at the start of October. On July 25, OpenAI announced a SearchGPT prototype. Then, on October 31, it launched ChatGPT Search.

ChatGPT are calling the new product, which I’ve tested for this article, as ChatGPT Search, but parts of the industry are already starting to the names ChatGPT Search and SearchGPT interchangeably as they are effectively one and the same.

Both the Google and Bing AI launches, alongside the increase of multi-modal search (e.g., TikTok), have been impactful, but the ChatGPT Search launch feels different.

ChatGPT already drives traffic to websites, and as of August 2024, it reported 200 million weekly average users (WAU) – this is all while AI is gaining momentum and user trust, with skepticism still a barrier to mass adoption.

In February 2024, Gartner predicted that “search engine volume” will drop 25% by 2026.

Having played with SearchGPT, this prediction feels like it could become a reality.

ChatGPT Search Observation Summary

• SearchGPT will (and can) cite webpages that do not rank in the top 100 classic search results of Bing. This raises the importance of understanding the difference between indexed and ranked.
• There are discrepancies between Bing’s own generative results and the SearchGPT responses. More often than not, these are substantial.
• A single domain can have multiple pages cited within a single response.
• SearchGPT’s response for the same query can vary, even if the search is made from the same ChatGPT account and IP.
• On some niche queries I tested that have limited information online, SearchGPT was more accurate than both Google and Bing.
• The local search experience feels lacking, and for a number of queries, not trustworthy.
• Some SearchGPT maps can be populated wholly by results from a single source, and the number of “map results” tends to decrease the more sources are included.
• Some cited links from SearchGPT were appended with the parameter ?utm_source=chatgpt.com when clicked on. UTM tracking is a key feature.
• Ranking higher in Bing doesn’t mean referencing better in SearchGPT, but it’s more closely aligned than AI Overview data.
• Responses are heavily text-driven. Some do trigger a map (local searches) and others images – but these aren’t guaranteed.
• Queries in fashion and travel tend to trigger images in the response ahead of other queries when tested.
• The image sources in carousels don’t appear to be included in the citations sidebar.
• It is possible for content behind a paywall to be cited by SearchGPT.
• SearchGPT can cite a webpage that returns an active 404 when you try to view it.
• SearchGPT exhibits a better understanding of time-sensitive queries than AI Overviews, even when a time variable isn’t included in the query.

Increasing Bing Market Relevance & Importance To ChatGPT Search Inclusion

Bing is fast becoming a search engine we need to change our views on. For as long as I can remember, Bing has always been an afterthought to the industry and has only really been prominent in specific sectors and target markets.

While its direct market share might not be growing to rival Google in that sense, the fact that Bing is being utilized by multiple large language models (LLMs), and ChatGPT Search is no exception.

OpenAI has been using Bing for a long time, so its inclusion in SearchGPT isn’t a revelation, but a reinforcement of Bing’s evolving place in the overall “search” landscape. We need to stop just comparing search engines and their direct market share.

This means that websites (and webpages) not indexed by Bing will not appear in ChatGPT Search.

Anecdotally, traffic from LLMs to all websites I have GA4 access to has increased exponentially over the past three months.

Looker Studio Report: Increasing LLM Sessions (Year to Date) (Image from author, November 2024)

Of all LLM traffic, ChatGPT accounts for 31% of my website (year-to-date), with it as much as 60% on some websites alongside Perplexity, Claude, Copilot, etc.

Breakdown of LLM traffic (year to date). GA4 data. Looker Studio visualized (Image from author, November 2024)

However, this doesn’t necessarily mean you need to rank high in Bing to be included. Much like Google’s AI Overviews, ChatGPT selects sources outside of the top-ranking results.

It’s also worth noting that while Bing is important to the SearchGPT ecosystem, it isn’t the only source.

Comparing SearchGPT, Google, And Bing Search Results

To see how the search results in ChatGPT compare to both Bing and Google, I’ve looked at a number of queries across Local, Your Money or Your Life (YMYL), ecommerce, and some informational/time-sensitive searches that I’ve personally performed over the weekend.

I’ve tried to live with ChatGPT search for 48 hours rather than subjecting it to a barrage of random queries.

I’ve not recorded all of the queries in this article, but I have included the key ones as well as some oddities and differences.

The number of webpages referenced that don’t appear in the top 100 classic ranks of Bing really highlights that other sources are at play here, and that optimizing for Bing doesn’t mean optimizing for ChatGPT and other LLMs directly.

In AI Overviews, when a webpage is referenced outside of the top 100 classic results, I feel we just assume it’s come from Google’s overall database. After all, Google crawling and building a database(s) of pages on the internet is known – but could OpenAI be doing the same?

ChatGPT Search “Maps” & Local Search

Local queries are important to a number of businesses. SearchGPT (at the time of writing) has a somewhat limited local search experience.

When searching for [enterprise technical seo agency], it provided me with three results: Two close to my IP address and the third in a random location in the South of England.

Image from author, November 2024

There are a handful of “agency hotbeds” in the UK, and a lot of agencies in my local area. So, if SearchGPT were going off IP, it would have included more agencies locally. To then stretch the net further and ignore London in the results for broader results doesn’t make sense.

While none of the three recommendations were wrong, SearchGPT seemed to hallucinate on the second result.

In the first and third results, it added the label “Advertising Agency,” but the second result was given the label “Telecommunication Service.”

I have spent time looking through the agency’s website, social profiles, backlink profiles. I’ve crawled the website with various custom path extractions setup. I’ve looked at all the schema markups that were implemented. I cannot fathom or find a reason as to why it has been classified as a telecommunications service.

When repeating the search for [digital marketing agency in new york], it provides the map, but with eight results all from the same source.

Image from author, November 2024

This is less useful as a result, as being a reference for this query is basically a game of appearing in a certain website list – which is a paid membership only.

The website used to compile all the map results ranks No. 4 in Bing classically, so it is at least a high-ranking result – but this just feels like scraping a list on a website and is a lazy result.

Triggering The Map

The map result (as shown in the screenshot above) doesn’t always trigger.

For a number of queries when I specified a location, it just provided the list results without the option to view a map. This is different from the “list” toggle on the Map, as this keeps the map but creates a list underneath it (which is very similar to the Google Map Pack).

Speaking of Google Maps, if you ask SearchGPT explicitly to show you the results on a Map, it directs you to Google Maps:

Image from author, November 2024

YMYL Searches

YMYL queries refer to search queries that could potentially impact a person’s health, safety, or financial stability.

It is widely understood (and communicated) that Google and Bing place greater emphasis on these queries and ensure that accurate information is provided from trustworthy sources.

“Remineralizing Gum”

According to Glimpse, “remineralizing gums” saw exponential search interest in October 2024.

As this is an emerging query in the health space, one could assume that the trend will be picked up by websites selling the product (or adjacent products) and producing content around the topic.

SearchGPT feels like it played it safe with this query, only utilizing WebMD and Wikipedia as citations before providing references to a variety of other websites.

“How To Lower Cholesterol”

The response for this query doesn’t highlight anything dangerous or out of the ordinary and is measured in response.

I also recognize a number of the websites in the citations, which help with trusting the information.

Interestingly, SearchGPT is pulling from two separate URLs on the Mayo Clinic website, despite one of them not ranking in the top 100 for Bing.

“Who Can Sign A Contract On Behalf Of A Company”

On the first run of the query, the response contained a more textbook answer, citing the four main criteria under English Law for a contract to be valid (Offer, Acceptance, Consideration, Intent), as well as two additional bullet points around capacity and that the contract itself is legal.

Image from author, November 2024

From this response, the cited source URLs and top referenced URLs barely ranked in the top 100 results of Bing:

However, when I perform the same query on a different computer – on the same account and same internet connection – I get a completely different response:

Image from author, November 2024

The citations and search results provided also differ, with only one of the original citation sources remaining.

On the second run of this query, even fewer citations ranked in the top 100 Bing results.

Interestingly, when looking for the domains on the second data pull for this query in Google, the domains ranked higher with different URLs than those cited by ChatGPT Search.

“How Many Credit Cards Should I Have?”

Testing financial queries, I saw the closest alignment to both Bing and Google’s first page of results.

Four of the five cited sources in the table above appear on Google’s first page (in the UK) for the query, with the only exception being the bank.marksandspencer.com URL, which appears in the middle of page two on Google.

Ecommerce Searches

Transactional searches in ecommerce refer to queries where the user intends to make a purchase or complete a transaction.

These are often high-intent searches, where the user is actively seeking a product or service, and they’re close to making a buying decision.

In my models, I’ve talked about how AI can’t necessarily satisfy the full extent of an ecommerce query (as AI can’t facilitate the shopping and purchase experience); I’ve classed this user group as “Purchasers.”

Dan Taylor’s Model of AI Influence on User Types (Image from author, November 2024)

The simpler and more open the query, the more likely AI can steer and influence, but as a user narrows down to make a purchase (moves from being a shopper to a purchaser), it still needs to engage with brands and ecommerce websites.

“Best Christmas Gifts For Him 2025”

Another positive point for SearchGPT is that all of the webpages referenced and cited for the query have been published (or at least claim to have been published) in the past month.

An adjacent test I’m running on AI Overviews at the moment for similar queries has content as far back as 2017 being referenced and cited in AI Overviews.

“Black Friday Deals”

Image from author, November 2024

While being very single-source heavy, the second citation for [Tips for maximizing Black Friday Savings] came from an Australian website.

Google’s AI Overviews have received some criticism on X (Twitter) and other platforms from a number of SEO professionals as to the inclusion of Australian sources in non-AU markets.

This could be an indication that SearchGPT may exhibit the same behavior across queries.

“Nike Air Max Size 10 Deals”

Just searching for products in SearchGPT seems to be interpreted as informational, but adding a modifier as “deals” brings about the commercial intent and changes the response, while maintaining the conversational tone:

Image from author, November 2024

This result also brought about some hallucinations/misinformation.

On the Nike source cited:

• I couldn’t find a pair of Air Max ’90s for £115.99 – they were all cheaper.
• The only Air Max 97 on the cited webpage was a children’s shoe, reduced to £52.49 – again, a lot cheaper.

This isn’t ideal, as if I was looking to buy a pair of Air Max ’90s at a cheap price through Google’s PLAs, I can find them for £80 (and some cheaper depending on the model). This is a lot cheaper than what SearchGPT is saying Nike is selling them for directly, so it could detract sales from your brand website if this misinformation persists.

This query was another instance of Google ranking URLs for the domain, e.g., JDSports, for the query in a good position – but not the URL cited by ChatGPT Search.

Travel Searches

“Best Holiday Destinations 2025”

Much like a number of queries in the travel sector, this triggered an image pack above the carousel.

Image from author, November 2024

Having worked with a large number of travel companies and supported the industry through events and the Institute of Travel & Tourism, I can honestly say this is a very eclectic list of destinations to consider.

The citations are very publisher-heavy, with other travel companies (and travel blogs) in the search results section.

“Luxury Hotels In Napa Valley”

To my surprise, this query triggered the SearchGPT Map.

Image from author, November 2024

The Telegraph.co.uk result was referenced twice for two different hotels in the SearchGPT Map Pack.

Reviewing the recommendations and sources, there doesn’t appear to be a logical connection as to why these sources and (specifically) these locations were chosen.

Google’s hotel feature is a lot more comprehensive and interactive, so I don’t see this threatening the hotel search journey any time soon.

“Abu Dhabi Grand Prix”

Sports tourism is growing in popularity, and F1 is a globally popular sport.

Searching for specific races is also a tricky query, which is what I wanted to test out.

Searching for “[location] grand prix” has a number of common interpretations. It could be an informational search, a navigational search, or with transactional intent and wanting to research ticket prices.

Image from author, November 2024

I feel like SearchGPT understood this, as the five websites referenced as citations cover these bases quite well, and it doesn’t lean into being too heavily informational or transactional.

This is a similar result to any Grand Prix query, e.g., Bahrain GP, Belgium GP, Singapore GP.

From the list of referenced webpages, the **P1Travel URL posed an anomaly across all of the searches I’ve done to try and break SearchGPT. The URL returns a 404:

Screenshot from p1travel.com, November 2024

Travel SEO has been evolving for a number of years now, and in my opinion, LLM platforms pose the greatest potential to disrupt this industry more consistently than others.

Other Query Observations

Over the past 48 hours, I’ve made notes of queries made throughout the day through Google and repeated them on Bing and SearchGPT.

“England V Samoa Score”

Despite being at the match itself, what triggered me to look at this query was my wife commenting on the score – and the score she quoted was wrong.

So, I replicated her search on Google and found the below result:

Screenshot for search for [england v samoa score], Google, November 2024

While the “Top Stories” section does bring accurate results, Google’s feature at the top of the SERP brings the result of a game over a year ago – in a different sport.

Now, I can understand why Google is falling down here.

Rugby Union is the second most popular sport in the UK, and Rugby League is the 11th most popular. So, choosing to show this sport over the other would be acknowledging a greater common interpretation of the query, and looking to cater to a larger potential search audience.

I then wanted to see how ChatGPT Search fared with the same query:

Screenshot of search for [england v samoa score], ChatGPT Search, November 2024

While it isn’t as visually prominent, the generative text cited from the BBC contains the correct scoreline as well as other factually correct information.

The downfall then comes that the news stories being pulled through relate to a different game (that happened the previous week).

As I reviewed all of the citations provided by ChatGPT Search, I noticed that it pulls through news stories relating to the same incorrect match that Google cited prominently in the SERP feature.

But the AI has been able to distinguish the two and provide a timely result, potentially showing a greater understanding of queries that may be time-sensitive and the concept of QDF against leaning towards the “more common” dominant interpretation.

As Bing plays a prominent role in SearchGPT, it’s no surprise to see Bing also prioritizing the more recent incident of England vs. Samoa through the Bing News feature, although Bing does then feature the same match from over a year ago in the special sports feature.

Screenshot of search for [england v samoa score], Bing, November 2024

“Manchester United v Chelsea”

To test true timeliness around live events, I’ve performed (and compared) this search query during the live soccer game.

I actually found the ChatGPT Search result to be the most informational and useful – outside of reporting the live score.

Screenshot of search for [manchester united v chelsea], ChatGPT, November 2024

The ChatGPT overview provides a lot of information in a single snippet – including broadcast information.

If you search for broadcast information on Google and Bing, you don’t get this depth of information directly in the results – and unfortunately, a lot of publishers bury this information deep in an article, even if that article is written (and headlined) to target the query specifically.

To see if ChatGPT Search would show the in-game score, I tried repeatedly with variations of “Manchester United v Chelsea live score|in game score|current score|score now.”

But despite Bing showing the live score in a SERP feature and a number of sources being accessed by ChatGPT that contain the information, all it would give was variations of the below response – refreshingly asking the user to visit the source website for the information.

Image from author, November 2024

“Cheese Not Sticking To Pizza”

This was a prominent query that Google’s AI Overviews fell down on earlier this year, drawing scrutiny from the mass media.

Image from author, November 2024

I’ve tested a number of the queries that tripped Google up, and so far, SearchGPT seems to avoid hallucinations and confusing results.

Do I Think This Will Disrupt Google?

Google has been facing disruption since the mass market’s relationship with mobile devices changed.

Moving away from desktop and utilizing apps and browsers to research, shop, and be entertained opened the doors for new platforms to compete for screen time and attention.

On a recent podcast I ran, John Mueller said that when Google first started to push towards mobile-first, a lot of SEO pros and webmasters were skeptical and vocal that users wouldn’t shift from mobile to desktop to view their websites.

SEO as a practice is only 30 (or so) years old – and in 30 years, how users interact with businesses has changed drastically.

AI platforms, such as SearchGPT, will disrupt Google as adoption increases and the “early majority” of consumers start to use LLM platforms daily for tasks they otherwise would have engaged the wider Google ecosystem with.

We can draw some parallels to the Model T, and when Henry Ford introduced the car, it was seen as a novelty and not for the masses – with people seeing them as “faster horses” than a revolutionary mode of transport.

As SEO professionals, we can see the “revolution” part of it in motion, but we must remember that to the mass market, a lot of AI is merely gimmicks such as removing things and people from the background of images on phones or using things like Circle Search.

SearchGPT will contribute to the overall disruption, and it might pioneer the “what’s possible, “much like Jeeves was a pioneer in mass market internet search engine adoption.

Also, I don’t feel this would be a complete answer if I didn’t ask SearchGPT directly if it will disrupt Google:

Image from author, November 2024

Which Search Is Best?

In testing SearchGPT felt as though it had a better understanding around time sensitive queries and used more recent sources.

Removing visual experience from the equation, SearchGPT is better as objectively you want recent content when you’re searching for information around holidays, gifts, products for “now,” and not viewpoints from over a year ago.

For local search, ChatGPT Search still has a long way to go and falls short of were Google and Bing are.

In my opinion, information was better in ChatGPT Search than in Google and Bing, but not on all queries. Overall I still feel it is behind Google’s AI Overviews and Bing’s Generative Search in terms of answer queries behind text-driven informational results – and the ChatGPT Search interface is a little bit too detached from what we’re used to seeing from a “Search engine”.

Closing Thoughts

Brett Tabke has referred to SearchGPT as the “final” search engine, and I agree with this.

While SearchGPT currently offers certain advantages over Google and traditional search models, it is still a work in progress.

However, its potential trajectory is one of the key insights to take away from SearchGPT.

This development indicates that user interaction with the internet is evolving once again, and as SEO pros, we must adapt as an industry.

The rise of platforms like TikTok highlights the shift toward multi-modal search, urging us to move beyond a narrow focus on the “engine” aspect of our work.

The core principles of SEO, particularly technical SEO, will always remain relevant, as AI crawlers must crawl, identify, and access content similarly to traditional search engine crawlers.

I believe it’s time for us to adjust our SEO terminology, shifting the conversation from ranking to referencing, and realigning what the overall success of an SEO campaign is.

And we finally start taking credit for upper funnel visibility and touchpoints and are not measured solely on bottom-of-the-funnel conversions.

Rather than asking the questions “How do we optimize for SearchGPT?” or “How do we optimize for LLMs?,” we need to be looking at our market and the different LLM platforms and asking questions like:

• How does our audience interact with SearchGPT, and how will SearchGPT stop our audience leaving the ChatGPT ecosystem?
• What information does the user get if they stay in this ecosystem versus what they would receive “traditionally”?
• When (and if) the user leaves this ecosystem, what is their new entry point in our perceived funnel? How well adapted is our funnel for new-to-brand (NTB) touchpoints at this stage?

More resources:

• SearchGPT vs. Google: Early Analysis & User Feedback
• 8 ChatGPT Alternatives You Can Try In 2024
• Leveraging Generative AI Tools For SEO

Featured Image: Roman Samborskyi/Shutterstock