The Future of Search

Type​ a few words into Google and hit ‘return’. Almost instantaneously, a list of links will appear. To find them, you may have to scroll past a bit of clutter – ads and, these days, an ‘AI Overview’ – but even if your query is obscure, and mine often are, it’s nevertheless quite likely that one of the links on your screen will take you to what you’re looking for. That’s striking, given that there are probably more than a billion sites on the web, and more than fifty times as many webpages.

On the foundation of that everyday miracle, a company currently worth around $3 trillion was built, yet today the future of Google is far from certain. It was founded in September 1998, at which point the world wide web, to which it became an indispensable guide, was less than ten years old. Google was by no means the first web search engine, but its older competitors had been weakened by ‘spamming’, much of it by the owners of the web’s already prevalent porn sites. Just as Google was to do, these early search engines deployed ‘web crawlers’ to find websites, ingest their contents and assemble an electronic index of them. They then used that index to find sites whose contents seemed the best match to the words in the user’s query. A spammer such as the owner of a porn site could plaster their site with words which, while irrelevant to the site’s content, were likely to appear in web searches. Often hidden from the users’ sight – encoded, for example, in the same colour as the background – those words would still be ingested by web crawlers. By the late 1990s, it was possible, even usual, to enter an entirely innocent search query – ‘skiing’, ‘beach holidays’, ‘best colleges’ – and be served up a bunch of links to porn.

In the mid to late 1990s, Google’s co-founders, Larry Page and Sergey Brin, were PhD students at Stanford University’s Computer Science Department. One of the problems Page was working on was how to increase the chances that the first entries someone would see in the comments section on a website would be useful, even authoritative. What was needed, as Page told Steven Levy, a tech journalist and historian of Google, was a ‘rating system’. In thinking about how websites could be rated, Page was struck by the analogy between the links to a website that the owners of other websites create and the citations that an authoritative scientific paper receives. The greater the number of links, the higher the probability that the site was well regarded, especially if the links were from sites that were themselves of high quality.

Using thousands of human beings to rate millions of websites wasn’t necessary, Page and Brin realised. ‘It’s all recursive,’ as Levy reports Page saying in 2001. ‘How good you are is determined by who links to you,’ and how good they are is determined by who links to them. ‘It’s all a big circle. But mathematics is great. You can solve this.’ Their algorithm, PageRank, did not entirely stop porn sites and other spammers infiltrating the results of unrelated searches – one of Google’s engineers, Matt Cutts, used to organise a ‘Look for Porn Day’ before each new version of its web index was launched – but it did help Google to improve substantially on earlier search engines.

Page’s undramatic word ‘recursive’ hid a giant material challenge. You can’t find the incoming links to a website just by examining the website itself. You have to go instead to the sites that link to it. But since you don’t know in advance which they are, you will have to crawl large expanses of the web to find them. The logic of what Page and Brin were setting out to do involved them in a hugely ambitious project: to ingest and index effectively every website in existence. That, in essence, is what Google still does.

One way to approach the problem would have been to buy the most powerful computers available. When Google launched it had around $1 million in the bank. It raised $25 million from venture capitalists in 1999, but that still wasn’t enough to pay for a decent number of expensive machines. Instead, Google’s engineers lined metal trays with electrically insulating cork, and packed them with low-cost computer components of the kind found in cheap PCs. One early Google employee, Douglas Edwards, remembers visiting the Santa Clara data centre where Google was renting space for the hardware. ‘Every square inch was crammed with racks bristling with stripped-down CPUs [central processing units],’ he writes. ‘There were 21 racks and more than fifteen hundred machines, each sprouting cables like Play-Doh pushed through a spaghetti press. Where other [companies’] cages were right-angled and inorganic, Google’s swarmed with life, a giant termite mound dense with frenetic activity and intersecting curves.’

By June 2000, Google’s bargain-basement web crawlers had ingested more than a billion webpages. In the months before that, though, the company had encountered a crisis. Cheap machines crash, and cheap computer memory is easily corrupted by overheating or even by the impact of cosmic rays. James Somers told the story in the New Yorker in 2018. Google’s crawlers kept failing, making it hard or even impossible to update its most crucial data structure, the web index. The solution the company came up with changed computing for ever. It was to conceive of computing not as something done by a single machine, but as something done by an array of tens of thousands of machines, housed together in a warehouse and automatically managed so as to circumvent the inevitable failure of individual units. As Somers put it, Google found a way to give its programmers the capacity ‘to wield the machines in its data centres as if they were a single, planet-size computer’.

‘Planet-size’ is an exaggeration, but it paints a picture: as early as 2011, Levy suggests, Google may have been deploying a million machines, still mostly cheap commodity hardware, in roughly two dozen data centres around the world. How do you ‘parallelise’ a giant data analysis task: in other words, how do you distribute it across a huge ensemble of machines? How do you organise the communication that has to take place among those machines? What does your program do when, as is inevitable when it is running on tens of thousands of machines, one or more of them crashes? After all, the search engine has to come up with results in real time, and can’t afford to stop and reboot. Will your engineers and researchers have to spend all their time dealing with such issues, rather than getting on with the data analysis?

It is astonishing that, given the centrality of these questions to what was rapidly becoming digital capitalism, Google openly published its answer to them. In an electronically available preprint of a paper presented in 2004 to a computer science symposium in San Francisco, two Google engineers, Jeff Dean and Sanjay Ghemawat, described the system they had built for the company. MapReduce, as they called it, permitted programs to be ‘automatically parallelised and executed on a large cluster of commodity machines’:

The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilise the resources of a large distributed system.

Google didn’t actually release the code for MapReduce, but Dean and Ghemawat said enough to prompt the computer scientist Doug Cutting, then working for the web portal Yahoo, and his colleague Mike Cafarella to lead the production of Hadoop, a free, open-source analogue to MapReduce. If you are involved in the analysis of big data – as many people in the tech sector are – you probably aren’t now using MapReduce or Hadoop, but you are likely to be using one or more systems derived from them.

The implementation of MapReduce helped Google discover, painfully at first, how to ‘scale’, which is the process at the heart of digital capitalism. That Google had done it gave others confidence they could do it too, and the fact that Google didn’t try too hard to keep its innovations secret helped them learn how. Once you can scale, goals that seemed hopelessly ambitious suddenly appear within your grasp. If Google’s systems could ingest and index close to the entirety of the world’s webpages and respond to billions of search queries every day, then why not begin to ingest all the world’s literature too (Google Books), or create a detailed, interactive digital map of the world (Google Maps), along with an interactive photo-image of the planet from satellites and aircraft (Google Earth), and panoramic images of its streets, at least in countries that allow Google’s camera-carrying cars (Google Street View)? While you’re at it, why not offer a free, high-quality email service, and not worry too much about the strain on your servers if more than a billion people sign up for a Gmail account?

All of this involved Google in creating, collecting and assembling a historically unprecedented quantity of data. (The US National Security Agency might previously have come close, though we don’t really know, and Facebook was soon to accomplish something similar. Facebook was narrower than Google in the scope of its activities, but the data its users uploaded about themselves and their lives was richer.) You could therefore be forgiven for assuming that the legal troubles currently besetting Google are to do with its misuse of this data. On balance, however, Google seems to have behaved fairly responsibly in this respect. Its legal difficulties concern instead whether it plays too dominant a role in advertising markets. In 2015, Google was incorporated into a new holding company, Alphabet, but the latter’s ‘other bets’, as it calls them in its financial statements, contributed less than 1 per cent of the $350 billion it earned in 2024. Google’s fast-growing cloud computing business was more substantial, contributing 12 per cent. However, fully three-quarters of Alphabet’s 2024 revenue came from advertising, the bulk (nearly $200 billion), as has been the case ever since Google’s earliest years, from selling the ads that accompany Google search results.

In two lawsuits, the US Department of Justice and several US states have accused Google of monopolising two different areas of advertising. The first case, heard by a federal court in the District of Columbia, concerns the markets for ‘general search’ (i.e. searches of the kind for which you use Google, not the more specific searches for which you might use Amazon, Facebook or Expedia) and the standard ads, often consisting simply of text, that accompany the results of such searches. The DoJ had little difficulty in establishing that, in the words of the court’s judgment in August 2024, Google possesses ‘a dominant market share’ in general search: in the US, 89.2 per cent overall, ‘which increases to 94.9 per cent on mobile devices’. Unsurprisingly, that dominance was mirrored in the market for the accompanying advertising: Google’s share of what the court calls ‘the text ads market’ was 88 per cent in 2020.

Google, the court ruled, ‘is a monopolist’. It ‘has violated section 2 of the Sherman Act’ – since 1890, the cornerstone of US competition law – ‘by maintaining its monopoly in two product markets in the United States: general search services and general text advertising’. Google has done this, the court concluded, via ‘exclusive distribution agreements’ which secure the presence of Google’s ‘search widget’ on the home screen of Android phones and its position as the default search engine for Apple’s Safari and Mozilla’s Firefox browsers. In return, Google shares its search-advertising revenues with Apple, the Mozilla Foundation and phone manufacturers. The longest-standing agreement, dating back more than twenty years, is with Apple. It is also the most important agreement, since Apple devices account for more than half of general search queries in the United States; the court estimated that in 2022 the deal brought Apple around $20 billion. The exact percentage of revenue that goes to Apple is redacted in the court documents, but an expert witness for Google is reported to have said in open court that it is 36 per cent.

The world of giant digital platforms often turns on surprisingly small matters, such as whether or not users are willing to spend a few seconds doing something they don’t absolutely need to do. Google’s being the default search engine doesn’t stop you from using a different one. Apple, for example, makes it perfectly easy to switch. It takes maybe twenty seconds to open the settings on your laptop (or your iPhone) and change the default to, say, Microsoft’s Bing. From then on, if a query you enter into Safari’s address bar isn’t answered directly by Apple’s systems, it will go to Bing, not Google, and the revenue from the associated ads will flow to Microsoft – in which case, as far as I am aware, no revenue will go to Apple. But you’ve probably never tried switching search engines. That might be because you positively want to use Google, but perhaps like me (and, I fear, most people) you are a lazy sod and tend to stick with the digital world’s preloaded default options.

The second competition law case, heard by a federal court in Virginia, is more esoteric in that it concerns the inner workings of digital advertising. Two systems are at issue. The first is Google’s ‘ad server’. This is a cloud service that Google sells to publishers (not just news publishers but providers of online content of all kinds); it takes the final decision about which ads to show users when they visit the publisher’s website. The second is Google’s ad exchange, AdX, which conducts ad trading in real time. Visit the Guardian’s website, for example, and the opportunity to advertise to you is usually auctioned on AdX and similar, smaller exchanges.*

The Virginia court found that Google’s ad server is responsible for roughly 90 per cent of open-web display ads globally, and that AdX has a 63-71 per cent share of the corresponding ad trading, ‘nine times that of its next closest rival’. The court conceded that the way Google has acted is in some respects quite different from a traditional monopolist: for example, it has not raised the fees it charges publishers for use of its ad server. It concluded, nevertheless, that Google has acted to preserve its structural centrality to the ad server and ad exchange markets, thus ‘acquiring and maintaining monopoly power’ in ways that violate US competition law.

The appeals process began in the summer, and Google’s lawyers will, no doubt, continue to argue, among other things, that Google has default position because it is the best search engine. But if Google is the best, it may be at least in part because it has the most users. Algorithms such as PageRank are far from the only thing that determines search quality. There’s a great deal that can be learned from such simple things as whether users immediately return to the search results page after clicking a link on that page, which suggests the link didn’t provide what they were looking for. The more users a search engine has, the more data of this kind its engineers can analyse in order to improve it.

In the background to both cases is a notoriously contested issue in US competition law: defining the relevant market. If, for instance, it is ‘general search’ of the kind conducted using Google, then Google does indeed have a very large market share. But expand the definition to include digital searches of all kinds, and Google’s share diminishes, with Amazon, in particular, emerging as a powerful rival. I would expect issues of market definition to be prominent in the appeals.

The effects on Google if it loses these appeals may well not be seriously damaging. The District of Columbia court has rejected dramatic options such as forcing Google to sell Chrome or Android, opting instead to insist that Google shares search data to help improve any potential rival search engines. Google is allowed to continue its revenue-sharing agreements, though these can’t be exclusive: they can’t, for example, prevent other search engines being preloaded as well. None of this is likely to cause Google to lose very much market share, if only because users, when offered an explicit choice between search engines, may well opt for the one with which they are familiar (to the degree that Google’s name has become the verb most of us use for search in general). Mozilla, for instance, has repeatedly experimented with changing the default search engine in Firefox, and found that a large proportion of its users switched back to Google, often straightaway.

The Virginia court hasn’t yet said what it is going to demand. It may instruct Google to divest itself of its ad server and AdX, its ad exchange. The details of the way these technologies work are hugely important to publishers’ income (and therefore to the future of journalism), but the money they bring in to Google isn’t, by its standards, huge. What Alphabet calls ‘Google Network’, the kind of advertising for which the ad server and AdX are the infrastructure, accounted for less than 9 per cent of Alphabet’s revenue in 2024, and it could certainly survive having to sell the two systems.

Aquite​ different, and potentially more serious, threat to Google is a development that it did a great deal to foster: the emergence of large language models (LLMs) and the chatbots based on them, most prominently ChatGPT, developed by the start-up OpenAI. Google’s researchers have worked for more than twenty years on what a computer scientist would call ‘natural language processing’ – Google Translate, for example, dates from 2006 – and Google was one of the pioneers in applying neural networks to the task. These are computational structures (now often gigantic) that were originally thought to be loosely analogous to the brain’s array of neurons. They are not programmed in detail by their human developers: they learn from examples – these days, in many cases, billions of examples.

The efficiency with which a neural network learns is strongly affected by its structure or ‘architecture’. A pervasive issue in natural language processing, for example, is what linguists call ‘coreference resolution’. Take the sentence: ‘The animal didn’t cross the street because it was too tired.’ The ‘it’ could refer to the animal or to the street. Humans are called on to resolve such ambiguities all the time, and if the process takes conscious thought, it’s often a sign that what you’re reading is badly written. Coreference resolution is, however, a much harder problem for a computer system, even a sophisticated neural network.

In August 2017, a machine-learning researcher called Jakob Uszkoreit uploaded to Google’s research blog a post about a new architecture for neural networks that he and his colleagues called the Transformer. Neural networks were by then already powering Google Translate, but still made mistakes – in coreference resolution, for example, which can become embarrassingly evident when English is translated into a gendered language such as French. Uszkoreit’s example was the sentence I have just quoted. ‘L’animal’ is masculine and ‘la rue’ feminine, so the correct translation should end ‘il était trop fatigué,’ but Google Translate was still rendering it as ‘elle était trop fatiguée,’ presumably because in the sentence’s word order ‘street’ is closer than ‘animal’ to the word ‘it’.

The Transformer, Uszkoreit reported, was much less likely to make this sort of mistake, because it ‘directly models relationships between all words in a sentence, regardless of their respective position’. Before this, the general view had been that complex tasks such as coreference resolution require a network architecture with a complicated structure. The Transformer was structurally simpler, ‘dispensing with recurrence and convolutions entirely’, as Uszkoreit and seven current or former Google colleagues put it in a paper from 2017. Because of its simplicity, the Transformer was ‘more parallelisable’ than earlier architectures. Using it made it easier to divide language processing into computational subtasks that could run simultaneously, rather than one after the other.

Just as Dean and Ghemawat had done, the authors of the Transformer paper made it publicly available, at Neural Information Processing Systems, AI’s leading annual meeting, in 2017. One of those who read it was the computer scientist Ilya Sutskever, co-founder of OpenAI, who says that ‘as soon as the paper came out, literally the next day, it was clear to me, to us, that transformers address the limitations’ of the more complex neural-network architecture OpenAI had been using for language processing. The Transformer, in other words, should scale. As Karen Hao reports in Empire of AI, Sutskever started ‘evangelising’ for it within OpenAI, but met with some scepticism: ‘It felt like a wack idea,’ one of his OpenAI colleagues told Hao.† Crucially, however, another colleague, Alec Radford, ‘began hacking away on his laptop, often late into the night, to scale Transformers just a little and observe what happened’.

Sutskever was right: the Transformer architecture did scale. It made genuinely large, indeed giant, language models feasible. Its parallelisability meant that it could readily be implemented on graphics chips, originally designed primarily for rendering images in computer games, a task that has to be done very fast but is also highly parallelisable. (Nvidia, the leading designer of graphics chips, provides much of the material foundation of LLMs, making it the world’s most valuable company, currently worth around 30 per cent more than Alphabet.) If you have enough suitable chips, you can do a huge amount of what’s called ‘pre-training’ of a Transformer model ‘generatively’, without direct human input. This involves feeding the model huge bodies of text, usually scraped from the internet, getting the model to generate what it thinks will be the next word in each piece of text, then the word after that and so on, and having it continuously and automatically adjust its billions of parameters to improve its predictions. Only once you have done enough pre-training do you start fine-tuning the model to perform more specific tasks.

It was OpenAI, not Google, that made the most decisive use of the Transformer. Its debt is right there in the name: OpenAI’s evolving LLMs are all called GPT, or Generative Pre-trained Transformer. GPT-1 and GPT-2 weren’t hugely impressive; the breakthrough came in 2020 with the much larger GPT-3. It didn’t yet take the form of a chatbot that laypeople could use – ChatGPT was released only in November 2022 – but developers in firms other than OpenAI were given access to GPT-3 from June 2020, and found that it went well beyond previous systems in its capacity to produce large quantities of text (and computer code) that was hard to distinguish from something that a well-informed human being might write.

GPT-3’s success intensified the enthusiasm for LLMs that had already been growing at other tech firms, but it also caused unease. Timnit Gebru, co-founder of Black in AI and co-head of Google’s Ethical AI team, along with Emily Bender, a computational linguist at the University of Washington, and five co-authors, some of whom had to remain anonymous, wrote what has become the most famous critique of LLMs. They argued that LLMs don’t really understand language. Instead, they wrote, an LLM is a ‘stochastic parrot … haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning’. What’s more, Bender, Gebru and colleagues noted, training such a model consumes huge quantities of electricity, and the giant datasets used in the training often ‘encode hegemonic views that are harmful to marginalised populations’. (They quoted the computer scientists Abeba Birhane and Vinay Uday Prabhu: ‘Feeding AI systems on the world’s beauty, ugliness and cruelty but expecting it to reflect only the beauty is a fantasy’.)

Gebru sought permission to publish the paper, but senior figures at Google objected. She was asked either to retract it or remove the names and affiliation of any of its authors who worked for Google. But she was prepared to do so only under conditions that were unacceptable to her superiors (she had wanted them to name the individuals at Google to whom the paper was sent for review), and ended up losing her job. Nearly 2700 Google employees signed a letter of protest.

There has been much speculation about the reason it was OpenAI, not Google, that was first to turn the Transformer architecture into a truly successful chatbot. One reason, ironically, seems to have been that some of the concerns expressed in the ‘stochastic parrot’ paper were shared by senior executives at Google. In 2016, Microsoft had launched a Twitter chatbot, Tay, which was designed to interact with and learn from human users’ tweets. It picked up the world’s ugliness remarkably rapidly. A number of Twitter users deliberately fed Tay racist content, successfully training it to be an automated fascist (one user asked it, ‘Do you support genocide?’, to which it responded: ‘I do indeed’). Microsoft was forced to withdraw it within 24 hours of its release. Google’s executives evidently had no desire to make the same mistake. Its researchers developed a Transformer-based chatbot called Meena (later renamed LaMDA, Language Model for Dialogue Applications) but did not get permission to release it. A Google spokesperson later told the Wall Street Journal that ‘the chatbot had been through many reviews and barred from wider releases for various reasons over the years.’

A start-up such as OpenAI has to take risks, but major corporations often focus on improving well-established products or services rather than innovating more radically. It was far from predictable that a sophisticated chatbot would become globally famous almost overnight; even OpenAI itself was taken aback by the rapidity of ChatGPT’s uptake. One of the authors of the Transformer paper, Niki Parmar, told the Financial Times that Google’s preference was ‘to optimise for the existing products’. While the Transformer was used to improve Google Translate and Google Search, only after ChatGPT’s success did Google throw its huge resources into an all-out effort to launch a chatbot, Bard (now called Gemini), with eighty thousand members of staff donating their time to test it.

The successes of LLMs have changed the digital world. For anyone who, like me, teaches a subject in which students are no longer assessed primarily by means of traditional exams, the most pressing concern is these models’ ability to generate essays that read like the work of a reasonably proficient if intellectually unambitious student. A more disturbing thought is that the models’ capacity to do this may have revealed that something is wrong with our pedagogy. Have we been teaching students to be stochastic parrots?

For Google, the big worry is what will happen to search in the long term. Most search queries can easily be rephrased as a prompt for a chatbot, and that is a clear threat to what has been, for a quarter of a century, Google’s most important, and still very healthy, source of revenue. Several AI companies are developing automated purchasing assistants on top of LLMs, and I am already starting to read articles in the trade press about how to market products to such assistants rather than directly to human beings. Google itself is working on an assistant of this sort, called Project Mariner, but if they are adopted by the public they will reduce the demand for search ads, and in that reconfigured form of electronic commerce Google would not enjoy incumbency advantages of the kind the antitrust litigation focused on.

I’m starting to feel some pre-emptive nostalgia when I do a Google search. Yes, it’s true, search can sometimes take you to places you don’t want to go. In 2010, when the information science scholar Safiya Umoja Noble was using the search term ‘black girls’ to Google ‘things on the internet that might be interesting to my stepdaughter and nieces’, the top hit presented to her was ‘HotBlackPussy’. As she argued in Algorithms of Oppression (2018), search engines can echo and reinforce a racist and sexist culture. But at least a ‘classical’ search engine like Google in the 2000s and 2010s took you outside itself, and perhaps implicitly prompted you to evaluate critically what you found there.

The experience of using a chatbot powered by an LLM is, by contrast, largely self-contained. You can usually prompt it to say something about its sources, but that’s a bit like the ‘further reading’ at the end of a textbook chapter: you know you should read them, but you probably won’t. It’s seductively easy to treat a chatbot as an oracle. That’s economically as well as cognitively dangerous. Classical search creates economic incentives for making new content available on the web, and keeping existing content up to date, because money can be made by advertising to visitors brought to websites by a search engine. Of course the process can also incentivise clickbait, but if search atrophies what will happen to the web? As the commentator Eric Seufert nicely puts it, Google was the open web’s ‘imperfect benefactor’.

There is a road that must be crossed, by Google and also by the rest of us. On one side is a digital world with its largely familiar structure, fuelled by familiar kinds of advertising, in many of whose forms Google excels. To make liveable whatever lies on the other side of the road requires that we confront some challenges. One is to avoid an overreliance on self-contained systems trained on whatever text, speech, images, audio and video are digitally available – inevitably both a biased subset of what human beings are able to produce and an impoverished version of what human knowledge consists in. Another challenge is to unlearn scale. Much of the success of LLMs has come simply from making everything bigger: the number of parameters they contain, the quantity of data employed to train them, the size and energy intensity of the data centres in which they run. That trajectory is unsustainable, and not just environmentally: it’s getting harder to find adequate volumes of fresh data on which to train new models, since much of what exists is already potentially compromised by having been generated by previous models. It’s going to be a hard road to cross. Can we navigate it successfully? Can Google? If it turns out to be too tired to make it, I’ll be a little sad.