Blink, Bid, Buy

Click​ on a link to an article on a news website. If you have a fast internet connection, you’ll see the article almost immediately, but the slots for adverts will usually remain empty for another second or two. The ads which then appear aren’t generally chosen in advance. Instead, a near instantaneous, automated auction of each slot goes on behind the scenes to determine which ad you will be shown. Your phone or laptop may itself be gathering bids for the auction. Sometimes, an enigmatic pattern or grey rectangle will appear instead of an ad.

What goes on during those few seconds is vital to the economics of journalism and is the subject of sharp, subterranean conflict among news publishers, the big online platforms (especially Google), and independent providers of ‘AdTech’, the technology of digital advertising. Sales of print newspapers have been falling for years, and revenues from online subscriptions alone are generally insufficient to support large-scale news reporting. So advertising income matters. British newspapers were making £4.6 billion a year from print advertising as recently as 2007, but by 2017, according to the consultancy Mediatique, earnings had fallen to £1.4 billion. Online advertising, by contrast, has boomed, but the revenue has largely bypassed news publishers. Less than 5 per cent of online advertising money in Britain is spent on ads that appear on newspaper websites, and the resulting revenue for newspapers (£487 million in 2017, Mediatique says) falls well short of making up for the decline in print advertising.

One way for an advertiser or advertising agency to get their ads onto a news website is by striking a direct deal with the site’s publisher, at a fixed price per thousand ‘impressions’ (each time an ad is shown to a viewer counts as one impression). But an advertiser can also buy ad slots, often far more cheaply, in what they call ‘open auctions’ or the ‘open marketplace’. There, demand from multiple advertisers is brought together electronically with supply from any website whose content carries ad slots (news publishers are distinctly in the minority in the open marketplace). Each ad slot is auctioned separately, and its price is influenced not just by the quality of the website – which often matters less than one might hope – or by the nature of the content surrounding the slot, but by what, if anything, is known about the user to whom the ad will be shown in a couple of seconds’ time.

Prices in these auctions are usually very low: the winning bid for an ad slot will normally be no more than a fraction of a penny. But fractions of pennies add up when earned many, many times over. I’m told, for example, that the Guardian gets about a billion online ad impressions every month, which brings in £50-£70 million a year, somewhere between a fifth and a third of the paper’s total revenue. Without that, its newsroom would be much depleted.

The biggest host of instantaneous auctions by volume is Google’s ad exchange, but news publishers and other big providers of online content also bring their supply of ad slots to market through AdTech systems known as ‘supply-side platforms’ or SSPs. Each SSP pumps out to potential buyers a ‘bid stream’ of individual, momentary opportunities to show an ad in a specific slot on a particular site. These bid streams amount to a Niagara of ad slots, bought and sold via what participants call the ‘programmatic pipes’: a global digital infrastructure of datacentres, AdTech platforms and fibre-optic cables.

To process the torrent flowing through the pipes and assess each and every ad slot – whether it is worth bidding for, and if so how much it is worth – is very demanding. Most advertisers and ad agencies, instead of building a system to do it themselves, contract out the job to a ‘demand-side platform’. A DSP might have to process tens of millions of ad opportunities a second, and will have only about a fifth of a second to respond to each opportunity: if it takes longer than that, it is simply timed out of the auction for that slot.

The programmatic pipes are technologically impressive but economically opaque, leaking money in ways that are hard to trace. In 2016, the Guardian experimented by bidding for ad slots on its own website, and discovered that sometimes only 30 per cent of what it paid found its way back to the newspaper. A couple of years later, a team from PricewaterhouseCoopers was commissioned by the Incorporated Society of British Advertisers to follow the money through the pipework more systematically. The team worked with fifteen huge advertisers – including Nestlé, Unilever, Vodafone and British Airways – whose aggregate annual expenditure on online advertising comes to about £100 million a year in the UK. Joining the study were twelve newspaper or magazine groups, at the other end of the pipework. The PwC team tried to match ad impressions the advertisers bought with the ad slots the publishers sold. Negotiating access to data held by intermediaries such as SSPs and DSPs proved complicated – the study ended up taking fifteen months – and difficulties such as differences in data format often made matching impossible.

For the 31 million impressions that the PwC team was able to match end to end, it found that on average only 51 per cent of what the advertisers spent reached the publishers. Two-thirds of the remaining 49 per cent was absorbed by intermediaries’ fees. (On average, 7 per cent of what advertisers spent went to advertising agencies, and a further 8 per cent to DSPs. Supply-side platforms charge their publisher clients a similar amount. Most transactions involve both a DSP and an SSP, and very probably other vendors of data and technical services.) But PwC couldn’t find where almost a third of the 49 per cent – that’s around 15 per cent of advertisers’ spending – had ended up.

Given all this, it isn’t surprising that publishers in the UK and elsewhere have been making concerted efforts to re-engineer the pipework. Their first big initiative was to make the reader’s device (phone, tablet, laptop or desktop) more central to the process of ad selection. Previously, as soon as a reader accessed a publisher’s webpage, its ad server would immediately be informed that ad slots were about to become available. The ad server is the system that decides which advertisements to show you: it weighs up, for example, the value of the highest bid from the auction against the demands of the publisher’s direct deals. These days, it isn’t a machine that sits in a publisher’s computer room, but a cloud service, in English-language news publishing almost always provided by Google. (UK news publishers often worry that advertising revenue they should be getting is instead going to Google. Distrust of Google and resentment of its dominant position are common. But Google also supplies much of the traffic to newspaper websites, and its ad server is integrated deeply into publishers’ systems, which makes moving to a different server difficult. One publisher told me that even though his parent company part-owned one of the handful of other ad server suppliers, shifting to it would have been risky and very disruptive.)

Turning your phone into an auctioneer enables news publishers and other websites to reduce their dependence on Google without turning their backs either on Google’s ad server or on the demand for ad slots that flows through Google’s systems. What a publisher does is to add a snippet of computer code to its webpage ‘headers’; the code instructs your browser, whenever it accesses a page, to send electronic messages to around half a dozen SSPs, inviting them to submit bids for the right to show adverts to you. (The header is the invisible initial part of a webpage which, among other things, tells your browser how to format the page content.)

Only once your browser has received the SSPs’ bids does it alert the ad server, meaning that if Google’s systems are to win the auction they have to improve on the highest bid your browser has been able to obtain. For publishers, another attraction of header bidding is that the SSPs are placed in direct competition with one another: a publisher told me he hoped this would trim what he called the ‘fat’ in SSPs’ fees.

Google, conversely, is not enthusiastic about header bidding, though it has gradually adjusted its systems to accommodate it. The issue is central to an anti-trust lawsuit against Google launched by a consortium of US states with largely Republican attorneys general, who argue that the company ‘schemed about how to quash header bidding’. Google rejects the charge. ‘We do not participate in header bidding … for good reason,’ it told the UK Competition and Markets Authority in 2020. Header bidding, it argued, ‘is characterised by increased latency [electronic delays], reduced transparency and significant user trust and privacy concerns’.

Publishers themselves discovered that the original form of header bidding, in which your device is central to the auction, can have some of the disadvantages that Google cites. The time it takes your browser to establish electronic links to SSPs can cause ads to load slowly, and in the process data is flowing from your phone. In particular, if an SSP has previously deposited a cookie (a small string of letters and numbers that identifies your browser) on your device, it can retrieve that cookie, and so learn something about you. The availability of that information increases the value of the bids your browser receives, but also means that the publisher loses a degree of control over data about its readership.

Increased control is what news publishers often say they want when you ask them about the changes they would like to see in the way their ad slots are auctioned. Many of them think they can gain a greater degree of control through ‘server-side’ header bidding, a different form of the technique whereby bids are gathered from SSPs not by your phone or laptop, but by a computer server designated by the publisher. There is no longer a direct link between the SSPs and your device, which returns to a more passive role. For a publisher wishing to implement header bidding itself, the server-side version of the technique is, however, more complex than its original version. A manager familiar with its challenges tells me that it can require around ten person-years of highly specialist technical work – ‘pretty heavy lifting for an individual publisher’.

For this reason among others, the UK’s news publishers have set up their own collective AdTech development effort, the Ozone Project. It brings together organisations with quite different political orientations, such as the Guardian, the Telegraph and News UK, publisher of the Times and the Sun. The biggest obstacle to the collaboration, an AdTech specialist who works for right of centre titles told me, was long-standing ‘emotional attachments … Once you let go of that, and understand that your true competitors are the Facebooks, the Googles, the Amazons … that’s when it becomes easier.’ The Ozone Project does the heavy lifting of setting up server-side header bidding, and provides advertisers with a more direct, shorter, less leaky programmatic pipe through which to bid for ad slots.

Leaks​ from the pipes aren’t the only way in which news sites lose potential advertising revenue. Another is ‘ad-blocking’, in which a news story or other sort of online material is judged, usually by an automated system, to be unsuitable for advertisers, either in general – because, say, the story reports on distressing events or controversial issues – or for reasons that are specific to the product or service being advertised: no tanning salon wants its ads to appear alongside an article about skin cancer. (Ad-blocking in this context is quite different from the software that users themselves install, which tries to prevent any ads appearing at all.)

Newspapers usually arrange their content into broad categories: ‘news’, ‘politics’, ‘sport’, ‘travel’. Some advertisers simply don’t allow their adverts to be shown alongside ‘news’. In November last year, for example, the editorial staff of one major UK newspaper quite reasonably chose to classify its reporting of COP26 not under ‘environment’ but as ‘news’ or ‘politics’. As a result, advertising alongside those stories dried up almost completely. The ad-blocking of ‘news’ is particularly widespread at the moment, I’m told, because of the extensive coverage of the war in Ukraine. While it’s easy to see why advertisers don’t want their ads appearing alongside reports on atrocities, they seem to be underestimating responsible news publishers’ efforts to avoid this. One publisher tells me that in the early weeks of the invasion it entirely removed its homepage masthead ad slot, because of the news content likely to appear beneath it. Another said: ‘We’re blocking ads ourselves against stories with images of civilian casualties.’

Big advertisers, though, don’t rely on publishers’ own classifications of articles. Because they place ads in a wide range of contexts online, they have learned that it’s wise to employ an AdTech firm that specialises in ‘verification’. These firms check that ads are displayed for long enough to count as viewable, and that the viewer is actually a human being, not a programmed ‘bot’ that simulates clicks on ads on a website designed to earn money from unwary advertisers. Verification firms now also do the job of trying to stop ads appearing alongside content that advertisers consider unsuitable. A firm’s system will scrutinise the giant flood of ad slots as it flows through the programmatic pipes, and if it judges a slot to be unsuitable, will prevent the DSP that acts on the advertiser’s behalf from bidding for it.

This form of ad-blocking doesn’t always work perfectly, though, so the verification firm’s system will often do another check just before an ad is displayed. (The time this checking takes can be another reason ads are often slow to load.) If the final check suggests a probable violation of brand safety or suitability, and if it’s too late for the publisher’s ad server to find a different ad, the system provides a stock neutral image to fill the slot. One such image, a pattern evoking white clouds against a blue sky, became briefly famous at the start of the pandemic, when it was spotted in the masthead ad slots on the homepages of the New York Times and Wall Street Journal, normally prime real estate for online advertising.

A plausible rough estimate is that UK news publishers lost £50 million in the early months of the pandemic because of ad-blocking of their stories about it; in April 2020 the culture secretary, Oliver Dowden, appealed to advertisers both publicly and privately to relax the blocking. A year later, though, articles were still being ad-blocked even when they were connected to the pandemic only indirectly – a discussion of working from home, say. The Black Lives Matter protests in 2020 also seem to have been the subject of widespread ad-blocking. Vice reports that its coverage in this area attracted less than half the advertising revenue it would have expected.

How does an automated system judge the suitability for advertisements of a news story or anything else on the web? One way, crude but still surprisingly common, is to use a ‘blocklist’: a list of keywords – several hundred, I’m told, or perhaps even several thousand – associated with content alongside which their ads must not appear. Blocklists are kept confidential, but it seems likely that the ones used in 2020 contained terms such as ‘coronavirus’, ‘pandemic’, ‘lockdown’ etc.

In June 2017, however, the verification provider Integral Ad Science did publish a list of the twenty most commonly blocked terms at the time. First was ‘explosion’, then ‘terror’. Others included ‘dead’, ‘shooting’, ‘gun’ and ‘kill’. Domestic terrorism seems to have first prompted the inclusion of these words on blocklists, but today the same words are probably ad-blocking reporting of the war in Ukraine. Publishers also say that a wide range of articles can be penalised accidentally: many things explode, metaphorically at least; footballers shoot; celebrities ‘dress to kill’. And some of the words that appear on blocklists are surprising. Scott Gatz, of the LGBTQ-oriented US electronic publisher Q.Digital, tells me that one big advertiser entered into a direct deal with it to advertise alongside its Pride coverage, but no ads then appeared on Q.Digital’s websites. He asked to see the blocklist that the advertiser had given to its verification provider – on it were ‘lesbian’, ‘gay’, ‘bisexual’, ‘transgender’ and ‘queer’. ‘Muslim’, too, seems to be common on blocklists, as is ‘Trump’; ‘politics’ is sometimes a blocked word as well as a category to be avoided. Inertia may be part of the reason some words linger on blocklists: several people have told me they suspect that the lists are seldom pruned. Certainly, Gatz finds that the staff in an advertiser or advertising agency who design campaigns usually don’t have responsibility for its blocklist and, as in the Pride case, may not even be aware of its contents.

No one I’ve talked to about blocklists really likes them; the verification firms, in particular, urge advertisers to use the more sophisticated services they offer. These involve the wide-ranging use of ‘crawlers’: programs that systematically digest the content of all the webpages on which their clients are likely to bid for ad slots, sometimes also analysing the links from those pages to other sites. The firms’ systems use artificial intelligence techniques to classify all this web content, ‘reading … the adjectives as well as the nouns’, as Richard Reeves of the Association of Online Publishers puts it. Crawling ranges far beyond news websites, but these sites are updated continuously, so are crawled intensively: one publisher estimates that every new addition to his paper’s site will be crawled and assessed for brand safety and suitability within fifteen minutes to an hour.

This giant-scale algorithmic reading of web content already seems to have had a considerable influence on the flow of advertising money. While many regard it as better than the mechanical use of blocklists, publishers aren’t always happy with the results. Gatz, for example, says that Q.Digital’s webpages are often classed as ‘adult’, a category that includes sexually explicit material and is often avoided by advertisers. Nor have AI-based techniques entirely displaced blocklists. An executive at one advertising agency warns that you can ‘lose a client’ by inadvertently advertising alongside content the client regards as unsuitable. A manager at another agency says that she personally is prepared to advertise without a blocklist, but understands why others won’t: ‘It’s so hard from having so many safety nets to going, we are just going to remove this … It’s like bungee jumping and going, we are going to remove the net below you.’

If,​ however, you are trying to identify the single most significant cause of journalism’s current economic difficulties, one publisher has said to me, it might be the third-party cookie. This is a cookie deposited in your browser not by the website you are visiting but by a different system, operated perhaps by a supply-side or demand-side platform. As the sociologists Thomas Beauvisage and Kevin Mellet point out, these cookies form an infrastructure that permits an individual – or their browser at least – to be identified in multiple online contexts and by multiple organisations. (There are giant matching tables that can, for example, translate the cookie by which you are known to an SSP into the cookie used by a DSP.) Why advertise to a user through an expensive direct deal with a high-quality news website when you can reach them with the same ad more cheaply on other sites they visit, ones that rarely change or have content that can be produced without paying professional journalists?

But third-party cookies are on the way out: by late 2023, all major browsers will block them by default.* Users can still be identified in other ways, but no framework for doing this has been adopted sufficiently widely to form an equivalent infrastructure, and regulators may try to ensure that such an infrastructure doesn’t come about. In November, Elizabeth Denham, then the UK’s information commissioner, warned that her office, which has long-standing concerns about the sharing of user data in ad auctions, ‘will not accept proposals based on underlying AdTech concepts that replicate or seek to maintain the status quo’.

There is therefore a revival of interest in ‘contextual targeting’: advertising focused not on the identity of the user but on the content with which the user is interacting at any given moment. News websites, with their rich, diverse and ever-changing content, fit the bill nicely. That’s grounds for a little optimism, for no one has a deeper and more up to date understanding of the content of a news website than its publisher. A verification firm, though, with its active crawlers and AI-based analytical tools, may hope to come a close second, and several such firms have now added contextual targeting to the services they offer advertisers. The publishers I have spoken to believe it is they, not AdTech firms, who should be giving advertisers the information they need for effective contextual targeting. One publisher even speculated about denying access to AdTech’s crawlers. ‘If you’re a Times, Telegraph or an FT, you can probably do that,’ he says, but ‘if you’re a Mirror, a Mail or a Sun, then it’s going to be harder.’ If a news site is, as he put it, a ‘mass-market proposition’, then it will of necessity depend on open auctions, and advertisers need the reassurance of the automated assessment of brand-safety risks.

There is a growing sense among advertising practitioners that they need to take responsibility not just for the content of adverts, or even for brand safety as narrowly conceived, but for the wider impact of ad spending. As the Conscious Advertising Network, an influential UK-based pressure group, points out, this means not only making sure you aren’t funding hate speech or misinformation, even inadvertently, but also not relying on the mechanical use of blocklists. One current concern of the group’s co-founder, Harriet Kingaby, is blocklists that include words related to climate change.

If you’re a big advertiser, to advertise ‘consciously’ almost certainly means being more selective. On average, the firms that took part in the PwC study each advertised, in just three months, on more than forty thousand websites in the UK alone. That’s advertising on a scale that defies genuine human oversight. ‘Conscious’ ad spending must surely involve supporting activities that contribute to human betterment. Much of what we know about the world we have to learn from others, including journalists, who must draw on their experience to distinguish unfounded, attention-seeking speculation from frequently mundane facts, and exercise the courage to pursue the truth even when it is awkward, discomfiting and dangerous to do so. Intermediaries’ excessive fees damage the economic base of this crucially important activity, just as indiscriminate ad-blocking of ‘news’ disincentivises its production.

It is encouraging that the Guardian’s subscribers and ‘supporters’ (who make regular donations) now number more than a million. The Financial Times also has more than a million digital subscribers and the New York Times more than ten million. But a handful of successful titles isn’t enough. Local and regional newspapers can still make a big contribution to their communities – local, not just national, decision-makers must be held to account – but have suffered especially severely from declines in circulation and advertising revenues. And then there are outlets that specialise in reaching the very audiences whose sexual orientation, or ethnic, religious or gender identity are associated with words that too often populate blocklists. Such outlets are worth supporting in themselves, of course, but they also offer a way for advertisers to broaden their reach. In the end, however, advertising revenue alone will seldom be enough to support extensive in-depth journalism. That will require diverse sources of income, above all from sales and subscriptions, but also from membership schemes, events, philanthropy and in some cases – local journalism included – even central government support.