404bot ads.txt cover

404bot: The Ad Fraud Scheme Exploiting ads.txt

Sam Carr

Sam Carr

March 19th, 2020

Sam obsesses over three things in life: Animals, Films and Marketing. We may have over exaggerated his love for marketing.

ppc protect webinar

It’s only a few years ago that ads.txt was introduced to help advertisers keep track of their ad inventory. At the time it was seen as a brilliant new way to tackle ad fraud and keep track of authorised resellers.

However, less than 3 years on, not everything has gone to plan.

A new type of ad fraud involving the ads.txt file has recently been detected and is costing advertisers precious money. Known as 404bot, this scheme has been around since 2018 and has been growing year on year at a steady rate.

If you’re currently using an ads.txt file on your website, then this is something you definitely should be aware of.

But before we dive into what the 404bot is and how it works, let’s take a quick recap on what the ads.txt file is and what it does.

What Are Ads.txt Files?

what are ads.txt files

Released back in 2017, the ads.txt file or authorised digital sellers file is a public list of authorised resellers who are authorised to resell a specific website’s ad inventory. Introduced by the Interactive Advertising Burea (IAB), the aim of the file is to help combat the growing problem of domain spoofing in programmatic advertising.

Prior to the introduction of the ads.txt file, many large scale fraud rings such as Methbot and Hyphbot were taking advantage of this lack of authorisation. By spoofing domains, these fraud rings were able to scam advertisers out of millions every single year.

Since the introduction of the ads.txt file, there has been a significant drop off in fraudulent activity amongst programmatic advertising. But recently, some people have found there is a little trick to take advantage of the file, which has been used to set up the ad fraud scheme.

So how exactly is a file that was introduced to stop fraud, now being used to create fraud? Here’s how it all works.

How The 404bot Is Defrauding Advertisers

The idea behind the ads.txt file is that it helps prove who is authorised to sell a website’s ad inventory. This is turn helps avoid sellers who arbitrage inventory and spoof domains which results in the advertiser being defrauded.

Since the launch of the file, many fraudsters have begun exploiting the fact that many buyers don’t check this ads.txt file beforehand. But it’s not just advertisers, many publishers are also guilty of not regularly auditing and verifying their own files sometimes leaving on websites and vendors they no longer work with.

This is where the 404bot comes in. Just like before the ads.txt file came into circulation, the fraudsters are still spoofing domains and trying to impersonate other publishers websites. However, in this ad fraud scheme, there is no inventory that exists. Instead, there is a 404 error page (hence the 404bot name). To the advertiser, it looks like an authorised seller, when in fact, it’s not.

The 404bot was first spotted back in 2018 by Integral Ad Science and has since defrauded advertisers out of more than $15 million worldwide. How this has been made possible is mainly down to the lack of auditing and updating of these files.

Many publishers often work with a range of vendors and don’t always update their ads.txt file when they stop working with them. This means they’ll continue to be listed on the file even though they aren’t an authorised seller, making them the perfect target for fraudsters. 

But publishers aren’t the only ones to blame, many advertisers are also not reviewing and auditing these ads.txt files before purchasing as it could theoretically reduce the number of ads that they could buy. Not wanting to miss out on the extra placements, they often turn a blind eye and just “hope for the best”.

Another reason why this type of fraud has increased is also due to the difficulty of auditing files. With some ads.txt files having thousands of lines, just the sight of these files can put anyone off auditing them.

How To Protect Yourself From 404bot

The 404bot is most definitely something to look out for if you’re an advertiser looking to buy ads on websites. But how exactly can you protect yourself from wasting your precious ad spend on 404 pages?

Well, the unsurprising answer is to carefully audit the ads.txt files before spending any money. During their research, Integral Ad Science noted that when publishers and advertisers worked together to verify their thousands of lines of ads.txt file, the instances of 404bot weren’t as common.

This means as a publisher regularly checking and updating your ads.txt file to remove and out of date vendors. For advertisers, it means checking every line in the ads.txt file and asking the publisher for proof that they are authorised. This might seems like it kind of defeats the point in having an ads.txt file in the first place. But if publishers actually updated their ads.txt file regularly and audited it then we wouldn’t have this problem!

Now you know how to protect yourself from this sneaky ad fraud scheme, be sure to regularly check any ads.txt file you’re dealing with. You never know who might be hiding in those thousands of lines.

2020's Must Have Tool For Google Ads Users

Do you run Google Ads campaigns? We'll increase your campaign performance by up to 30% through eliminating click fraud. Save time and money with the world's most advanced click fraud prevention technology for Google Ads campaigns.

Improve PPC results for free

Full Featured Trial.

Eliminate clicks from bots, competitors & more.

Boost PPC performance by +30%.