Perplexity accused of scraping websites that explicitly blocked AI scraping

[highplainsdem]

Source: TechCrunch

AI startup Perplexity is crawling and scraping content from websites that have explicitly indicated they don’t want to be scraped, according to internet infrastructure provider Cloudflare.

On Monday, Cloudflare published research saying it observed the AI startup ignore blocks and hide its crawling and scraping activities. The network infrastructure giant accused Perplexity of obscuring its identity when trying to scrape web pages “in an attempt to circumvent the website’s preferences,” Cloudflare’s researchers wrote.

-snip-

Perplexity appears to be willingly circumventing these blocks by changing its bots “user agent,” meaning a signal that identifies a website visitor by their device and version type; as well as changing their autonomous system networks, or ASN, essentially a number that identifies large networks on the internet, according to Cloudflare.

“This activity was observed across tens of thousands of domains and millions of requests per day. We were able to fingerprint this crawler using a combination of machine learning and network signals,” read Cloudflare’s post.

-snip-

Read more: https://techcrunch.com/2025/08/04/perplexity-accused-of-scraping-websites-that-explicitly-blocked-ai-scraping/

---

Cloudflare also said Perplexity has been using "a generic browser intended to impersonate Google Chrome on macOS."

Very crooked company. But then, I don't know of any generative AI company that isn't based on theft and deceit.

The AI bots are doing terrible damage to the internet. Including here at DU. As EarlG explained last week

https://www.democraticunderground.com/101316061

the downtime and update then were at least partly about the bot problem, especially AI scrapers.

[Quanto Magnus]

1. Lie and steal

this is how they all act....

Lie about the product
Steal material from others
Lie about the theft
Lie some more...

[customerserviceguy]

2. It doesn't really make any difference

what laws or regulations we make, some geeks are always going to try to get around them, any way possible.

[Karasu]

3. While true, that is not an excuse to never regulate the AI industry. Until something is done, they see this as a blank

check to do whatever the fuck they want, whenever they want, however they want.

It is beyond absurd that something this world-altering has gone completely unchecked for as long as it already has.

[customerserviceguy]

4. Pass laws and regulations

if it makes you feel better, but don't be under any illusion that you've solved a problem.

[Karasu]

5. I agree it won't solve the problem. But I also think it's better than ACTIVELY enabling them through inaction, certainly

in the case of the AI industry.