For a while now our server costs have been going up and up, month after month, and earlier this year we had to upgrade our database to deal with excess traffic (some DUers may recall that prior to this upgrade we were frequently having to switch to "members-only" mode during the day because the database would crap out).
As it turned out, the problem was that most of that excess traffic was bogus -- not real people, just bot crawlers. If you're a site that relies on advertising for revenue, that's a double problem, because advertisers don't like bogus bot traffic.
You can see where this is going. The bogus excess traffic caused our server costs to climb, and caused our ad revenue to drop. We were getting screwed at both ends, all because these AI assholes feel like they have the right to crawl everyone's sites and steal everyone's content.
Last month we made some changes to our server configuration and firewall which acts as strong block on a lot of bot traffic -- but not all of it. We've reduced the problem significantly, but it hasn't completely gone away:
https://www.theverge.com/news/718319/perplexity-stealth-crawling-cloudflare-ai-bots-report
It's disgusting, and I don't disagree that it is endangering the open Internet. Ultimately, if it came right down to it and DU could no longer afford to run because of these crawlers, my next move would be to make DU a private site which is unavailable to anyone who is not signed in. This would necessitate that everyone pay a subscription in order to use DU (since we would no longer be able to make ad revenue, and because confirming that someone is a paying subscriber would protect against bots accessing the site), so whether or not it is feasible would depend on how many people would be prepared to do that. To be completely clear, I have absolutely no plans to make this happen at the moment, but it's something I've had at the back of my mind as a "break glass in case of emergency" type situation.
= new reply since forum marked as read