Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News Editorials & Other Articles General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

General Discussion

PSPS

(15,337 posts)

2. Their description of what they're doing makes perfect sense.

Reply to 50 Shades Of Blue (Reply #1)

Sun Apr 23, 2017, 11:33 AM

Apr 2017

Malicious spiders have always ignored robots.txt anyway and Archive will continue to honor requests. This change seems targeted toward parked inactive domains to keep its history available, which is a very good idea.

Edit history

Please sign in to view edit histories.

Recommendations

0 members have recommended this reply (displayed in chronological order):

9 replies

= new reply since forum marked as read

Highlight:

Internet Archive to ignore robots.txt directives [View all] blogslut Apr 2017 OP

I disagree with this. If you own a web site, it should be your decision. 50 Shades Of Blue Apr 2017 #1

Their description of what they're doing makes perfect sense. PSPS Apr 2017 #2

I disagree. I think they should respect robots.txt files. 50 Shades Of Blue Apr 2017 #3

I don't. Things on the net shouldn't be allowed to disappear down the memory hole emulatorloo Apr 2017 #6

If you do not want content hosted on a domain you own to MineralMan Apr 2017 #7

I disagree with you. Your argument is not logical. bitterross Apr 2017 #9

I've got a "book" (80 pp) on Internet Archives. I don't understand the article, but am interested. UTUSN Apr 2017 #4

robots.txt is a very, very weak tool. MineralMan Apr 2017 #5

I'm unexpectedly ambivalent about this. hunter Apr 2017 #8