Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News Editorials & Other Articles General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

General Discussion

hunter

(40,702 posts)

8. I'm unexpectedly ambivalent about this.

Reply to blogslut (Original post)

Sun Apr 23, 2017, 12:17 PM

Apr 2017

Maybe it's because the bad guys of the internet never paid attention to robots.txt files anyways.

I haven't paid attention to robots.txt on my personal sites for many years now, but I do remember when I used to care.

Edit history

Please sign in to view edit histories.

Recommendations

0 members have recommended this reply (displayed in chronological order):

9 replies

= new reply since forum marked as read

Highlight:

Internet Archive to ignore robots.txt directives [View all] blogslut Apr 2017 OP

I disagree with this. If you own a web site, it should be your decision. 50 Shades Of Blue Apr 2017 #1

Their description of what they're doing makes perfect sense. PSPS Apr 2017 #2

I disagree. I think they should respect robots.txt files. 50 Shades Of Blue Apr 2017 #3

I don't. Things on the net shouldn't be allowed to disappear down the memory hole emulatorloo Apr 2017 #6

If you do not want content hosted on a domain you own to MineralMan Apr 2017 #7

I disagree with you. Your argument is not logical. bitterross Apr 2017 #9

I've got a "book" (80 pp) on Internet Archives. I don't understand the article, but am interested. UTUSN Apr 2017 #4

robots.txt is a very, very weak tool. MineralMan Apr 2017 #5

I'm unexpectedly ambivalent about this. hunter Apr 2017 #8