General Discussion

MineralMan

(151,293 posts)

7. If you do not want content hosted on a domain you own to

Sun Apr 23, 2017, 12:11 PM

Apr 2017

be visible to search engines and others, there are far, far better ways than using robots.txt. In fact, that file can be used to find out what isn't visible on a website and go peek into it.

One of my old websites exists now only on archive.org. When I shut down a website, I remove all of the content on it, post a notice at the root URL that the site is gone, and then discontinue hosting. However, I do still occasionally look at that site as it appears on archive.org. I can go back to its early, early days in the mid 1990s, if I like. I like having that archive available, just for historical interest.

Edit history

Please sign in to view edit histories.

Recommendations

0 members have recommended this reply (displayed in chronological order):

9 replies

= new reply since forum marked as read

Highlight:

Internet Archive to ignore robots.txt directives [View all] blogslut Apr 2017 OP

I disagree with this. If you own a web site, it should be your decision. 50 Shades Of Blue Apr 2017 #1

Their description of what they're doing makes perfect sense. PSPS Apr 2017 #2

I disagree. I think they should respect robots.txt files. 50 Shades Of Blue Apr 2017 #3

I don't. Things on the net shouldn't be allowed to disappear down the memory hole emulatorloo Apr 2017 #6

If you do not want content hosted on a domain you own to MineralMan Apr 2017 #7

I disagree with you. Your argument is not logical. bitterross Apr 2017 #9

I've got a "book" (80 pp) on Internet Archives. I don't understand the article, but am interested. UTUSN Apr 2017 #4

robots.txt is a very, very weak tool. MineralMan Apr 2017 #5

I'm unexpectedly ambivalent about this. hunter Apr 2017 #8