SI
SI
discoversearch

We've detected that you're using an ad content blocking browser plug-in or feature. Ads provide a critical source of revenue to the continued operation of Silicon Investor.  We ask that you disable ad blocking while on Silicon Investor in the best interests of our community.  If you are not using an ad blocker but are still receiving this message, make sure your browser's tracking protection is set to the 'standard' level.
SI - Site Forums : Silicon Investor - Welcome New SI Members! -- Ignore unavailable to you. Want to Upgrade?

To: mmmary who wrote (18158)6/1/2003 12:45:16 PM
From: David Lawrence  Read Replies (1) | Respond to of 32936
 
How am I supposed to prove it? I looked at the robots.txt file, perhaps a year ago, when I went to see if I could find old threads via archive.org.

You prove that it just happened under Bob's watch.



To: mmmary who wrote (18158)6/2/2003 9:58:34 AM
From: SI Bob  Read Replies (1) | Respond to of 32936
 
prove that robots were set up this way earlier.

Robots.txt isn't "setting up robots".

It's a text file used to limit the activity of robots, or software written specfically to troll all available URLs at a website, usually for building a search database.

And I didn't write that robots.txt that you found. Heck, I only recently found the site-wide style sheet, so have no idea what would make you think I wrote that robots.txt or what in it is so evil.


HomeMailHotSubjectMarksPeopleMarksKeepersSettings
Terms Of UseContact UsCopyright/IP PolicyPrivacy PolicyAbout UsFAQAdvertise on SI
© 2026 Knight Sac Media.  Data provided by Twelve Data, Alpha Vantage, and CityFALCON News