SI
SI
discoversearch

We've detected that you're using an ad content blocking browser plug-in or feature. Ads provide a critical source of revenue to the continued operation of Silicon Investor.  We ask that you disable ad blocking while on Silicon Investor in the best interests of our community.  If you are not using an ad blocker but are still receiving this message, make sure your browser's tracking protection is set to the 'standard' level.
Pastimes : Things That Amuse Me
 Public ReplyPrvt ReplyMark as Last ReadFilePrevious 10Next 10PreviousNext  
To: mr.mark who started this subject11/20/2001 2:55:13 PM
From: Quahog   of 12669
 
SI's robots.txt file:

#----------------------------------------------------------
# no shoes, no shirt, no service
User-agent: ia_archiver
Disallow: /

# robots generally aren't so bad
User-agent: *
# legacy paths that will always give redirects to the current page anyway
Disallow: /~wsapi
Disallow: /investor
# duh, you can't reply
Disallow: /stocktalk/msg_reply.gsp
# quotes, charts, etc. don't have any useful info for robots
Disallow: /research/quote.gsp
Disallow: /research/chart.gsp
# some value for robots, but more expensive than it is worth
Disallow: /research/industrygrps.gsp
# robot can't have have a portfolio
Disallow: /portfolio
# robots shouldn't try to do anything
Disallow: /stocktalk/do.gsp
# all these are dupes of content available through other URLs
Disallow: /stocktalk/msg_multireplies.gsp
Disallow: /stocktalk/msgs.gsp
# robots can't have personal messages
Disallow: /msgcenter
#----------------------------------------------------------

web.archive.org
Report TOU ViolationShare This Post
 Public ReplyPrvt ReplyMark as Last ReadFilePrevious 10Next 10PreviousNext  

HomeMailHotSubjectMarksPeopleMarksKeepersSettings
Terms Of UseContact UsCopyright/IP PolicyPrivacy PolicyAbout UsFAQAdvertise on SI
© 2026 Knight Sac Media.  Data provided by Twelve Data, Alpha Vantage, and CityFALCON News