Archives
 
 
 
  Special
 
 
 
  About Us
 
 
 

Newsletter
Free E-mail Newsletter from BYTE.com

 
    
           
Visit the home page Browse the four-year online archive Download platform-neutral CPU/FPU benchmarks Find information for advertisers, authors, vendors, subscribers Request free information on products written about or advertised in BYTE Submit a press release, or scan recent announcements Talk with BYTE's staff and readers about products and technologies

ArticlesNo Robots, Please


February 1997 / Inbox / No Robots, Please

In the text box "Getting Along with AltaVista" (November 1996 Web Project), Jon Udell forgot to mention that support for the file robots.txt is not automatic. The indexer might simply choose to ignore it. We have the biggest computer-virus infobase on the Internet, and it has already been sucked by several competitors. In the process, they recursed through our entire directory tree, ignoring the exclusions. Beta versions, customer-support files, and so forth could easily have been taken along. A subdirectory is never really a safe place to store confidential or private data.

Pierre Vandevenne, M.D.
Neupre, Belgium
http://www.datarescue.com

Thanks for pointing that out. Obeying the directives in robots.txt is just good manners, not a hard requirement. I should have made that clearer. --Jon Udell, executive editor


Up to the Inbox section contentsGo to previous article: Web MattersGo to next article: HAL SpeaksSearchSend a comment on this articleSubscribe to BYTE or BYTE on CD-ROM  
Flexible C++
Matthew Wilson
My approach to software engineering is far more pragmatic than it is theoretical--and no language better exemplifies this than C++.

more...

BYTE Digest

BYTE Digest editors every month analyze and evaluate the best articles from Information Week, EE Times, Dr. Dobb's Journal, Network Computing, Sys Admin, and dozens of other CMP publications—bringing you critical news and information about wireless communication, computer security, software development, embedded systems, and more!

Find out more

BYTE.com Store

BYTE CD-ROM
NOW, on one CD-ROM, you can instantly access more than 8 years of BYTE.
 
The Best of BYTE Volume 1: Programming Languages
The Best of BYTE
Volume 1: Programming Languages
In this issue of Best of BYTE, we bring together some of the leading programming language designers and implementors...

Copyright © 2005 CMP Media LLC, Privacy Policy, Your California Privacy rights, Terms of Service
Site comments: webmaster@byte.com
SDMG Web Sites: BYTE.com, C/C++ Users Journal, Dr. Dobb's Journal, MSDN Magazine, New Architect, SD Expo, SD Magazine, Sys Admin, The Perl Journal, UnixReview.com, Windows Developer Network