Technology News Updated 10:19 AM ET April 3, 1998
Internet Search Engines Inefficient, Study Finds
By Maggie Fox, Health and Science Correspondent
WASHINGTON (Reuters) - If you use a search engine like Lycos or HotBot to search the Internet, you may not get as much coverage as you think, computer science researchers said.
Steve Lawrence and Lee Giles of the NEC Research Institute in Princeton, New Jersey, said their study found that people who rely on the major search engines to find information on the Web only get a small proportion of the available documents.
"The engines index only a fraction of the total number of documents on the Web; the coverage of any one engine is significantly limited," they wrote in a report in the journal Science.
This means that people cannot use just one engine to get the full benefits of the World Wide Web, they said.
"With the introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos and Northern Light, the Web can be viewed as a searchable 15-billion-word encyclopedia," they wrote.
"The Web search engines have made a large and growing body of scientific literature and other information resources accessible within seconds."
But search engines are not all equal, they found. They analyzed various responses by the search engines to queries made by employees of the NEC Research Institute.
They used the entire lists of documents offered by each search engine, but left out documents that did not contain any of the search terms specified, which frequently happens.
They estimated the Web contains 320 million pages of information -- much more than previously published estimates.
HotBot covers 34 percent of these, they found, AltaVista 28 percent, Northern Light 20 percent, Excite 14 percent, Infoseek 10 percent and Lycos 3 percent.
"Combining the six engines in this study covered about 3.5 times as much of the Web as one engine," they wrote.
"Given that the coverage of any one search engine is limited, the simplest means of improving the coverage of Web searches is to combine the results of multiple engines, as is done with new search engines such as MetaCrawler (www. metacrawler.com)," they wrote.
Or perhaps there was call for a stronger search engine for scientists, they suggested.
Their analysis did not include Yahoo's search page, because it does the searches manually and tends to be used for more general purposes.
Not everyone may need the broadest search possible, they warned.
"A lot of people tend to search for common things and single terms and these can return a huge number of results from all the engines, and this is a reason one of the smaller engines may be preferred," Lawrence said in a telephone interview. Also, some engines may be easier for some people to use.
The authors, whose company homepage can be seen at www.neci. nj.nec.com, also checked to see which engines returned out-of- date documents. Lycos had the fewest with 1.6 percent, while HotBot, with 5.3 percent, had the most.
Last month Internet research firm RelevantKnowledge said Yahoo was the most popular Web site. Excite's was the fourth most visited site and Digital Equipment's AltaVista was 10th.
|
