What about the issue of which search engines metacrawler uses....are these arrangement with the ones they have fixed in stone or could a search engine say sorry you can't search us anymore unless such and such.
Very good question. I don't know the answer and was hoping someone else would jump in.
I'm happy to make a few long-winded guesses, however. So:
In the old days, when Metacrawler was a non-commercial research project at the University of Washington and AltaVista was a demonstration project by engineers at Digital and other search engines were also ideas in search of a business plan, I don't think there was much worry about "borrowing" results. But things have changed.
Most of the spider-based search engines are now parts of big web media companies. I don't know if Metacrawler needs to make agreements with the search engines they include, but I'd guess there must be some kind of interaction. It's possible that a search engine would consider this kind of listing to be a form of advertising since the meta searchers always seem to provide a link to the index returning the results. It wouldn't be all that difficult for a search engine to block queries from Metacrawler or Dogpile, so they seem to give at least tacit approval to the process.
Why don't they include Google? Again, I don't know, but it might be because it's still called a "beta" site. It might be that Metacrawler hasn't gotten around to adding it, or that Google doesn't have the server capacity yet to return results to other sites, or just simply doesn't want to share its data.
---
In another message, I believe you asked about WebCrawler. It is now owned by Excite. AOL owned it for awhile, but sold it to Excite in '97 (http://www.news.com/News/Item/0,4,15512,00.html). |
