- All About Search Indexing Robots and Spiders
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
http://www.searchtools.com/robots/
(Added: Thu Apr 06 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- HTTP User Agent Index
An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
http://www.siteware.ch/webresources/useragents/db.html
(Added: Sun Apr 02 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- List of Robot Agent Strings
A list from PGTS of Web robots with the identifying data they leave in Web site logs.
http://www.pgts.com.au/pgtsj/pgtsj0208d.html
(Added: Mon Apr 03 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Psychedelix List of User-Agents
Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
http://www.psychedelix.com/agents.html
(Added: Thu Apr 06 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Robot IP Address
Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
http://www.briandunning.com/seo/
(Added: Thu Apr 06 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Robots, Spiders and Other User Agents: a Resource for WebMasters
José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.
http://joseluis.pellicer.org/ua/
(Added: Mon Apr 03 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Search Engine IP Addresses
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
http://www.iplists.com/
(Added: Wed Apr 05 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Search Engine Robots and Other User Agents
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
http://www.jafsoft.com/searchengines/webbots.html
(Added: Wed Apr 05 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
- Spider Track
Tracks and displays real time hits and IP addresses of MSNbot, Yahoo's Slurp, and Googlebot. Provides aggregate hits of each spider over the last 30 days.
http://www.spidertrack.org
(Added: Mon Apr 03 2006 Rating: 0.00 Votes: 0)
Rate It
Review It
|
|
