About Search Engines

To Search      Search Tips

For all practical purposes, the only thing you really need to know about Search Engines is that if you type some words in the box, it will go find a document with those words in it.

They generally look like this:

However, if you really want to know how they work, read on.

A Search Engine is basically any program that sends out little programs (often called "robots", "spiders", "worms", or "ants") that scan information on the network, and stores the results back at their home, so that a catagorical list of the data may be made.

You see, the World Wide Web has become very popular in the last couple of years, and is now one of the primary means of publishing on the Internet. When the size of the Web increased beyond a few sites, and a small number of documents, it became clear that manual browsing through all the pages was no longer possible, let alone effective.

This problem prompted a number of experiments with automated browsing by "robots". A Web robot is any program that retrieves, or "hits on" a document, and then systematically retrieves, or "hits", all the links on that document. The information that it gathers, is then catagorized and referenced in a database of some sort. So when you type in the box, and click on the "Search" button, you are actually searching through the database that these robots have helped create.

The names of these robots are actually rather misleading. "Spider", "Wanderer", and "Web Crawler" all imply that they actually move, when in reality they just sit in on a computer in some building somewhere and hit on pages; just like you do. They are just a bit more systematical about it. And quite a bit faster.

Each robot program is a little different from the next. Some read an entire document, while some only read the title or certain key words. Some only test to see if the links on a document are valid, while another's only purpose is to count links as fast as it can, in order to guage how quickly the Web is growing.

The whole goal of all these robots, and their search engines, is to find out what's out there, as quickly as it can, so that you may more effectively utilize the World Wide Web to your advantage.

For Tips on Using a Search Engine, click here.

Further Reading:
"Entering the World Wide Web: a Guide to Cyberspace" by Kevin Hughes
"WWW Robot Frequently Asked Questions" maintained by Martijn Koster
World Wide Web Robots, Wanderers, and Spiders by Martijn Koster