Demystifying Search Engines

Search engines are complex and dynamic, constantly changing how they search and what they search. To keep up with all the changes is a job in itself. The article below will hopefully help you understand them a lot better.

How important is being listed (indexed) with search engines? See the following statistics below.

• 85% Web surfers use search engines to find sites
  
• 91% Web surfers will change their queries if they are dissatisfied with the first 30 hits
  
• The first site listed after a search receives three times more hits than the fifth site listed
  
• The first 10 hits are visited 78 per cent more often than sites listed 11th to 30th
  
• Only 30% of corporate Web sites have basic meta-tagging (see http://www.suestudios.com/gloss.htm#M on our website for definition)
  
• Only 7 % of corporate Web sites are optimized for indexation

Top

Search Engine Components

A Search Engine has 3 basic parts:

1. Spider (crawler, robot, worm or link finder): a computer program that roams the Web looking at HTML documents, collects data about them and brings back, sorts and then indexes the results. The data the spider collects will be a summary of keywords and a description of the page.

2. Index: a database of the results collected by the spider that is organized and searchable. When using a search engine, you are actually searching the database, not "the Internet". The index is NOT organized by subject categories but are ranked by a computer algorithm.

3. Search and retrieval mechanism: Software that allows users to search the Index and return results in a predetermined order. This component is the interface (the program you actually use when using a search engine) between the end-user and the database.

Search engines vary as to the:

• Number of sites they index. Not even the biggest search engines come close to searching the entire web.

• Frequency of updating or "refreshing" the database.

• Methods of searching. The same search with different engines will yield different results.

• Versatility and complexity of the interface.

• Order of the results list. Some engines look at the number of pages that link to the site and incorporate this "popularity" component into the mix.

• Parts of each website they index (varies from URL and title to full-text of every web page). Some index words in the title, URL, introductory paragraphs, or full-text of all documents on a web site. Some use a combination of these words and phrases, all of which are entered into the search engine's database.

Top

Getting Listed on Search Engines

If you want to make your Web site as profitable as possible, Web surfers have to be able to find it using search engines and directories. If you can manage to get a good ranking for some of your top key phrases, you can get a steady stream of traffic.

1. Manual Submissions

   Manually submitting to search engines is usually done by visiting their front page and scrolling down to the bottom of the page where you will see a link titled 'Add URL' , 'Add Your Site' or 'Submit a Site'. It must be done directly at each search engine. This can be a very time consuming and complex exercise, as each search engine has different submission (indexation) guidelines. For example, search engines will vary on the number of keywords you can submit, how they are formatted and the length of the description of the site. Free submissions to search engines will vary the length of time that your site will actually be listed. Some will take months to actually be listed.

   Concentrate on being listed in the top 15 search engines: Yahoo, MSN, AOL, Lycos, Google, Netscape, Excite, iWon, Alta Vista, Looksmart, Direct Hit, Hotbot, AlltheWeb, The Open Directory and WebCrawler.

   Here are the submission URL's for some of the top "crawler-based" search engines. Click on the links below and fill out each of the forms (look for the link that says "free submission").
   http://www.google.com/addurl.html
   http://www.alltheweb.com/add_url.php
   http://addurl.altavista.com/addurl/new
   http://insite.lycos.com/searchservices/

2. Submission Software

   Submission software is a much faster method that can be less accurate because of the different submission guidelines that are constantly changing. It's not one-size-fits-all, and no software program can customize and update to the degree required for maintaining top listings.

3. Paying for listings

   The alternative to free submissions is to submit your site to what are called "Paid Inclusion" programs which charge a fee to include your site in their index. An advantage about paid inclusion programs is that they list your site usually within a week. The cost varies in pricing per URL for up to a year of inclusion in their listings. Given this, it is highly recommended that any site owner establish a search engine submission budget.

   Types of Paid Listings

   • Banner Ads: All major search engines carry keyword-linked banner advertising, either using graphical banners or text banners.

   • Content Promotion: Many major search engines will promote an advertiser's content, or their own content, on their search results pages. This is usually done in a separate area from the editorial results.

   • Paid Placement: Several major search engines carry paid placement listings, where sites are guaranteed a high ranking, usually in relation to desired words. The exact position of these listings can vary. Sometimes, they appear above editorial links. Other times, they appear at the bottom of editorial content. "Sidebar" style runs alongside the search engine's editorial area.

   • Paid Inclusion: This is where an advertiser might be more deeply listed than other sites in the editorial results. Unlike paid placement, this doesn't guarantee a particular position in the main search results. However, also unlike paid placement, it does interact directly with the editorial results. Being more deeply listed can help an advertiser be more likely to appear in response to a wide range of searches.

   • Paid Submission: This is where a search engine charges to process a request to be included in its listings. Typically, paid submission programs do not guarantee to list a site, only to review and possibly include it in a faster time frame than is normally done.

   • Pay-per-click search engines: With these search engines, you literally pay a certain amount of money for every click they send your way. You bid on keywords under which you want to be searched. The more you bid, the higher you'll rank for your search term and the more visible your link will be. The largest and most popular is Overture, http://www.overture.com/. A list of Pay Per Click programs is available from http://payperclicksearchengines.com/.

Top

Not every page is indexed by Search Engines

Search engine databases and spiders are optimized to "read" HTML, the basic language of the Web. Other types of programming languages contain codes and format requirements that are incompatible with HTML. HTML can carry links to these pages, but not full text of their content in their special format.

There are also some types of pages that search engine companies exclude by policy. It's a matter of selecting what and what not to include in databases that are already huge, expensive to operate, and low revenue producers.

Some types of web pages search engines choose to exclude:

• Format of the page - are frequently not included in search engine databases because surfers would infrequently or unsuccessfully search them.

• Pages with images and no text are also often omitted because without text, there are no keywords on which to search.

• Pages formatted in PDF and other pages written using very little if any HTML text.

• Search engines also omit the contents of Flash, Shockwave, and other programs like Word, WordPerfect, PowerPoint etc.

• Pages requiring passwords to access them are also closed to search engines, because spiders cannot type the password nor have access to the password.

Top