Robots and SEO

Search engines and Robots

Starting at the beginning, 4little1s.com has been trying to improve its page rank and listing in the search engines for a number of months and details some of the notes it has discovered so far: -

Robots are pieces of software that explore the Internet to discover, review and index what they find. You have very little control over them and are also known as trawlers, crawlers, spiders, wanderers and worms.

Robots come in 3 varieties, the good, the bad and the ugly. The good ones search and index the Internet and register the information it finds. The most common being the search engines such as Google or Yahoo which sends them out to explore. These then go through your website to assist in classifying and applying a rank to your site. Other robots simply check your web site for errors and report back on omissions, errors on links and any content that has been updated - there are others but the list goes on and on. That's it for the goodies, what about the baddies ?

Bad and ugly ones look to cause havoc by searching for weaknesses in your code and then extract information from you without your knowledge. The most common occurrence of this being to spam you or 'harvest' your email address to then sell on to other people….very annoying. As per the goodies there are others and the list goes on and on.

Very interesting, but so what ?

Well, in order to get listed in the search engines naturally and get as much targeted traffic to you as possible, you need to know this, so read on....

Most indexing is done automatically with the notable exception of two really important directories, DMOZ and the Yahoo Directory. Believe it or not, these are reviewed by humans and as a result, take longer. Note: The DMOZ submission is free but it can take many months to get listed.

Robots are a good thing so make your site available for them

What should you do?

On the assumption you do want a robot to visit your site, you must encourage them and make it easy for them to trawl your site. Finally see below for the top must do points:

- Add your URL to the Search Engines
- Submit to DMOZ (it free! But takes months)
- Submit to the Yahoo Directory (Not free but can be listed in a couple of weeks)
- Make a site map (search for Xml sitemaps)
- Link it to your Home Page
- ..........wait

Remember that robots like to see

- Links (internal and external)
- Text (lots of it)
- ALT Tags on your pictures
- Site Maps (on your pages + uploaded to the search engines)
- Page titles
- Meta descriptions
- Meta keywords

Robots don't like to see

- Java script
- FLASH
- Most animations
- Images (for the moment)

I'm scared of spiders and don't want them

Probably not the best thing in the world, but you can prevent spiders coming to see you. If you create a file called robots.txt, you have the ability to tell the robots what not to spider. This does have its uses, for example, if you have a telephone list on your site that you didn't realize was suddenly going to made public once the spiders came to town, add the exclusion to this file.

How do I know if they have been?

The easiest way is to look at your server logs, certain IP addresses can even tell you what type of spider came to visit. There are third party utilities that can extract your logs and make them more readable and friendly.


Author: 4little1s.com

Terms: This document can be copied as long as the link below stay intact.
4little1s.com is an online UK baby gifts, baby clothes and Izziwotnot nursery furniture web site located at http://www.4little1s.com