A website that is not Search Engine Friendly (SEF) can sabotage the best optimization efforts. Fundamentally, websites need be constructed in a manner that allows search engines to crawl, read web page content, and index website pages.

Crawl or spider are terms used to describe how a search engine robot (“bot” for short), inspects websites. All major search engine bots start at the website domain name – typically a home page. They read links on the home page and follow them to the site’s interior pages.
There are several reasons why a search engine bot may not crawl a website. Most are listed on search engine Webmaster Guidelines pages.

Google:http://www.google.com/support/webmasters/bin
/answer.py?answer=35769
Yahoo:http://help.yahoo.com/l/us/yahoo/search/basics
/basics-18.html
MSN: http://www.seoconsultants.com/search-engines/guidelines/

Yet it is still common to find websites that have been designed using techniques that are not SEF.

The most common infractions are:

• Passing session variables in the URL
• Improper re-directing of alias or other domain names
• Content copied from other websites
• Insufficient body content (flash or graphic websites)
• Hidden text (hidden div tags / css)
• Poor HTML coding practices

Basic observation combined with several publicly available tools may be used to check a website’s SEF condition.

Passing session variables in the URL may be observed by checking interior pages. If the URL appearing in a browser includes an attribute like “SID=” followed by more than 8 to 16 alphanumeric characters, the attribute is likely to be a session ID. Most search engines will not crawl these pages. Google states:“Allow search bots to crawl your sites without session IDs or arguments that track their path through the site.”

Improper re-directing of alias or other domain names is common. Publicly available HTTP header check services may be used to inspect current condition. Whenever possible, avoid using 302 redirects. Instead use 301 redirects.

Content copied from other websites is frowned upon and can negatively affect rankings. www.copyscape.com is a tool that reports copies of web page content on other websites.

Insufficient body content negatively affects rankings. If search engines are unable to determine what a web page is about, rankings may be compromised.

General guidelines suggest that SEF pages contain a minimum of 250 words in paragraph format. Since “stop words” (a, is, the, are, it, etc.) make up approximately 40-50% of typical content, the remaining words are what the search engines use to determine page substance. This amount of remaining text provides sufficient opportunity for keyword phrases to be repeated naturally throughout the text.

Hidden text is considered SPAM. Search engines will penalize, even ban a website when these conditions are found.
Currently, several HTML methods that produce hidden text are not detectable by search engines. However, aggressive competitors desiring stronger rankings can manually detect these conditions and report offending websites to search engines. 

Poor HTML can cause search engine bots to error. Search engine bots are different than browsers and, in many cases, more sensitive to poor HTML code. If a search engine bot errors on a page, it will not index that page nor will it continue to follow links on that page.

In some cases, excessive broken links will have a negative impact on rankings.