AJAX and the Search Engine Problems

Michael Schwarz on Saturday, August 6, 2005

There are already thousands of web sites using AJAX or similar techniques. One problem all of them have is the point that web crawlers will only search for HREF, SRC or some other link tags. I never saw a web crawler that is searching for AJAX JavaScript proxies. I think web developers only see the benefit of AJAX for the speed/performance advantage of web sites. If you want to build web sites that will be reachable through search engines you have to write two sites, one with the usage of AJAX for the human visitor and one for web crawlers that will need the complete HTML/text of the page.

Backbase.com is one of the web sites that is using two versions:

Note: You cannot open the "search engine" version directly, use telnet.exe or the Google cached page at [3]. If you click on the link for the search engines you will be redirected to the AJAX version.

We have to think about this problem when we are designing web sites, not web applications. On real web applications the need of be search engine compatible is not as high as for news and product information web sites. Or will Google change their search engine?