Indexing Dynamic Content and Active Server Pages
FAQ and guide to getting dynamic content and .asp pages indexed in the engines

Every month, some of the most frequently asked questions we receive pertain to the search engine issues revolving around pages with dynamic content -- page content that is generated on the fly by grabbing results from a lookup table or database in order to satisfied a specified query.

The most telltale sign that dynamic content is being used is the "?" (question mark) character located in the page's URL. Unfortunately, the ? (and dynamic content in general) presents indexing problems for search engine spiders -- it short, they tend to choke on the ?.

Fortunately, there are ways around this problem. One of the typical questions we get goes something like this...

Based on analysis of our traffic, we're noticing NONE of the SE spiders are able to spider our .asp* pages.

* Note: .asp = active server pages

Actually, it's NOT the .asp file extension that's causing the problem -- it's the ...?page=/products/sales/orders.asp. In other words, it's the ? and the = characters.

In such cases we suggest that you talk to your programmers and tell them to get rid of any characters that are incompatible with search engine indexing. Most notably these include, ?, =, &.

Have them find a way to put the page variables into the URLs to make it work. For example, a URL such as...

http://www.yourdomain.com/index.asp?page=
/products/printers/model/models.asp&inpage=
/products/printers/model/lpr412.htm&series=
lpr&model=412&tab=2

should be changed to something like this:

http://www.yourdomain.com/index.asp/page
/products/printers/model/models.asp/inpage
/products/printers/model/lpr412.htm/series
/lpr/model/412/tab/2

The difference between the two being the second link is devoid of the special characters ?, =, & -- the characters that cause spiders to choke.

If ...