How to get your web pages to "Stick" in Excite

For about three months now Excite has been dropping previously indexed pages apparently for no reason. It's normally pretty good about indexing new pages, but after a month or two those pages tend to drop out of the index. Normally only a few pages tend to stay in the index, the home page, (http:www.yourserver.com) and occasionally some of the pages in the root directory (www.yourserver.com/pages.html).

We have discovered a two step approach to keeping more of our pages in the index, including our pages in subdirectories. The first step involves making sure that pages you want to keep indexed have links pointing to them from a "home" page on another site.

We originally set up a series of links in an effort to get more pages indexed by the crawler. After a few months from when we first implemented this, we discovered to our surprise that these pages stayed in the index, when others tended to drop out after one or two months.

We found that placing a hidden link -- as described in trick #15 in the SE book -- would cause Excite's crawler to find our pages on it's own -- without having to submit them and run the risk of violating Excite's "25 page submission limit". This hidden link on the "home" page of a different web site led to another list of links all pointing at pages we wanted to be indexed. The pages were nothing more than a simple html page that had links with descriptions.

The second strategy we used involved keeping our pages "fresh" when Excite came calling to "crawled" them.