How Do Spiders Work?

Does the spider program run through all links on a Web page?

No spider ever has indexed every link on the Internet. The Internet is growing so fast that it is difficult for any to keep up. There is a strong chance that every page on YOUR website will be indexed, and that every link on each of your Web pages will be followed.

Generally, when a new site is launched, a search engine spider will index the home page only. On subsequent visits, the spider will go deeper and deeper into your site, over a period of time that is determined by the number and quality of links from other websites that point to your website.

If you’re anxious to get Google to spider your site ASAP, though, you can submit a request for them to re-index your site, and use Google’s Index Coverage Report tool to see if your request has been processed yet.