Does Password Protection Block Spiders?

Can a search engine index pages that are password protected?

No, search engines cannot index pages that are password protected. For that reason, password protecting certain pages is the best way to prevent them from being indexed by the search engines. There are other ways, but none are as effective as password protecting files you don’t want random visitors to see.

In some cases, a business may want to encourage visitors to “sign up” in order to see certain content on the site, and they password protect that content in order to push people into signing up. Unfortunately, that content is often the best content available at a site, and would make very good spider food for attracting visitors via the search engines. For example, you may have a series of white papers on your site that you would like people to see, but only after they register. If you were to make them public, those white papers could end up winning on the search engines for the phrases your potential customers are searching. You are forced to make a trade-off. You can either use your content to draw visitors from the search engines, or you can pay for visitors from the search engines, then ask them to register to see the content.

One way to avoid the trade-off is to allow visitors (and search engines) to view the first couple pages of each of your articles, but require registration for full viewing. That would enable the search engines to index at least part of your protected content, and would drive more visitors to your site.

See also: Is there any way to prevent a spider from grabbing URLs that you want to keep off search engines?