Google & the HTTP Referrer: Split Personality Disorder?

Google moving to obfuscate HTTP Referrer data. You should be concerned.

Background

Since “the beginning” of the Web as we know it, webmasters (and thus marketers) have relied on the HTTP Referrer to identify where visitors are coming from.

The HTTP Referrer is a web standard. Yes, it’s an “optional” standard, but it has almost universally been employed. When a web user clicks on a link to your website, his or her browser is really sending a request to your server, asking your server to send information to their browser so that they can view your web page. That request is made up of several parts.

  • From
  • Accept
  • Accept-Encoding
  • Accept-Language
  • User-Agent
  • Referrer
  • Authorization
  • Charge-To
  • If-Modified-Since
  • Pragma

The standard was created in 1992 and can be read in its entirety at HTTP: A protocol for networked information on the W3C website.

The W3C states specifically that referrer…

“allows the client to specify, for the server’s benefit, the address (URI) of the document (or element within the document) from which the URI in the request was obtained.”

“This allows a server to generate lists of back-links to documents, for interest, logging, etc. It allows bad links to be traced for maintenance.”

When a user enters a query into a search engine such as Google then hits the “search” button, the page is refreshed to a new page with a new URI. The new URI looks like this:

Google search: HTTP referrer
http://www.google.com/search?client=safari&rls=en&q=http+referrer&ie=UTF-8&oe=UTF-8

Bing search: HTTP referrer:
http://www.bing.com/search?q=http+referrer&go=&qs=n&sk=&sc=8-13&form=QBLH

Yahoo! search: HTTP referrer
http://search.yahoo.com/search;_ylt=AoFlxbJ.X1GaDvI2D_Ll6pKbvZx4?p=http+referrer&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-701

Those URIs all include the query. In Google’s case, it can be found in the “&q=http+referrer” part of the URL. For Bing, it’s “q=http+referrer,” and for Yahoo!, it’s “?p=http+referrer.”

The fact that the URI is new also enables the user to bookmark that search for future use.

How Marketers Rely on HTTP Referrer

If you are involved in any way with Internet marketing, you rely on data in the HTTP Referrer every time you check your website analytics, whether you’re using Omniture, Google Analytics, or another platform. You track referrals from search engines, paid search campaigns, Internet advertising or banner campaigns, email campaigns, and websites in general to see how your website is performing and how your marketing programs are performing.

You use that data to improve your website. For example, if you find that many people are visiting your website after searching Google for a specific phrase but then are dropping out at a high rate, you can adjust your website to make it more useful to people using that specific phrase. That improves your website which improves the Internet in general.

Google’s Personality Disorder

Google has stated repeatedly that their goal is to help webmasters make their sites more useful so that the Internet will be more useful. They have created many products to facilitate this, including Google Analytics, Page Speed, Website Optimizer, and hundreds of other Google Apps.

Of course, if you use Google Analytics, you already know that Referrer data is a major part of the platform.

So why did Google announce on October 18th (see Making search more secure) that they will be obfuscating referrer data for signed-in Google users? Webmasters will know that the visitor came from Google, but will not receive any additional information.

Their position is that they’re doing it to make search more secure.

That’s pure hooey.

First of all, no one is saying that their privacy is being risked by passing referrer data to servers in HTTP requests, a standard that has been followed since 1992. I.e., it’s a non-issue. Secondly, Google is still passing that exact data to ADVERTISERS, even for logged-in Google users.

Good Google: We’re devoted to making the Internet more useful.

Bad Google: We’re not going to let you see how people got to your site from Google anymore, EXCEPT through our AdWords program.

Talk about cognitive dissonance.

How can Google claim to be protecting privacy by not sending complete referrer to webmasters while at the same time sending that data to advertisers?

The more important question is WHY.

By gradually introducing this new “feature,” Google is hoping not to concern major companies. At first, only a tiny fraction of Google visitors’ search queries will be blocked to webmasters. Google hopes that major companies will not notice. But mark our words – this is only the first salvo. Over the next 1-2 years, Google will gradually move entirely to obfuscating referrer data sent to webmasters. Eventually, when a webmaster looks through their web analytics, they won’t know how any visitors searched on Google to find their websites – UNLESS THEY ARE USING GOOGLE ADWORDS.

Since Google “borrowed” the Overture patent on paid search (Patent ‘361) in early 2001, they have been pushing to get webmasters to rely more on AdWords. This move is a very forceful push in that direction.

References

HTTP: A protocol for networked information
Basic HTTP as defined in 1992
W3C ® (MIT, ERCIM, Keio)
1992
URL: http://www.w3.org/Protocols/HTTP/HTTP2.html

Making search more secure
Posted by Evelyn Kao, Product Manager
10/18/2011 11:07:00 AM
The Official Google Blog
URL: http://googleblog.blogspot.com/2011/10/making-search-more-secure.html

Google, Google Adwords, Google Analytics, Google SEO, Google Webmaster Tools, Internet Marketing, Internet Privacy, Pay Per Click Marketing, Search Engine Marketing, Search Engine Optimization, Search Engines,