The evolution of search
01-12-2009
I have taken an historical look into how search engines have evolved and how their techniques have evolved with them. I will attempt to describe the rise and fall of meta data and keyword stuffing and what the future may bring for SEO.
So to the beginning, or in this case 1993, with the advent of the World Wide Web Wanderer (Wanderer). Developed by Matthew Gray at the Massachusetts Institute of Technology, Wanderer was designed only to measure the size of the internet and ran until 1995. This was an important first for the internet as it had seen its first indexing spider. Its’ first report in June 1993 reported 130 websites, and by June 1995 counted over 23,500. Netcraft now estimates there are more than 106 million active domains, this figure does not include all the sub-domain sites like blogs etc... How times have changed!
Back in 1993, Jump Station was launched and was the first search engine that listed search results in a similar way to the Google, Bing and Yahoo search engines that we use today. It presented lists of URLs based on keywords but was not ordered by rank. The engine found relevant content by indexing document headers and titles. The age of meta data and title keyword stuffing was born. Unfortunately Jump Station closed down in 1994 after Jonathon Fletcher, the original developer, could not obtain the required investment to continue his work. One of the biggest investment misses ever? Maybe, but I am not sure “Jump Station it”, has the same ring to it as “Google it” though....
In April 1994, “David’s Guide to the World Wide Web” was renamed as the well know brand “Yet Another Hierarchical Officious Oracle” (Yahoo). By January 1995 the designers Yang and Filo had realised the business potential in their idea and raised $33.8 million by selling 2.6 million shares at $13 each. The internet search market was now monetized and the flood gates were opened. This arguably was the deal that shaped search into what it is today and perpetuated the massive online marketing machine that we now have.
Another major player in the early search engine landscape was formed in 1995. Bob Davis joined Lycos as the company’s CEO and first employee. The company was formed with around $2 million in venture capital funding. By 1999 it was the most visited online destination in the world with a global presence in more then 40 countries and in May 2000 it was sold to Terra Networks in Spain for $13 billion. I wish I had invested in Jump Station a year earlier though.
AltaVista lunched in December 1995 with the URL www.altavista.digital.com. It was the first searchable, full-text database of a large chunk of the World Wide Web. At launch, the AltaVista database was stored on 20 machines with 500GB of hard disk space each. Although not much is known about the size of the Google database today, we can safely assume it is somewhat larger! In 1998 Compaq paid AltaVista $3.3million for their domain name and in 2003 was taken over by Yahoo.
Up until this point the major search engines were still ranking their search results by analysing meta, page title data and page text. Search engines needed to evolve to become smarter as unscrupulous SEO techniques were rife, such as serving different pages to search spiders than to normal users. This particular technique got a certain large car manufacturer’s site banned in Germany in 2006.
In 1996 one of the most important projects in online search was started nicknamed “back rub” by Larry Page. Page went on to launch a larger, better known search engine. The “back rub” project was to understand the link structure of the Word Wide Web. This information on back link data would form the basis of the way in which Google ranked sites and was the birth of page rank - now such an important ingredient in SEO.
In 1998 Open Directory Project (ODP) launched and a new, novel approach was taken to ordering internet content. This search engine was different with the idea being that search engine result pages (SERPs) would be maintained by humans, through networks of unpaid moderators and would not use search spiders. This search engine does still exist but has suffered trying to keep up with the growth of the internet. It was a nice idea but realistically it was never going to be achievable.
In 1998 Google was formed on the back of a research project started by Page two years earlier. It was designed to rank sites on the search engine results page by a more complex method than just the number of times a particular keyword featured on that page. The idea of relevancy and linking was born and this has changed the way that both on site and off site optimisation is done forever.
Although this method is still not ideal and link exchange schemes, buying of links and the sculpting of link juice still exists, it is still currently the best way to organise the internet today.
There are lots developments in search engines they are now incorporating social, location and mobile data into their algorithms. We look forward to seeing all of these developments as search engines strive on to unlock the masses of data currently on the internet. Search has come along way since those 130 websites in 1993 and we look forward to witnessing its further evolution.
Written by: Martin McAndrew