The Creepy-Crawlies

This week’s fare: How Google Crawls, Indexes, and Serves the Web.

How does Google know about so many sites serving so many interests all around the world? How does Google compile a list of millions of results from one simple search for something as mundane as “bread”? Do people pay money to have their name at the top of the search page? And what about the little guy – does he/she lose out when it comes to a chance at the top?

The more I learn about Google indexing, the more fascinated I am. Of course, I’ve said it before, I am decidedly right-brained and even the mention of the word “algorithm” makes the left side of my brain yawn, open one eye, sort of groan, and then complain about being woken up yet again to try to explain something to me that I am never going to get. Once we’ve all acknowledged this proclivity, it should come as no surprise that I really do picture tiny little robots going to town on all the pages of the web. (I mean, c’mon, this is the same person who relished the image of “hand-to-hand spam fighting” from last week.)

In all seriousness, I am thoroughly amazed in the ability of Google to index sites over and over again, registering new, out-of-date, or updated sites. I’m sure the hardware investment alone is significant, warranting this comment from its Webmaster Tools article on the Googlebot: We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. (What does Google consider a “huge set of computers”?) The most welcome part is the end of my perception that people or organizations pay to be in the top ten. (I guess, in a way, they “pay” to optimize their chances of getting seen in a search, but they don’t pay Google.)

In this forum, everyone has a chance of being seen in the top of the search order if they’ve done what they needed to do in SEO on the front end — keyword placement, URLs, title tags, ALT tags, site maps, meta tags, back links, and more. PageRank alone is determined by 200 factors. Every page is fair game for the spider, but it’s up to site managers to leave the right trail of food bits on the web.

I’m wondering if there is a resource out there – a case study of sorts – that really shows a “little” site doing things right enough to threaten one of the “big” ones? And is there a point with all these factors feeding the spider when enough is enough?



6 thoughts on “The Creepy-Crawlies

  1. Hi Shannon! Just so you know, if I had an limitations towards understanding SEO, you really spelt it out in my language. As a fellow right-brainer, “Every page is fair game for the spider, but it’s up to site managers to leave the right trail of food bits on the web” was right on point for me 🙂

    I definitely think a smaller site can utilize the SEO-ness of the web in a competitive way in that a smaller site is sometimes more tailored to a certain speciality of which a company can definitely leave food bits for the spider to grab when larger sites leave an equal amount of food bites for a lot more specialities. Like many things in life, you need to know where to invest that is smart for you company to stand out.

  2. I think that’d be a fascinating case study. The problem is simply the AMOUNT of information available for it. Even if we just focused on two sites, a larger one and a smaller one, there is so much data we can get regarding the visitors that it becomes really hard to focus on it all. I think one of the largest problems in the information age is all of the information we can access. It’s tough to know what is important to pay attention to when there is so much competing for our focus.

    • You’re right, the information overload would be phenomenal. I think for a case study on this, we’d have to narrow the parameters and select a limited, but diverse, number of variables. Not only that, the research into deciding which big site and which small site would be just as daunting. Oh well, it was a thought. Still, curiosity is getting the better of me and I’m going to think about how to do this a little more.

  3. I can relate with your feeling about algorithms, it makes my hair stand up just hearing the word. I did think your question was very interesting and it made me think. Google has become a household term and the services they offer definitely helps make life easier for individuals and companies alike. However, is it helping “little” sites take the spotlight away from the “bigger” sites? I think it might be, but I think its a good thing. In our society, it seems as long as you have money, things will go your way and this is true for many large companies swallowing up start-ups and entrepreneurs. Its nice to see that Google does not base its search results on how much money the company paid to put their link first, but rather if the website deserves that page rank.

  4. Shannan, great post! I also wonder whether a mom and pop shop that sells, let’s say donuts, can compete with larger websites, like Krispy Kreme or Dunkin Donuts. Even if a smaller brand is able to SEO-ize their site perfectly, they likely can’t make their way on the top of the search return for “donuts”, right? Why is this? I just did this “donuts” search myself and low and behold, Dunkin Donuts and Krispy Kreme, as well as a Wikipedia page, came up in the top five spots. Go figure.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s