This indexing is done by a software program called spiders, spiders start by fetching a few web pages and they follow the links on those pages and fetch the new web page they just found. This process continues to bring new web page and web content in index of Google
Understanding
how Google works should be the first step towards developing a great website.
(just in case you want visitors from Google
search)
According to Google when you do a Google search,
you are actually searching the Google index of web pages, and not the live web. http://www.google.com/howgoogleworks/#watch
This indexing is done by a software program called
spiders, spiders start by fetching a few web pages and they follow the links on
those pages and fetch the new web page they just found. This process continues
to bring new web page and web content in index of Google.
As you know, internet has got many billions of
websites and they all are talking about different things in different
languages. Maintaining an index of all known websites is a huge task and Google
does that by creating several layers of indexes.
If we assume that there is a layer “A” open for
public search, another layer “B” that is kind of backup and layer “C” that is
for catch.
Indexing software spiders should be interacting
with layer “C” and once they reach the limit of new updated content, another
program governed by ranking algorithm (based on website ranking logic) starts
work for giving a specific point to each page (page value assignment). Once
this process is over, “A” layer updates it self with layer “C”. Here they must
have kind of quality check on this new updated index, if everything is fine
they move to get new content from the internet. Just incase they find that
something went wrong and results are not correct, they can always update layer
“A” with backup layer “B” to restore old index.
Now, we have to understand the concept of search
so we can better understand how it works.
Google or any responsible search engine wants to
show results based on the demand by the user. Let’s take an example; I want to
search for a local SEO company in Bangalore.
What should I search for?
I will open google.com and will search for “seo company
in Bangalore”,
is not it? Now Google will look for all the pages available in its layer “A”
where there is word “SEO Company in Bangalore”,
they will check the quality, page rank and other relevant check lists
associates with this search term and show me a list of websites shorted in a
relevancy order.
A search result takes about half a second but in
this half second, software programs do check billions of pages, run hundreds of
quality checks before producing the actual result.
These software programs are fast and smart, and
they need to be updated continually to fight with ever growing leg pullers.
I am going to explain this in detail with
examples, stay tuned…
