One of the biggest reasons why many organizations are not doing even simple things to help people find their websites in search engines is they simply don’t understand how search engines work. This article will explain the basics of how search engines work and shatter some search engine myths and help you understand what you can do to help people find your website in search engines.
One of the biggest reasons why many organizations are not doing even
simple things to help people find their websites in search engines is
they simply don’t understand how search engines work. This article is
intended to explain the basics of how search engines work and in the
course of doing so shatter some search engine myths and help you
understand what you can do to help people find your website in search
engines.
What is a search engine?The first to step to
understanding how search engines work is to understand what a search
engine is. The simplest explanation is that search engine is a tool for
finding things online. There are many different types of search engines
but they can be put into two major categories…
Human-Powered DirectoriesThe
first “search engines” were not actually search engines as we think of
them today,

but searchable directories of websites organized by
hierarchical categories. A site is added to a directory when the
website owner fills out a submission form on the directory’s website
requesting their site be included in the directory and include their
website’s title, description, URL (web address), and category. A
moderator later reviews the site and if it meets the directory’s
criteria a listing for that site is added or activated. The original
Yahoo was originally a human-powered directory, and directories like
Open Directory Project continue on today.
The downside of the
human-powered search engine is that it only includes websites that have
been submitted to it, which means you may not find what you’re looking
for especially if it’s a new web page. The other downside from the
directories’ point of view is that reviewing every site submitted is
very labor-intensive and costly. I know because my company’s “search
engine”, OurChurch.Com’s Directory of Christian Websites, like almost
all church/Christian “search engines”, is a human-powered directory.
Crawler/Spider-Powered Search EnginesThe
next generation of search engines has programs which actively seek out
new sites and read them into their indexes. These programs are called
crawlers, spiders, robots, or bots. All of largest and most popular
search engines today are of this type, including Google, Yahoo, MSN,
and Ask.
The rest of this article is focused on how these
crawler/spider-powered search engines work because more than 99% of
searches are done on this type of search engine.
The CrawlerAs
mentioned above, the first part of a search engine is the crawler (AKA
spider, robot, and bot). The crawler reads pages into the search engine
raw database of every page the search engine crawler has read. Some
important things to know about crawlers…
1) Search bots periodically reread every web page in their database. Why does this matter?
•
You don’t have to do anything when you change your site. If your site
is already listed in a search engine and you make changes to your
website, search engines will eventually update their information.
•
You do have to be patient. The frequency of re-crawling varies
depending on the search engine, the importance of your site (as
determined by the search engines), and how often you update it. It
could take a day or it could take more than a week.
2) Search bots follow links on the pages that have already been crawled in order to find new pages. Why does this matter?
•
If you add a new page to your website or create a new website, it’s
important to add a link to it on a web page that is already in the
search engines.
• If a web page that is already in the search
engines has a link to your new page or new website, you do not need to
submit a request to the search engines to crawl the new page or
website, though it may still be worth doing just to be sure your
website will be found quickly.
3) Some search engines have forms you can submit to request a website be crawled. Why does this matter?
•
If you have a new website and no sites link to it, search bots will not
be able to find it. In this case, submitting a form to the search
engine requesting your site be listed or indexed can get it into search
engines.
• Because human-powered directories do not have search bots/crawlers, to be listed in them you must submit a request form.
The Ranking AlgorithmSome
time after a web page has been crawled by the search bot or crawler,
the search engine then processes or indexes the page to determine what
search words and phrases the page is relevant to as well as how
relevant that page is compared with other web pages for those phrases.
During this processing the search engine looks at many different
factors including how many times each word and phrase occurs on the
page, which words are in headings or bold, the domain name of the site,
filename of the page, the pages that link to the page, and many more.
Exactly
which factors a search engine looks at and how they’re weighted is
called the search engine’s search ranking algorithm. It’s like the
search engine’s “secret sauce.” Each search engine’s algorithm is
different and each is a heavily guarded secret.
Why does this matter?
•
There is time between when your site is crawled (or recrawled) and when
it is processed or indexed. So, it can take as little as a few hours to
many days or weeks before changes to your website may produce changes
in its search rankings.
• Because each search engine’s ranking
algorithm is different, a web page can be #1 in Google but #20 in Yahoo
for a particular phrase.
• Because each search algorithm is a
heavily guarded secret nobody outside of a few select engineers at each
search engine knows exactly how much each particular factor weighs into
the rankings of each search engine. But, people who spend their
professional lives helping sites rank better in search engines have
gained very good idea as to what factors matter most.
• Because
the search ranking algorithms look at text, headings, and other
elements on a web page, changing things on the web page can change
where that web page appears in the search results.
• Search
ranking algorithms look at factors outside of a web page, such as the
age of a website and links to the web page. So, there are other factors
which you may have less influence over.
The goal of every
search engine is to display to the user the information or websites the
user is looking for. In other words every search engine wants to
provide the best, most relevant results. As a result, search engines
are constantly improving their algorithms and including new factors
which they think will produce better search results. Why does this
matter? As search algorithms change, so will your website’s search
engine rankings.
The Search Engine Results Pages (SERPs) The
crawling of websites and indexing of web pages are constantly going on
even when no one is searching. The last part in the search process is
part that you’re probably most familiar with – the actual search. You
type in a word, short phrase, or question, and the search engine
displays a list of websites.
That list of websites is called the search engine results page and sometimes referred to as the SERPs.
If
you do a search and look at the SERP, you’ll notice two columns. Above
the right column it says “Sponsored Links.” These are paid
advertisements. The companies and organizations listed here pay a fee
to Google for each person who clicks their ad, so they’re often
referred to as Pay-Per-Click or PPC ads.
In the left column are
the unpaid search results also sometimes called the organic results or
natural results. These are the websites the search engine believes are
most relevant to the search phrase that was queried. Sometimes search
engines also display sponsored links in the left column above the
natural results or will even place a block of pay-per-click ads in the
middle of the natural results. Sponsored links are always labeled,
though not always very clearly.
Why does this matter? From
the searcher’s perspective, there is nothing wrong with clicking on a
paid link. You may find what you’re looking for there. But it’s
important to be aware of which websites paid to be in the results and
which are there naturally.
From the web administrator’s
perspective, it’s important to understand there are two opportunities
to get to the top of the search engine rankings, through natural
results and by purchasing pay-per-click advertising.
Help the Search EnginesWith
a better understanding how search engines work, you can make better
decisions about the marketing of your website though search engines.
Search engines need your help to find your website and to know what
words and phrases it’s relevant for. Give them the help they need.
That’s where search engine optimization comes in, but I’ll get to that
on another day.