How Do Search Engines Actually Work?

Think about how often and why you use Google, or other search engines to find products or information. As we begin our journey, the main question we have to ask is ‘How do search engines work to categorize websites and serve them back in results to searchers?’ For the majority of searches, this can be achieved by applying SEO techniques to drive visibility in search results and increase search traffic to our website.

Optimization for SEO

As a digital marketer, knowing how to get your brand, website, or company found by searchers is a core skill, and understanding how SEO is evolving will keep you at the top of your game. However, while SEO changes frequently in small ways, its key principles do not. Since Google came onto the scene in the late 1990s, we can break SEO into three components:

1.  Technical Optimization
2.  On-Page Optimization
3.  Off-Page Optimization

Technical Optimization is the process of completing activities on your site that are designed to improve SEO but are not related to content. It often happens behind the scenes.  

On-Page Optimization is the process of ensuring the content on your site is relevant and provides a great user experience. It includes targeting the right keywords within your content and can be done through a content management system.

A Content Management System, or CMS, is a piece of software used to manage a website. It is where the content of a website, such as text, images, and videos, can be uploaded or modified. A CMS can be used to create optimized page titles and descriptions for SEO purposes.

A CMS will usually have a user-friendly interface and in some cases, drag and drop functionality for text boxes and images so that you can easily design your own page layouts. Common examples of content management systems include Wordpress, Wix, Drupal, Joomla, Magento, Shopify, and Expression Engine.

Off-Page Optimization is the process of enhancing your site’s search engine rankings through activities outside of the site. This is largely driven by backlinks, which help to build the site’s reputation.

Search engine algorithms are computer programs that look for clues to give the searcher the exact results they are looking for. Search engines rely on algorithms to find web pages and decide which ones to rank for any given keyword. There are three steps to how search engines work:

1. Crawling: discovering a website's themes and topics
2. Indexing: registering the page in a search engine's database which is linked to certain topics; and
3. Ranking: sorting the position of search results based on the search parameters and themes associated in the search engine database

1. Crawling

The first step is crawling. Search engines send out web crawlers to find new pages and record information about them. We sometimes call these web crawlers ‘spiders’ or ‘robots. Their purpose is to discover new web pages that exist, and also to periodically check the content on pages they’ve previously visited to see whether they've changed or been updated.  

Search engines crawl web pages by following links they’ve already discovered. So, if you have a blog post and it's linked from your homepage, when a search engine crawls your homepage, it will then look for another link to follow and may follow the link to your new blog post. Websites sometimes instruct search engines not to crawl certain web pages so that they are left out of the index.

2. Indexing

The second step is indexing. Indexing is when a search engine decides whether or not it is going to use the content that it has crawled. If a crawled web page is deemed worthy by a search engine, it will be added to its index. This index is used at the final ranking stage. When a web page or piece of content is indexed, it is filed and stored in a database where it can later be retrieved. Most web pages that offer unique and valuable content are placed into the index. A web page might not be placed in the index if:

• it’s not mobile-friendly
• its content is considered duplicate
• its content is considered low value or spammy
• it couldn’t be crawled
• the page or domain lacked inbound links

3. Ranking

The third step is really the most important step, and that is ranking. For any given keyword, search engines sort or rank the results to give the searcher the most useful and relevant results they can find. Ranking can only happen after the crawling and indexing steps are complete. So once a search engine has crawled and indexed your site, your site can be ranked.

There are more than 200 ranking signals that search engines use to sort and rank content, and they all fit under the three pillars of SEO: technical optimization, on-page optimization, and off-page optimization. Some examples of signals that search engines use to rank web pages are: keyword presence in the title tag, loading speed of the web page, and website reputation.

Keyword Presence in Title Tag – Whether the keyword or a synonym was mentioned on the page and within the title tag.

Loading Speed of Web Page – Whether the web page loads quickly and is mobile friendly.

Website Reputation – Whether the web page and website is considered reputable for the topic being searched for.

Google's Algorithms

Google’s main search algorithm is called Google Hummingbird, and it is responsible for deciding how to order and rank search engine results.

Google also has a machine-learning search engine sub-algorithm called RankBrain.

• If RankBrain sees a word or phrase it isn’t familiar with, it uses artificial intelligence to better understand.
• It allows Google to understand these queries by converting keywords into known topics and concepts, meaning it can provide better search engine results – even when queries are unusual.
• Rather than attempting to be the best keyword optimized result, RankBrain rewards websites that provide user satisfaction and return the result that the user expects.

A good SEO strategy is to optimize your website to improve user experience and satisfaction and get the most out of the RankBrain ranking factor.

The three most effective ways to do this are:

1. Optimize for medium-tail keywords
2. Optimize page titles and descriptions for clicks so that when someone searches, your listing is more likely to be clicked. The clickthrough rate is the percentage of people who see you on Google and then go ahead and click through to your website. You can optimize CTR by using CTAs, including the most important keyword, and testing using brackets (because they really work).
3. Optimize content to increase dwell time (the length of time people stay on the page) and reduce bounce rate (the percentage of visitors who leave after only viewing one page).

Google’s top three ranking factors are:

• Links – it’s believed this reference is largely towards backlinks and off-page optimization
• Content – this applies to on-page optimization; and
• RankBrain – Google’s machine-learning search engine sub-algorithm that rewards sites that provide user satisfaction

In 2018, Google introduced the BERT algorithm, BERT is an enhanced way for search engines to understand more about the language of searcher in relation to context to deliver better results.

Related

• Google BERT: What Is It and Should It Change Your Approach to SEO?
• DMI Podcast: How Does SEO Really Work?
• eBook: The A to Z of SEO
• On-Site SEO Audit Handbook and Toolkit