Home Blog Contact

Seo Tutorial — Lesson 2 of 10 [ Updated: 12 Aug 2026 ]

# How Search Engines Work: Crawling, Indexing, and Ranking

In the previous tutorial, you learned what SEO means and why Search Engine Optimization is important for a website.

Before learning keywords, content optimization, or technical SEO, you need to understand one of the most important foundations of SEO:

How do search engines actually find, understand, and display webpages?

When you publish a new article on your website, it does not automatically appear in Google Search.

A search engine needs to discover the page, process and understand its content, and decide whether the page should be included in its search index. When someone searches for something, the search engine then determines which results are most relevant and useful for that particular query.

For beginners, this process can be simplified into three major stages:

Crawling → Indexing → Ranking

In this tutorial, you will learn each stage step by step.

1. What Happens When You Publish a New Webpage?

Imagine that you publish a new article on your website:

"How to Learn HTML for Beginners"

You publish the article and expect people to find it through Google.

But several things need to happen first.

A simplified process looks like this:

Your webpage is published

↓

Search engine discovers the URL

↓

Search engine crawls the page

↓

Search engine processes and understands the page

↓

Page may be added to the search index

↓

A user performs a relevant search

↓

Search engine evaluates available results

↓

Your page may appear in the search results

This is why SEO is not simply about publishing an article and waiting for visitors.

Search engines need to discover and process your content.

2. What Is Crawling?

Crawling is the process through which search engine systems discover webpages and follow links to find additional URLs.

Search engines use automated programs to discover and revisit webpages.

These programs are commonly called:

  • Crawlers

  • Bots

  • Spiders

  • Search engine crawlers

For example, Google's crawler is commonly referred to as Googlebot.

A crawler can discover a webpage through different pathways.

For example:

Page A → Link to Page B

If a crawler already knows about Page A and can access the link to Page B, it may discover Page B by following that link.

3. A Simple Crawling Example

Imagine your website has three pages:

Homepage

↓

SEO Tutorial

↓

Keyword Research Tutorial

Your homepage contains a link to the SEO Tutorial.

The SEO Tutorial contains a link to the Keyword Research Tutorial.

A crawler may follow this structure:

Homepage → SEO Tutorial → Keyword Research Tutorial

This demonstrates why website structure and internal linking are important.

When related pages are connected logically, users and search engines can more easily navigate your website.

4. How Does Google Discover a New Page?

There is no single method through which a search engine discovers every URL.

A new page can potentially be discovered through:

  • Links from other webpages

  • Internal links

  • XML sitemaps

  • Previously known URLs

  • Other discovery mechanisms

For example, suppose you publish:

example.com/seo/keyword-research/

If this URL is included in your XML sitemap, it can provide a useful signal about the existence of that URL.

Similarly, if another page on your website links to it, a crawler may discover it by following that internal link.

This is why website architecture and internal linking are important parts of SEO.

5. What Is an XML Sitemap?

An XML sitemap is a file that provides information about important URLs on a website.

A simple sitemap may contain URLs such as:

  • Homepage

  • Blog pages

  • Tutorial pages

  • Category pages

For example:

example.com/

example.com/blog/

example.com/seo-tutorial/

example.com/keyword-research/

A sitemap can help search engines discover URLs, especially on larger or more complex websites.

However, submitting a sitemap does not guarantee that every URL will be crawled or indexed.

This distinction is important.

Discovery is not the same as indexing.

6. What Is Indexing?

After a search engine discovers and crawls a webpage, it can process information from that page.

If the page is eligible and the search engine chooses to include it, the page can be added to its search index.

This process is called indexing.

Think of a search engine's index as a huge organized collection of information about webpages.

When a page is indexed, the search engine has information about that page available for potential use in search results.

7. Crawling vs Indexing

Beginners often confuse these two concepts.

They are different.

Crawling

The search engine discovers and accesses a URL.

Indexing

The search engine processes the page and may include it in its search index.

A page can be:

Crawled but not indexed.

This means a search engine has accessed the page but has not included it in its search index.

Therefore:

Crawled ≠ Indexed

This is an important concept that you will use later when learning technical SEO.

8. Why Might a Page Not Be Indexed?

There can be many reasons why a webpage does not appear in a search engine's index.

For example:

  • The page may have technical problems.

  • The page may be blocked from crawling.

  • The page may contain a noindex directive.

  • The URL may be considered a duplicate.

  • The page may not provide enough useful value.

  • The search engine may not have processed the page yet.

  • Other technical or quality-related factors may be involved.

Do not assume that simply publishing a page guarantees indexing.

Later in this course, we will study indexing problems in much more detail.

9. What Is Ranking?

After understanding crawling and indexing, we reach the third major concept:

Ranking

When someone searches for something, a search engine has to determine which results should be shown and in what order.

For example, imagine a user searches:

"how to learn HTML"

Thousands or even millions of webpages may contain information related to HTML.

The search engine needs to determine which available results are most relevant and useful for that particular search.

The ordering of results is commonly referred to as ranking.

10. A Simple Ranking Example

Suppose five websites have pages about:

"How to Learn HTML for Beginners"

A search engine may evaluate many different signals and characteristics to determine which results are most appropriate for a particular query.

The results might appear approximately like:

1. Website A

2. Website B

3. Website C

4. Website D

5. Website E

The exact ranking can vary depending on the search query, user context, location, device, freshness, and many other factors.

This is why SEO cannot guarantee that a particular page will always remain in one fixed position.

11. Does Google Rank Pages Only by Keywords?

No.

Keywords are important because they help communicate the subject of a page, but search engines use much more information than keyword repetition.

Search engines can consider many aspects related to relevance and usefulness.

These can include things such as:

  • The meaning of the query

  • Content relevance

  • Content quality

  • Page experience

  • Website structure

  • Links

  • Freshness where relevant

  • Other search-specific signals

This is one reason why keyword stuffing is not a good SEO strategy.

Simply repeating a keyword hundreds of times does not make a page more useful.

12. Understanding the Three Stages

At this point, you should understand the basic difference between the three stages.

Stage 1 — Crawling

"Can the search engine discover and access this page?"

Stage 2 — Indexing

"Can the search engine process this page and include it in its index?"

Stage 3 — Ranking

"When someone searches, how relevant and useful is this page compared with other available results?"

A simple way to remember this is:

Find → Understand → Show

Crawling helps search engines find pages.

Indexing helps them store and understand information about pages.

Ranking determines which results to show and in what order for a search.

13. Why This Matters for SEO

Understanding this process helps you make better SEO decisions.

For example, imagine you publish an excellent article, but the page cannot be accessed properly by search engine crawlers.

The content quality alone may not solve the problem.

Or imagine that your page is accessible but is not included in the search index.

Again, simply having a keyword in the article does not solve the issue.

This gives us an important SEO principle:

Good content needs a technically accessible and understandable website.

14. What Can Help Search Engines Discover Your Pages?

Several website practices can support discovery.

Internal Links

Link relevant pages together.

Example:

SEO Tutorial → Keyword Research Tutorial

This creates a logical relationship between your content.

XML Sitemap

Maintain an appropriate XML sitemap containing important URLs.

Clear Website Structure

Organize your pages logically.

For example:

Homepage

→ SEO

→ Keyword Research

→ On-Page SEO

→ Technical SEO

A clear structure makes navigation easier.

External Links

Links from other websites can also contribute to URL discovery.

However, you should not create artificial links simply for the purpose of forcing search engines to discover a page.

15. What Can Help Search Engines Understand Your Pages?

Search engines need to understand what your page is about.

You can make this easier by creating clear webpages.

Important elements include:

  • Descriptive page titles

  • Logical headings

  • Clear content

  • Relevant internal links

  • Descriptive URLs

  • Useful image descriptions where appropriate

  • Structured data when relevant

  • Clear website navigation

For example, compare these two page titles:

Page 1: "Article 25"

Page 2: "How to Choose Keywords for SEO"

The second title gives users and search engines much clearer information about the page topic.

16. Search Query and Search Results

Now let's connect everything together.

Suppose a user searches:

"how to do keyword research"

The search engine receives the query.

It then needs to identify relevant information from its available index and determine appropriate results.

A simplified process is:

User enters a query

↓

Search engine interprets the query

↓

Relevant indexed pages are considered

↓

Search systems evaluate the available results

↓

Search results are displayed

Your job as an SEO learner is not to control every part of this process.

Instead, your job is to create useful content and a website that search engines can access, understand, and potentially surface for relevant searches.

17. Crawling, Indexing, and Ranking Example

Let's use a real-world-style example.

You create a webpage:

incomeva.com/seo/keyword-research-for-beginners/

Step 1 — Crawling

A search engine discovers the URL through your website structure, sitemap, or another discovery path and accesses the page.

Step 2 — Processing and Indexing

The search engine analyzes the page and may include it in its index.

Step 3 — User Search

Someone searches:

"keyword research for beginners"

Step 4 — Ranking

The search engine considers relevant available pages and determines which results are most appropriate for the query.

If your page is considered relevant and useful, it may appear in the search results.

That is the basic relationship between your webpage and search.

18. What Beginners Often Get Wrong

Mistake 1: "I published my article, so it must be indexed."

Not necessarily.

Publishing a page and getting it indexed are separate processes.

Mistake 2: "If my page is indexed, it will rank number one."

No.

Indexing only means the page may be available for consideration in search results.

Ranking is a separate process.

Mistake 3: "Adding the keyword many times will guarantee ranking."

No.

Keyword stuffing does not create a useful page.

Mistake 4: "Submitting a sitemap guarantees ranking."

No.

A sitemap helps with URL discovery, but it does not guarantee indexing or rankings.

Mistake 5: "SEO is only about Google."

SEO concepts can apply to multiple search engines and search platforms, although Google is particularly important for many websites.

19. A Simple SEO Model for Beginners

Remember this model:

Create a webpage

↓

Make it accessible

↓

Help search engines discover it

↓

Help search engines understand it

↓

Make the content useful for the intended search

↓

Monitor its performance

↓

Improve it over time

This is a much better way to think about SEO than simply asking:

"How can I rank number one?"

20. What You Should Remember From This Lesson

You do not need to memorize every technical detail yet.

For now, remember these three words:

Crawling

Search engines discover and access webpages.

Indexing

Search engines process pages and may include them in their index.

Ranking

Search engines determine which relevant results to show and their order for a particular search.

The basic flow is:

Crawling → Indexing → Ranking

Understanding this flow will make the next SEO lessons much easier.

Quick Practice

Before moving to the next lesson, answer these questions yourself.

Question 1

What does SEO stand for?

Question 2

What is crawling?

Question 3

What is indexing?

Question 4

What is ranking?

Question 5

Is a crawled page automatically guaranteed to be indexed?

Question 6

Does indexing guarantee a number-one ranking?

Try answering these without looking back at the lesson.

Key Takeaways

  • Search engines need to discover webpages.

  • Crawling is the process of discovering and accessing webpages.

  • Indexing is the process of processing pages and potentially including them in a search index.

  • Ranking determines the order in which relevant search results may be displayed.

  • Crawling, indexing, and ranking are different processes.

  • An XML sitemap can help search engines discover important URLs.

  • Internal links can help search engines discover and understand relationships between pages.

  • Publishing a webpage does not guarantee indexing.

  • Indexing does not guarantee high rankings.

  • SEO is about building useful, accessible, understandable, and relevant webpages.

Next Tutorial

Now that you understand how search engines work, the next step is to understand the most important concept behind successful SEO:

Search Intent — Understanding What Users Really Want When They Search

In the next lesson, we will learn how to identify what a person actually wants when they type a query into a search engine.

← Previous Next →