This beginner-level module introduces the concepts of crawlability and indexation, explaining how sitemaps and robots directives shape search engine behavior. It is intended for learners new to SEO who need a clear, practical understanding of how to ensure content is discoverable and correctly indexed.
After completing this module, learners will be able to:
Explain how search engine bots discover and process web pages.
Create and validate XML sitemaps and understand when to use HTML sitemaps.
Use robots.txt and meta robots tags appropriately to control indexing and crawling.
Interpret basic Search Console reporting related to crawling and indexing.
The beginner module is organized into short lessons, each paired with a hands-on exercise:
How crawling works: user agents, crawl budget, and politeness.
Indexation fundamentals: canonical tags, meta robots, and why not all crawled pages are indexed.
Sitemaps: XML syntax, priority, lastmod, and best practices for large and changing sites.
Robots.txt: allow/disallow directives, sitemap declaration, and common mistakes.
Validating changes with Search Console and simple log checks.
Beginner exercises emphasize practical validation and interpretation:
Generate an XML sitemap for a small site and submit it to Search Console; review coverage reports and address discovered issues.
Create a robots.txt file with an intentional disallow and test its effect with a crawler and the robots.txt tester.
Compare meta robots settings on similar pages and determine which pages should be noindexed vs canonicalized.
Beginners often accidentally block entire sites with misconfigured robots.txt, submit incomplete sitemaps, or misunderstand canonicalization. Emphasize incremental testing: make one change, verify in a staging environment and use Search Console tools and crawlers to validate results before production deployment.
Assessment consists of a practical audit where learners submit a short report identifying crawlability and indexation issues on a sample site and propose prioritized fixes. The grading rubric measures clarity of analysis, accuracy of tool usage, and the feasibility of remediation steps.
Provide templates for sitemap XML, robots.txt examples, and a checklist for pre-launch indexing verification. Offer sample Search Console screenshots and log excerpts so learners can practice interpreting common signals without needing access to a live site.
Once learners understand crawlability and indexing basics, the next modules should cover rendering and JavaScript SEO, performance optimization, and structured data — topics that directly affect how and whether content appears in search results.
A solid foundation in crawlability, indexation, and sitemaps reduces the risk of content being invisible to search engines. This beginner module equips learners with practical skills to audit and fix basic discovery issues and prepares them for more advanced technical SEO topics.