The marketing team spent weeks on the content. The pages are beautiful. Search Console says: "Discovered, currently not indexed".

Or worse: the new site launches, traffic drops off a cliff, and someone eventually finds a single line left over from staging, Disallow: /, sitting in robots.txt.

A lot of SEO is not about keywords at all. It's plumbing: status codes, URLs, headers, markup and speed. That's developer territory, and the decisions we make quietly decide whether great content can be found. The easiest way to understand it is to follow a page through the search engine, one stage at a time.

Stage 1: Discovery, or "how does Google know this page exists?"

Search engines find URLs in two main ways: by following links from pages they already know, and by reading sitemaps you give them.

That has a developer consequence that's easy to miss. Crawlers follow real links, meaning <a href="..."> elements. A "link" that's a div with a click handler, or navigation that only appears after JavaScript runs a function, may never be followed. If a page matters, at least one real link should point to it.

A sitemap is the other half: a list of the URLs you want found, with an optional last-modified date. It doesn't guarantee indexing, but it makes discovery reliable, especially for new sites and pages deep in your structure.

A sitemap in Laravel, simplified

For a blog or a product catalogue, a small route and a Blade view are enough:

Route::get('/sitemap.xml', function () {
    $posts = Post::query()
        ->whereNotNull('published_at')
        ->latest('updated_at')
        ->get(['slug', 'updated_at']);

    return response()
        ->view('sitemap', ['posts' => $posts])
        ->header('Content-Type', 'application/xml');
});

{!! '<?xml version="1.0" encoding="UTF-8"?>' !!}
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
@foreach ($posts as $post)
  <url>
    <loc>{{ route('blog.show', $post->slug) }}</loc>
    <lastmod>{{ $post->updated_at->toAtomString() }}</lastmod>
  </url>
@endforeach
</urlset>

Three details that matter:

• The XML declaration is echoed rather than written literally, because if PHP's short_open_tag setting is on, a raw <?xml in the compiled view is read as PHP.
• Only include pages you want indexed: published, canonical, returning 200. A sitemap full of redirects and drafts teaches search engines not to trust it.
• Keep lastmod honest. Google has said it ignores priority and changefreq, but it uses lastmod when it's consistently accurate. Setting every page to "today" on every request makes it useless.

A single sitemap can hold up to 50,000 URLs; beyond that, split it and use a sitemap index. Then reference it in robots.txt with a Sitemap: line and submit it in Google Search Console. The Google sitemap docs cover the format. For bigger sites, a package like spatie/laravel-sitemap saves you some work.

Stage 2: Crawling, where robots.txt lives

Next, the crawler fetches the page. Before it does, it checks robots.txt, and this is where the most misunderstood rule in SEO lives:

And for staging sites, don't rely on either. Put them behind authentication. It's the only option that also keeps people out.

Status codes matter at this stage too:

• 301 for content that moved permanently. When you change a slug, redirect the old URL instead of letting it 404.
• 404 or 410 for content that's really gone.
• Never 200 for "not found". A friendly error page returned with status 200 is a "soft 404", and search engines may index it as a real, thin page.

Stage 3: Rendering

Google renders pages with a modern Chromium, so JavaScript-heavy sites can be indexed. But rendering is extra work that can happen later than the initial crawl, and other search engines and link-preview bots are less capable.

The safe rule: important content, titles, meta tags and links should be in the HTML the server sends. Server-rendered Blade does this naturally. For single-page apps, server-side rendering (Inertia supports SSR, as do Next.js and Nuxt) or pre-rendering gets you there.

Stage 4: Indexing, and the duplicate problem

Now the search engine decides whether to store the page and which URL represents it. Developers create duplicate URLs constantly without meaning to:

• /blog/post and /blog/post/
• http://, https://, with and without www
• ?sort=price, ?page=1, ?utm_source=newsletter

Pick one version, redirect the obvious variants (protocol, www, trailing slash) with 301s, and add a canonical tag for the rest:

<link rel="canonical" href="{{ url()->current() }}">

url()->current() drops the query string, which is usually what you want for tracking parameters. For paginated or filtered pages that deserve their own listing, decide case by case. The canonical is a strong hint, not a command, so it works best when your redirects and internal links agree with it.

Stage 5: Understanding and ranking

Content and relevance decide most of ranking, and that part belongs to writers. But developers control the signals that help a search engine understand and present the page:

• A unique <title> and meta description per page, generated from the content, not one site-wide default. The description doesn't directly affect ranking, but it's often the snippet people read before deciding to click.
• Semantic HTML: one clear h1, real headings, descriptive link text, alt text on meaningful images.
• Structured data in JSON-LD, using schema.org types like Article, Product or BreadcrumbList. It helps search engines understand the page and can make it eligible for richer results. Build the array in PHP, output it with @json, and check it with Google's Rich Results Test.
• Performance. Core Web Vitals measure loading (LCP), responsiveness (INP) and visual stability (CLS). Fast pages won't rescue weak content, but slow ones hurt users and are part of how Google evaluates page experience. Image sizes, caching and avoiding layout shifts usually matter most.

A developer's SEO checklist

• Real <a href> links to every important page.
• An accurate sitemap, referenced in robots.txt and submitted to Search Console.
• noindex for pages to hide, authentication for staging, and no stray Disallow: / at launch.
• Correct status codes: 301 for moves, real 404s, no soft 404s.
• One canonical URL per page, backed by redirects.
• Key content and meta tags in the server-rendered HTML.
• Unique titles and descriptions, semantic headings, structured data.
• Core Web Vitals watched, not guessed.

None of this replaces good content. But it makes sure good content gets a fair chance, and that's our part of the job.

What's the most surprising SEO problem you've found that turned out to be a code issue, not a content one?