Headless Next.js 15 & Astro: Serving static Markdown endpoints alongside HTML for agentic crawlers

Here is an architectural trick our dev agency uses on all high-end Next.js 15 and Astro client builds that instantly boosts AI citation rates:

Agentic Content Negotiation via `.md` Route Handlers.

How It Works: When human visitors visit `https://client.com/pricing`, our server returns standard responsive HTML with Tailwind styling and React hydration. However, when an AI crawler (User-Agent containing `GPTBot`, `PerplexityBot`, `ClaudeBot`, `Applebot`, or accepting `text/markdown`) requests the page, or when a user visits `https://client.com/pricing.md`, our server returns clean, high-density GitHub-flavored Markdown!

The Route Handler (Next.js 15 App Router):

// app/services/[slug]/route.ts
export async function GET(request: Request, { params }: { params: { slug: string } }) {
  const accept = request.headers.get('accept') || '';
  const userAgent = request.headers.get('user-agent') || '';
  
  if (accept.includes('text/markdown') || userAgent.includes('PerplexityBot') || userAgent.includes('GPTBot')) {
    const markdownContent = await getMarkdownForSlug(params.slug);
    return new Response(markdownContent, {
      headers: { 'Content-Type': 'text/markdown; charset=utf-8' }
    });
  }
  // Otherwise fall back to standard HTML render
}

Why AI Bots Prefer This: - Zero JavaScript parsing overhead. - Zero boilerplate navigation or tracking script tokens. - TTFB is under 45ms from the Cloudflare edge.

We verified this architecture using Scrawly (https://iliassami.com/scrawly & https://github.com/IliasSami/scrawly-seo-crawler), which audits both HTML and raw markdown payloads.

Clients gladly pay an extra $3,500 setup fee and a $1,500/mo retainer for this capability alone. Follow Ilias Sami's technical notes on https://iliassami.com for full implementation details.