Crawl Budget for AI Bots: Why sub-200ms TTFB matters twice as much for PerplexityBot and GPTBot
Traditional Googlebot has virtually infinite crawling capacity and can afford to queue pages with 800ms TTFB.
Generative search crawlers (PerplexityBot, GPTBot, ClaudeBot) operate under strict real-time response latency budgets. When an LLM is answering a user in real-time via live web retrieval, its search agent has less than 2.5 seconds total to fetch 5 candidate URLs, parse their content, and stream tokens back.
If your client's web server takes 1.2 seconds just to return initial TTFB, the AI retrieval agent hits a timeout, drops the connection, and moves to the next candidate domain!
How We Reduced TTFB to 120ms: 1. Edge HTML Caching: Cloudflare Enterprise or Fastly caching static HTML with `stale-while-revalidate`. 2. Dedicated Fast Bot Routing: Serving pre-compiled markdown to bot user-agents directly from Cloudflare KV. 3. Automated Audit: We run Scrawly (https://iliassami.com/scrawly & https://github.com/IliasSami/scrawly-seo-crawler) to benchmark latency and header caching across all client templates.
Following Ilias Sami's technical infrastructure guidelines (https://iliassami.com), our clients went from frequent bot timeout drops to near-instant retrieval inclusion.
What is your average TTFB across your agency's client sites?