SEO2026-05-17
robots.txt
Also known as: robots.txt / クローラー制御ファイル
A text file placed at the site root that specifies which URLs crawlers may or may not access, using User-agent and Disallow/Allow directives to control crawl scope.
Overview
robots.txt is an industry-standard file based on the Robots Exclusion Protocol. Common uses include blocking admin areas, staging environments, and duplicate content pages. Note that Disallow prevents crawling, not indexing — pages already indexed remain indexed.
Controlling AI Crawlers
AI training crawlers such as GPTBot, ClaudeBot, and Google-Extended can be individually controlled. See the Google-Extended and llms.txt entries, along with our LLMO technical implementation guide.
Related Columns
SEO
llms.txt and Structured Data — Technical Implementation Guide for LLMO | Complete Checklist for AI Search Optimization
Comprehensive technical implementation guide for LLMO. From llms.txt creation and schema.org structured data to JSON-LD patterns, sitemap optimization, and Next.js implementation examples—a complete guide to technical SEO for the AI era.
SEO
SEO-Friendly Website Design Fundamentals: Building a Search Engine Optimized Site Structure
Learn the fundamentals of designing websites that rank well in search engines. From semantic HTML and structured data to internal linking strategies, our Shinagawa-based SEO-focused web team shares practical know-how.
SEO
Decoding Google's Official AI Optimization Guide — What Actually Works for AI Overviews / AI Mode, and Which GEO/AEO Tactics Google Just Killed (May 2026)
An evidence-based reading of Google's official Generative AI optimization guide, published May 15, 2026 in Search Central. Covers how AI Overviews and AI Mode actually work, what Google explicitly recommends, and — crucially — what Google explicitly denies: llms.txt, AI-only structured data, content chunking, AI-specific rewrites. Plus Search Console measurement reality (consolidated counts, the April 2026 impression-counting bug), and how GEO/AEO/LLMO map onto plain SEO.
Feel free to contact us
Contact Us