Generating an llms.txt file for your Astro Website

Every day more and more people are using ChatGPT, Perplexity and other AI tools to search and summarise content. As discussed in one my of earlier posts on SEO to SAO, It is only a matter of time when the majority of the visits to website and blogs will coe from AI agents,

When it comes to making your content accessible to language models, there’s a simpler way than complex crawling solutions - especially for statically generated sites built with Astro. In this post, I’ll walk you through creating an llms.txt file that exposes your blog content to LLMs in a clean, structured format.

Whats llms.txt?

An llms.txt file is conceptually similar to robots.txt but designed specifically for language models. It provides a structured, text-based representation of your content that’s easy for LLMs to parse and understand.

LLMs are not very good at navigating through multiple pages through links, instead LLMs are very good at scraping content from a single page and storing it into its memory.

Crawling tools like Crawl4AI may be an overkill

Tools like Crawl4AI offer powerful website crawling capabilities for LLMs, they while they are ideal for generating LLMs.txt for dynamic sites, they are a overkill for static sites.

For Astro sites especially, where content is typically stored as markdown files with frontmatter, you already have perfectly structured content ready to be exposed directly.

Implementing an llms.txt Endpoint in Astro

Here’s how you can genereate LLMs.txt files for your Astro site.

Create a file at src/pages/llms.txt.ts (or src/pages/api/llms.txt.ts depending on your Astro configuration). and put in the following piece of code.

import { getCollection } from "astro:content";
import type { APIRoute } from "astro";

export const GET: APIRoute = async () => {
    const posts = await getCollection("blog"); // adjust "blog" to your collection name
    const content = `# Vinci Rufus Blog\n
    ${posts
            .map(
                (post) =>
                    `# ${post.data.title}\n\n
                https://www.vincirufus.com/postss/${post.slug}.md \n
                ${post.data.description} \n
                  ${post.body}
                }`
            )
            .join("\n\n")}`;
    return new Response(content, {
        headers: { "Content-Type": "text/plain; charset=utf-8" },
    });
};

This code creates an API endpoint that:

1. Fetches all posts from your blog collection
2. Creates a text file starting with your blog title
3. For each post, includes:
   • The post title as a heading
   • A link to the post
   • The full post content

Build, Run and Deploy

Build and run your website / blog using the regular

pnpm dev or pnpm build commands

Your generated file will be available at https://yourdomain.com/llms.txt.

That’s as simple as it can get. A really simple way for you to make your sites content available to LLMs.