FastHTML x llms.txt

[Answer AI](https://answer.ai) recently proposed [llms.txt](https://llmstxt.org), which aims to standardize the usage of a `/llms.txt` file that provides Large Language Models (LLMs) with information about a website. It's becoming common for LLMs to have browser access, but websites are generally designed for humans. Websites also have a wide variety of formats, quirks and the limited context window of LLMs makes it difficult to process websites in its entirety. This is why an `llms.txt` file is essential for the modern web, where we use LLMs to process the web in bulk.

The idea is that every website will have a public `llms.txt` and `llms-full.txt` file. The file should provide clear and concise information in Markdown about the website with links to important pages. The file [can be simple](https://llmstxt.org/llms.txt) or [very elaborate](https://docs.anthropic.com/llms-full.txt). There is a [Kaggle Dataset by Meg Risdal](https://www.kaggle.com/datasets/mrisdal/llms-txt) available with many examples of `llms.txt` files. There is also a [directory of llms.txt files](https://directory.llmstxt.cloud/).

Since [FastHTML](https://fastht.ml) is also created by Answer AI, I thought it would be a good fit to explain how to add `llms.txt` to a FastHTML app. The simplest approach is to create a folder called `static` in the root of your FastHTML project and add a `llms.txt` file in it. Here are my current [static/llms.txt](/llms.txt) and [static/llms-full.txt](/llms-full.txt) files. This is what it looks like:

# Carlo Lepelaars

Personal website of Carlo Lepelaars

## Pages

- [Home](https://carlo.ai) - Personal introduction and overview of projects
- [Posts](https://carlo.ai/posts) - Blog Posts
- [TIL](https://carlo.ai/til) - Today-I-Learned (TIL) mini-posts
- [Books](https://carlo.ai/books) - Curated reading list with searchable interface
- [Bachelor Thesis](https://carlo.ai/thesis) - Bachelor Thesis on Financial Sentiment Analysis

## Online Profiles

- [LinkedIn](https://www.linkedin.com/in/carlolepelaars)
- [GitHub](https://github.com/Carlolepelaars)
- [X](https://x.com/carlolepelaars)
- [Bluesky](https://bsky.app/profile/carlolepelaars.bsky.social)
- [YouTube](https://www.youtube.com/@carlolepelaars)
- [Kaggle](https://www.kaggle.com/carlolepelaars)
- [Buy me a coffee](https://buymeacoffee.com/carlolepelaars)
            

Quick tip: To show raw unformatted text in FastHTML like above, use `Pre()` to avoid the text being rendered.

In addition to having the `llms.txt` file in a `static` folder, it is important to specify `static_path` in `fast_app`. from fasthtml.common import * app, rt = fast_app(..., static_path="static")

And that's all you need! There is also a proposal for adding [llms-ctx.txt (Example)](https://docs.fastht.ml/llms-ctx.txt), which uses an XML-based structure instead of Markdown and is generally more detailed. If you are interested, this XML file can be generated with [a CLI command provided by llms-txt](https://llmstxt.org/intro.html#cli). Though most people seem to be settling on Markdown, the [standardization is still being actively discussed](https://github.com/AnswerDotAI/llms-txt/tree/main). I'm a big fan of the `llms.txt` proposal. Large AI companies like [Anthropic](https://docs.anthropic.com/llms-full.txt), [Perplexity](https://docs.perplexity.ai/llms-full.txt) and [Cursor](https://docs.cursor.com/llms-full.txt) have already developed solid `llms.txt` files. You can find plenty of inspiration in the [llms.txt directory](https://directory.llmstxt.cloud) and [llms.txt Kaggle Dataset](https://www.kaggle.com/datasets/mrisdal/llms-txt).