Making Your Directory Machine-Readable: Feeds, llms.txt and Crawlers
A directory is structured data dressed as a website. What schema markup covers and misses, what llms.txt actually is, sitemaps and feeds at scale, robots.txt and AI crawlers, and how to check what a machine really sees.
Your AI chat can build this directory.
Describe the niche, watch the agent design the fields and fill the catalogue. Free plan, no card.
A directory is a database wearing a website costume. The visitor sees cards and filters; underneath are rows, typed fields and relationships. That gap matters more every year, because a growing share of the traffic you care about never renders your CSS. Crawlers, answer engines and agents read your pages as data, and what they extract decides whether you are quoted, ignored or quietly copied.
This is the machine layer: structured data, sitemaps, feeds, robots directives, and the newer conventions people are still arguing about. Here is what each does, and what none of them promise.
What schema markup already covers
Structured data is the most established part of the machine layer, and for a directory it is close to free: your listings are already typed, so emitting JSON-LD is a template exercise rather than a writing one.
What it handles well. The identity and attributes of a single entity — LocalBusiness and its subtypes, Product or SoftwareApplication, Organization — plus BreadcrumbList for hierarchy, ItemList for a category page's ordering, Offer for pricing, and AggregateRating where you actually hold ratings.
What it does not cover. Most of what a directory is for.
- Your taxonomy's meaning. Schema can say a listing belongs to a category. It cannot express why your "lightweight CRMs" category has a threshold, or what you excluded.
- Comparative judgement. People read a directory because you ranked and filtered. There is no vocabulary for "we recommend this for teams under ten people".
- Freshness in a useful sense.
dateModifiedexists and you should set it honestly, but it says nothing about which fields were verified when. - Your terms. Structured data carries no licence, attribution requirement or reuse restriction.
And it guarantees nothing. Valid markup makes a page eligible for certain rich results, and search engines say plainly that eligibility is not entitlement. Schema makes your data legible; it is not a lever that produces placement.
Most platforms generate listing markup for you. Confirm what yours emits and check it against the visible content — markup that disagrees with the page is worse than none.
llms.txt: what it is, and what it is not
llms.txt is a proposed convention: a Markdown file at the root of your domain giving a language model a curated map of your site — a short description, then a linked list of the pages that matter, sometimes with one-line summaries. A companion llms-full.txt holds expanded content in one file. The reasoning is straightforward: HTML pages are full of navigation, scripts and boilerplate, and a clean Markdown index is cheaper to read than a crawl.
Be clear about its status. It is a community convention, not a standard, and adoption by the major model and answer providers is uneven and not guaranteed. Publishing one does not get you indexed, cited or crawled. Anyone telling you it does is selling something.
Why it can still be worth twenty minutes. It costs almost nothing, it is trivially reversible, and it forces a useful exercise: writing down in plain language what your directory covers and which fifty pages a machine should read first.
What to put in one. A paragraph on scope — what is in, what is out, how entries are chosen. A link to each top-level category, to your collections, to your methodology page. What not to put in: all ten thousand listings. A file listing everything is a sitemap, and you already have one.
Keep the framing honest: a low-cost bet on a convention that may or may not matter, not part of your traffic plan.
Sitemaps and feeds for a large catalogue
Sitemaps are the boring, load-bearing part, and directories break them more often than most sites, because of scale.
The hard limits. A single sitemap file holds at most 50,000 URLs and 50MB uncompressed. Past that you need a sitemap index pointing at multiple files. Verify this once: a truncated sitemap on a large directory is an expensive silent failure.
Split by page type. One sitemap for listings, one for categories, one for collections, one for editorial. When indexation drops you can see which type is affected.
Use lastmod honestly. A sitemap claiming every URL changed today, every day, is a signal crawlers learn to discount.
Feeds are for change, not inventory. An RSS or JSON Feed of new and recently updated listings is the cheapest way to tell aggregators, partners and interested humans that something moved. It is not a substitute for a sitemap and should not try to be a full export.
When to expose a feed or an API, and when that is just donating your dataset
A judgement call, and the answer depends on what your moat actually is.
Expose more when your value is curation and audience. If your defensibility is the ongoing work — verification, editorial judgement, a community, a claim flow — a copy of today's rows is a snapshot that decays. A partner feed then brings referrals and citations a locked-down site never gets.
Expose less when the dataset is the product. If you spent six months compiling something nobody else has, a public JSON endpoint hands a competitor that work in one request. Directory data is not protected the way people assume, and a scraper does not stop to read your terms.
Middle positions that usually work.
- Publish a partial feed. Name, category, URL and a short description; hold back the fields that took the real research.
- Require a key for volume. Free, self-serve, rate-limited. The point is not the key — it is seeing who is pulling, and cutting off the one making 40,000 requests a night.
- State licence and attribution terms on a linkable page and in the feed itself. Terms will not stop a determined scraper, but they change what a legitimate business does.
- Seed canary entries. A handful of deliberately unique, harmless records makes copying provable later. Cartographers have done this for a century.
And accept the floor. If a page is publicly readable, it is extractable. The realistic goal is to make legitimate reuse easy and attributed, and bulk copying inconvenient.
robots.txt and AI crawlers
robots.txt is a request, not a fence. Well-behaved crawlers honour it; badly behaved ones ignore it.
The user agents are separable. Search crawling, AI training and answer-engine retrieval are often different agents from the same operator, and blocking one does not block another. Google-Extended, for example, governs certain generative uses and does not affect ordinary Search indexing. Agent names change, so read the operators' own documentation rather than copying a rules block from a blog post.
The trade is real. Blocking every AI crawler protects your data from being absorbed and removes you from the systems that increasingly answer the questions your directory used to answer. For most directories, being read and cited is the point.
What blocking does not do. It does not remove content already crawled, bind agents fetching a page on a user's behalf, or stop scrapers pretending to be browsers. If content must not be public, put it behind authentication.
Keep it simple. A long, clever robots.txt is a liability. Directories mostly need: allow the crawlers you want, disallow faceted-filter URL patterns that generate near-duplicates, point at your sitemap index.
How to check what a machine actually sees
Assume nothing. Every claim above is checkable in minutes.
Fetch the raw HTML. curl -s https://yourdirectory.com/listing/example | less. If the listing name, description and attributes are not in that output, they do not exist for anything that does not run JavaScript.
Turn JavaScript off and load a category page. What remains is roughly what a conservative reader gets.
Validate your structured data with a schema validator and a rich-results test — on a listing page, a category page and the homepage. They emit different markup, and only one of them tends to get checked.
Read your server logs. Which bots hit you, how often, which paths. The only source of truth about who is actually reading your directory.
Request your own sitemap and count the URLs against your listing count. A gap means something is being excluded.
Check indexability first. A perfect machine layer on a noindex host is decoration. DirectoryFast's free subdomain is noindex by design as an anti-spam measure, so organic and AI visibility require the Pro plan on a verified custom domain. Whatever you use, confirm the header and meta tag say what you think they say.
Where this goes wrong
Treating llms.txt as an indexing lever. It is a cheap, optional convention. Publish one if you like; do not build a plan around it.
Marking up claims the page does not make. Ratings you do not have, prices you did not verify, FAQ markup on a page with no FAQ. It gets pages ignored, and it is dishonest anyway.
Shipping a full public API before deciding what your moat is. Easy to open, hard to close once three sites depend on it.
Blocking every AI crawler on reflex. You are opting out of the surfaces where people ask exactly the question your directory answers. Decide it deliberately, not defensively.
Letting faceted URLs into the sitemap. Thousands of near-duplicate filter combinations dilute crawl budget and teach crawlers your site is mostly noise.
Never verifying the output. Machine-layer bugs are silent: a truncated sitemap, markup that stopped rendering after a template change, a stray noindex. Check quarterly.
FAQ
Does publishing llms.txt improve my chances of being cited in AI answers?
There is no guaranteed effect, and no major provider commits to consuming it. It is cheap and harmless, and the exercise of writing it is useful. Do not treat it as a ranking factor.
Is schema markup still worth it for a directory?
Yes. It is the most widely consumed machine-readable layer you have and it is templated once across every listing. It just does not guarantee rich results.
Should I block GPTBot, ClaudeBot and the rest?
Only if you have a specific reason — a licensing business, or genuinely proprietary compiled data. Most directories want to be read and cited. Check the current agent names in each operator's documentation before writing rules.
Do I need both a sitemap and a feed?
A sitemap, always. A feed only if you want to notify partners, aggregators or subscribers about new and updated entries. They do different jobs.
How do I stop competitors scraping my listings?
You cannot, fully. You can make bulk copying inconvenient with rate limits and keyed access, make reuse terms explicit, and seed canary entries so copying is provable. Plan for extraction rather than pretending it is preventable.
Build a directory that machines can read →