# katana.network
#
# This file used to be `Disallow: /` — correct while the site was a prototype on a
# public URL, and the strongest of the four things that were blocking indexing. It
# also mattered most: a crawler that is refused here never fetches a page at all, so
# the meta tags and the X-Robots-Tag header were never even read.
#
# The API routes stay out. They answer JSON, they are proxies rather than content, and
# one of them is metered by upstream quota — nothing there wants crawling. They also
# send `X-Robots-Tag: noindex` of their own, so this is belt and braces.
#
# The legal fallback is excluded too. It is served for any `/legal/` address this build
# does not know about, so it would be indexed as a thin, near-identical page. It also
# carries its own `noindex`, which is what covers the arbitrary paths this file cannot
# name. The blog has no fallback: an unknown post slug is a real 404.

User-agent: *
Allow: /
Disallow: /api/
Disallow: /legal/fallback.html

Sitemap: https://katana.network/sitemap.xml