# PicMany — a read-only viewer for public Instagram pages.
User-agent: *
Allow: /

# The browser's own endpoints. They hold no markup, and every hit is a direct
# call into the slow client-side provider.
Disallow: /api/

# '?cursor=' continuations are for visitors pressing Load more, not for
# crawlers, and they are the dearest documents the site renders. A place
# continuation is ~477 KB of paid upstream against ~111 KB for its page one, so
# crawling them would roughly quintuple what the place tree costs. Keyword and
# reels windows are cheaper (~27 KB) but chain on window after window under
# Instagram's opaque cursors, a URL space nothing on this site bounds. What the
# continuations hold is more posts, which matter less than the profile pages
# this site is indexed for; that trade is deliberate, not an oversight. The
# Load more links are rel="nofollow" to match, and the pages themselves still
# answer 'noindex, follow' with a canonical to page one for any client that
# fetches them regardless.
Disallow: /*?cursor=

# '?page=' is deliberately NOT disallowed. Numbered pages are finite, and for the
# locations directory they are the only path to most of the tree: page one lists
# 96 countries out of several hundred, and the rest exist nowhere else on the
# site. Those pages already answer 'noindex, follow' with a canonical to page
# one, which is exactly the instruction that lets a crawler walk them for links
# without indexing the listing itself. Disallowing them would make a crawler
# unable to read that instruction at all.

# Redirects and probes, not content. Anchored, because a group rule is a
# prefix match and profile pages live at the root: bare 'Disallow: /search'
# also beats 'Allow: /' on longest-match for every username beginning with
# 'search', and the site would serve those an indexable 200 that no crawler is
# allowed to fetch.
Disallow: /search$
Disallow: /search?
Disallow: /health$

Sitemap: https://picmany.com/sitemap.xml