How to handle AI crawlers on paywalled content

A subscription site has a problem that a free site does not. To be found, an engine has to read the article. To be paid, a reader has to be stopped before they finish it. Search engines resolved that tension with a documented mechanism about a decade ago. The AI-native engines have published nothing equivalent, which leaves publishers applying a search-era solution to a set of crawlers whose vendors have not said whether it applies.

What is the actual risk?

Not the crawl. The risk is that an engine reads the whole article, composes an answer that contains the part you were charging for, and cites you as the source — so your subscriber-only reporting reaches the reader with your name on it and none of your revenue. Traditional search had a natural brake on this, because the result was a link and the link led to your wall. An answer is not a link.

The second risk is the mirror image and easier to miss: a hard wall applied uniformly means the engine has nothing to read, so you are absent from answers in your own subject area while a free competitor with weaker material gets cited. Both failure modes are real, and a site usually has sections that belong on different sides of the line.

Where is the line between a paywall and cloaking?

Google defines cloaking as presenting different content to users and search engines with the intent to manipulate rankings and mislead users. Its spam policy then carves out paywalls explicitly: a paywall is not cloaking where Google can see the full content just as a person with access can, and where you follow the flexible sampling guidance.

The mechanism is structured data rather than a directive. You mark the article with isAccessibleForFree set to false, and identify the gated portion with a hasPart entry whose cssSelector names the class wrapping it. Google supports this on CreativeWork and its subtypes — Article, NewsArticle, Blog, WebPage and others. The markup is what turns “we served the bot more than the reader” from a spam signal into a declared arrangement. Without it, the same server behaviour is indistinguishable from the thing the policy prohibits.

That distinction is worth internalising before you extend the arrangement to anyone else, because cloaking is not defined by the mechanics of serving different bytes. It is defined by the deception. Declared asymmetry is a policy; undeclared asymmetry is a violation.

What have the AI vendors published about this?

Nothing comparable, as of this guide’s date. The crawler documentation for OpenAI, Anthropic, Perplexity and Meta covers tokens, purposes, address lists and robots handling, and does not address subscription content, sampling, or a structured-data equivalent of isAccessibleForFree. There is no documented way to tell one of those agents “this section is gated, here is a sample, cite us accordingly”.

Two honest conclusions follow. First, publishing the Google markup is still the right move, because it costs nothing and it is read by the crawler that matters most to the largest single share of your traffic. Second, you should not assume any other engine is acting on it, and you should not build a revenue plan on the assumption that it is.

What can you control per section?

Path rules, and not much else. A gated directory can be disallowed for specific agents while the free archive stays open, which is the only lever with broad support. Two things make this fiddlier than it sounds, and both are covered in how a robots file resolves: a group naming a specific agent replaces your wildcard rather than extending it, so each one has to restate the whole policy; and the most specific path match by length wins regardless of the order you wrote the lines in.

A workable shape is to keep the free archive and the article summaries under paths every agent may read, and put the gated body under a path you open selectively. That also gives you something to measure: crawl counts against the open paths tell you whether being readable is producing anything, which is the number the whole argument turns on.

What about metered paywalls?

A meter counts sessions, and a crawler has no session. Most metered implementations therefore serve a crawler the full article on every request, indefinitely, without anyone deciding that should happen. It is worth checking what your own meter does before worrying about anything subtler, because this is the most common way a subscription site turns out to be fully open to AI crawlers in practice while its operators believe otherwise.

The check is direct: request one gated URL with a crawler’s user-agent string and no cookies, and read what comes back. If the full body is there, that is what every agent in the crawler reference has been receiving.

Can I stay indexed but limit what gets shown?

For Google, yes, and that is the right instrument for this problem: nosnippet suppresses a text preview entirely, max-snippet:[number] caps its length, and the data-nosnippet attribute excludes a specific element while leaving the rest of the page quotable. Those are page-level controls over display rather than robots-level controls over access, which is exactly the distinction a paywall needs — you stay retrievable, and you decide how much may be surfaced.

No equivalent exists for the AI-native crawlers. Their vendors publish access controls and nothing that governs how much of what they read may be reproduced. For those engines the decision remains binary at the path level, which is a genuine gap rather than an oversight on your part.

What are the limits of all this?

Every mechanism here is voluntary and most of it is Google-specific. Marking content as gated does not prevent an engine from summarising it; it prevents you from being penalised for the arrangement. Nothing described here is enforcement, and a crawler that ignores robots.txt will not be stopped by structured data either.

There is also no way to price this from public information. Nobody publishes what a citation is worth against a lost subscription, so anyone offering you a ratio is guessing. What you can measure is your own crawl-to-referral figure on the open sections, which at least tells you whether the visibility you are buying with free content is producing visits at all.

Bottom line

Mark up your paywall properly so the asymmetry is declared rather than inferred. Check what your meter actually serves a crawler, because it is probably more than you think. Split free and gated content by path so that robots rules can act on them separately. And treat the absence of any AI-vendor paywall standard as a fact to plan around rather than a detail to hope about — if the reading is happening and returning nothing, the remaining questions are whether to close the door or to make the reading pay.