Advisory

Meta’s terms are on hold. Review who can access your website.

Use Meta’s pause to review crawler and AI access across the business, enforce the policy and verify it quarterly.

Meta made the right call in pausing its revised business and ad terms. Advertisers saw the pause notice in Ads Manager on 12 November. The UK competition investigation into Meta’s terms remains open. Advertisers have more time to consider the proposed changes, including who would carry responsibility when Meta’s AI changes an ad. The 20 November deadline has gone. Meta’s existing ad terms already give it permission to crawl advertiser websites.

Under the Self-Serve Ad Terms dated 22 August 2025, placing an ad order gives Meta permission to crawl content on the advertiser’s domains for purposes that include research and training. The Ad Creative Generative AI Terms dated 6 May 2024 separately allow Meta to crawl URLs advertisers supply. The Commercial Terms dated 6 March 2024 contain no crawling clause. The paused revision would have extended that permission to websites associated with a business’s products or services. Search Engine Journal’s comparison

I raised that broader permission in Meta wants your website in the bargain and discussed it with Mumbrella. My October checklist for Australian marketers started with the terms. This time, I’d start with a question for the business. Who do we want accessing our information, and what are we prepared to let them use it for?

Make crawling intentional

I want search engines to find my work. I’m happy for an AI assistant to send someone to an article that answers their question. Giving a model provider material to train on is a separate decision. Make that decision while you still control access to the source. Once data has been used to train a model, changing your rules can restrict future access. It can’t undo the training that already happened. Taking the source page down is a bit like trying to take the egg out of a baked cake.

Review every crawler and enforce the decisions that matter

Start with the traffic your site actually receives. Pull edge and origin logs for your main site, landing pages, documentation and other public hosts. Identify every crawler you can, including search engines, link preview bots, ad crawlers, AI training crawlers and assistants fetching a page at someone’s request. Record what each one accesses and why you’re willing to allow it. Keep unidentified traffic visible in the review.

Platform What to check
Meta Meta-ExternalAds for advertising and business products, Meta-ExternalAgent for uses including foundation model training, and Meta-WebIndexer for Meta AI search. Meta crawler documentation
OpenAI GPTBot for potential training use, OAI-SearchBot for ChatGPT search and ChatGPT-User for user-requested visits. OpenAI says robots.txt rules may not apply to those user-requested visits. OpenAI crawler documentation
Anthropic ClaudeBot for potential training use, Claude-SearchBot for search and Claude-User for user-requested retrieval. Anthropic crawler documentation
Google Review Googlebot for Search and Google-Extended for specified Gemini training and grounding uses. Google-Extended is a directive token. You won’t see it as a separate crawler in your logs. Google crawler documentation

Review AI and LLM access at the edge. Use Cloudflare or your equivalent edge service to see what reaches each host and enforce the blocks you’ve agreed to. Match robots.txt to the same policy so crawlers that honour it know your preferences. Include user-requested retrieval and any feeds or APIs you deliberately expose to AI services. Those routes may have separate access controls. A crawler name in a request is a starting point. Use verified bot identification where available. Cloudflare’s AI Crawl Control documentation

Check what happened. After a change, look at which bots requested which URLs, whether the edge allowed or blocked them, and what the origin served. A blocked request shows that enforcement worked. A successful request to a page you meant to exclude needs a closer look. Check that the search, ad and link-preview functions you want still work. This is the step teams tend to skip. It’s the one that tells you whether your policy worked in practice.

Give it an executive owner and review it quarterly

Your company’s information goes beyond the marketing site. It includes product documentation, support content, research and material you may have licensed from someone else. Give one executive accountability for the policy. Have them agree it with marketing, web engineering, security, legal and the teams that own the information. Marketing can explain the benefit of being found. The company needs to decide what it is willing to share in return.

Repeat the review every quarter. Keep a dated record of the crawlers and access routes you reviewed, the decisions and reasons, who approved them, what changed and what the logs showed afterwards. Use the record to pick up new bots, new platform terms and changes to the controls you rely on.

Microsoft is a good example. If you rely on NoArchive or NoCache, put Bing on the next review. Its new AI preferences start taking effect as it recrawls from 1 February 2027. Microsoft says to publish the replacement training preference by 1 March 2027. Eligible sites can set a temporary grounding preference in Bing Webmaster Tools from 15 December 2026. Bing’s transition guidance

For Meta advertisers, I’d also keep checking the ads that actually run. Record the approved creative and automation settings, then look at what customers see. Agree who can pause an unexpected variation and when the client needs to approve a change. I covered that separately in Meta’s AI ad changes put creative approval under pressure.

The question I’d put to the executive owner each quarter is simple. Who accessed our information, what did we agree they could use it for, and what evidence do we have that our controls worked?