For a regulated financial services website, robots.txt is not a technical afterthought. It can affect whether prospective clients find a mortgage guide, whether Google can render a calculator page, and whether automated visitors reach areas that should never have been public in the first place.
The central principle is simple: use robots.txt to guide cooperative crawlers, but never treat it as a security, privacy or compliance control. A disallowed URL can still be requested directly, linked elsewhere, logged, or disclosed through a technical mistake. Client records, pre-release promotions, internal documents and authenticated journeys need proper access controls.
That distinction matters as firms consider AI crawlers alongside Google and Bing. The right approach is not a blanket block or blanket allow. It is a documented, page-type-by-page-type decision that preserves helpful, approved public material and protects information that is restricted, personal or not ready for publication.
Start with what robots.txt can and cannot do
A robots.txt file sits at the root of a host, normally at https://www.example.co.uk/robots.txt. It gives instructions to crawlers that choose to follow the Robots Exclusion Protocol. Common directives include User-agent, Disallow, Allow and Sitemap.
It is useful for reducing crawl of low-value public URL patterns: internal search results, duplicate filter combinations, non-essential parameters, test directories accidentally left public, or large document libraries that are not intended for search. It is not an indexing removal tool and is not access control.
If a page must stay out of search but can safely be fetched by a crawler, a noindex meta robots tag or X-Robots-Tag is usually the more appropriate indexing signal. If the page must not be seen by unauthorised people, require authentication or remove it from the public web entirely. Blocking a URL in robots.txt before a search engine can see its noindex instruction can prevent that instruction being processed.
Google’s own documentation remains the most useful technical starting point for crawler directives and indexing controls: Google Search Central. Treat it as a product-specific reference, not a substitute for legal, information-security or compliance review.
Classify content before writing directives
In practice, most avoidable errors occur because a team starts with a crawler name instead of a content inventory. I recommend assigning every meaningful section of the site to one of four categories before changing the file.
| Content category | Preferred treatment | Typical examples |
|---|---|---|
| Approved public content | Allow crawling and index where appropriate; include canonical URLs in XML sitemaps. | Service pages, educational guides, fees, complaints information, author pages. |
| Public but not search-led | Allow access where needed; use noindex where search visibility is not wanted. | Thank-you pages, campaign variants, utility confirmation pages. |
| Low-value crawl traps | Control through URL design, canonicals, noindex or selective robots rules after testing. | Site search, endless parameters, some faceted filters. |
| Restricted or confidential content | Remove public access; protect with authentication and server-side controls. | Client portals, fact finds, adviser workspaces, draft promotions, exports. |
This is closely connected to site architecture. A clear distinction between public advice, service information, conversion journeys and restricted systems reduces the need for blunt crawl rules later. See this compliance-first website architecture framework for UK financial services for the wider structure.
Write a narrow, explainable robots.txt policy
A restrained file is generally easier to audit than a long list of historic exceptions. A basic pattern might disallow a genuinely non-public-by-purpose path and declare a sitemap:
User-agent: *
Disallow: /internal-search/
Disallow: /preview/
Sitemap: https://www.example.co.uk/sitemap.xmlThe syntax is easy; the operational decision is not. Never copy path rules from another site without checking redirects, JavaScript assets, CSS, API endpoints, alternate hosts and live URLs. A rule that blocks /assets/, for example, may stop a search engine rendering content properly. A broad block on /tools/ may conceal a useful affordability calculator that the firm intends to make discoverable.
Be especially careful with faceted navigation. Disallowing every parameter can hide useful category pages, while allowing every combination can waste crawl capacity and create duplicate pages. The better choice depends on the platform and whether a filtered URL has a legitimate user and search purpose. This guide to faceted navigation SEO for financial services sets out the related indexing decisions.
Manage AI crawlers without confusing search controls
“AI crawler” is not one uniform category. Some bots support search products, some collect material for model development, and some may not reliably identify themselves or honour robots.txt at all. Their purposes and published controls can change. Maintain an approved list of recognised user agents only where the relevant provider publishes verifiable documentation, and review it periodically.
Do not assume that blocking a training-related crawler removes a firm from Google Search, Bing, AI-generated answers or every third-party AI product. Equally, allowing a crawler does not guarantee use, citation or referral. Those outcomes are outside a site owner’s control.
For public, evergreen and compliance-approved guides, allowing mainstream search crawling is often the sensible baseline. Clear explanations of eligibility, risks, fees, complaints routes and terminology can be useful to people and discoverable systems alike. However, any claim on these pages still needs normal financial-promotion review. A content approval record should not become weaker simply because material is framed as an FAQ or glossary entry. For guidance on structuring answer-ready material responsibly, see how to create citation-ready financial services FAQs.
Where a provider offers a documented crawler-specific opt-out, decide on it through governance rather than an SEO preference alone. Consider the content’s public status, intellectual-property position, client expectations, privacy risks, commercial value and the firm’s stated policy. Record the decision, owner, date and rationale.
Keep staging, portals and draft promotions off the public internet
A staging site should be protected with authentication, network restrictions or equivalent server-side access controls. A robots.txt disallow and a noindex tag are sensible belt-and-braces measures for a temporary public test environment, but neither makes an exposed environment safe. Search engines and third parties cannot be relied upon to keep a discovered staging URL private.
Client portals and adviser areas require the same discipline. They should not merely sit behind a hidden navigation item or an unlinked URL. Apply authenticated access, least-privilege permissions, secure session handling and appropriate logging. Do not place personal financial information in query strings, downloadable filenames, page titles, open directories or client-side code.
Privacy obligations do not disappear because a page is technically crawlable. The UK GDPR and wider data-protection framework require a lawful, fair and secure approach to personal data; the Information Commissioner’s Office is the authoritative UK source for guidance. In my view, teams should regard any URL that can be reached without login as potentially discoverable, cached or shared, even where robots.txt asks compliant bots not to visit it.
Use sitemaps as a positive publication signal
An XML sitemap is not a command to index, but it is a valuable declaration of the canonical URLs the firm wants search engines to consider. Include live, indexable, approved pages that return a successful response and are internally supported. Exclude portal URLs, noindex pages, redirects, error pages, thin parameter variants and withdrawn promotions.
For larger firms, separate sitemaps by content type can make governance easier: core services, advice resources, locations, documents and news. This helps reviewers assess what is being presented for discovery and makes post-release checks more efficient. It also creates a useful control point when a promotion expires or regulatory wording changes.
Do not use a sitemap to compensate for poor internal linking or unclear intent. Each URL should have a defined purpose, audience and review owner. The same discipline supports the SEO governance workflow for FCA-regulated firms.
Monitor actual crawler behaviour, not assumptions
Check robots.txt after every deployment, CMS migration, domain change and major release. Then inspect actual outcomes. Google Search Console can show indexing issues and robots-related reports for Google, while server logs reveal requests from a wider range of user agents. Verify claimed bots where possible using the provider’s documented method rather than trusting a user-agent string alone.
A useful monthly review covers:
- changes to robots.txt, sitemap files and HTTP headers;
- newly discovered URLs, indexed pages and unexpected exclusions;
- 404s, redirect chains and crawler hits on retired campaigns;
- requests to staging, portal or preview paths;
- high-volume parameter URLs and crawl patterns that indicate a technical issue; and
- pages awaiting compliance review, expiry or removal.
Log files may contain IP addresses, identifiers or URLs with personal data. Limit access, minimise retention and ensure the monitoring process is assessed through the firm’s privacy and security procedures. For the operational SEO side, this log file analysis guide for financial services websites is a useful companion.
A practical governance process
Assign clear ownership. Marketing or SEO can propose crawl and indexing changes; web development implements and tests them; compliance approves promotional content decisions; data protection and security assess restricted-data risks. One accountable owner should maintain the current robots.txt and sitemap register.
- Inventory URL patterns, subdomains, documents and third-party tools.
- Classify each as public/indexable, public/noindex, crawl-controlled or access-restricted.
- Document the purpose, owner, technical control and review date.
- Test changes on a properly protected non-production environment.
- Deploy with version control, rollback steps and a post-release crawl check.
- Review quarterly and immediately after platform, product or policy changes.
The FCA’s official website should inform the firm’s broader regulatory interpretation, but robots rules themselves are an implementation control, not proof of compliance. Escalate uncertain cases, particularly where a URL contains a financial promotion, personal data or an eligibility outcome.
FAQ and conclusion
Can robots.txt protect client portal information?
No. It is a voluntary crawler instruction, not a barrier to access. Use authentication and server-side security. Keep portals and personal documents out of public sitemaps and navigation.
Should an FCA-regulated firm block all AI crawlers?
Not automatically. Review each recognised crawler’s documented purpose and the status of the content involved. Public, approved educational pages may be suitable for access; confidential, draft or personal information must be protected regardless of crawler rules.
Does disallowing a page guarantee it will not appear in Google?
No. A blocked URL may still be known through links or other signals. Use noindex for accessible pages that should not appear in search, and remove or restrict access where confidentiality is required.
How often should robots.txt be reviewed?
Review it at least quarterly, after releases and whenever a new portal, CMS feature, product area, document library or third-party tool is introduced.
Conclusion: The safest approach is purposeful rather than defensive. Let search engines reach accurate, approved public pages that help consumers assess services and support themselves. Control duplicate or low-value crawl paths carefully. Put genuinely restricted material behind real access controls. With a maintained URL inventory, clear ownership, tested directives and log-based monitoring, robots.txt becomes a useful part of financial-services SEO governance without being asked to solve privacy or security problems it cannot solve.