]> Google LLC.

Brandschenkestrasse 110 Zürich 8002 Switzerland garyillyes@google.com

robots.txt This document extends RFC9309 by specifying additional URL level controls through an HTTP response header and, for historical reasons, through HTML meta tags originally developed in 1996. Additionally it moves the HTTP response header out of the experimental header space (i.e., "X-") and defines the combinability of multiple headers, which was previously not possible.

Introduction While the Robots Exclusion Protocol enables service owners to control how, if at all, automated clients known as crawlers may access the URLs on their services as defined by , the protocol doesn't provide controls on how the data returned by their service may be used upon allowed access. Originally developed in 1996 and widely adopted since, the use-case control is left to URL level controls implemented in the response headers, or in case of HTML in the form of a meta tag as defined by . This document specifies these control tags, and in case of the response header field, brings it to standards compliance with . Application developers are requested to honor these tags. The tags are not a form of access authorization however.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This specification uses the following terms from : List, String, Parameter.

Specification

Robots control The URL level crawler controls are a key-value pair that can be specified two ways:

• an HTTP response header structured field as specified by .
• for historical reasons, in case of HTML, one or more meta tags as defined by the specification.

HTTP Response Header The robots-tag field is a List as defined in . Each member of the List is an Item representing a product token. Rules applicable to a product token are defined as Parameters of that Item. For historical reasons, implementors SHOULD also support the experimental field name X-Robots-Tag. The product token is either a specific string as defined in Section 2.2.1 of or the global identifier *. All rules defined in this specification are restrictive. The absence of a rule for a specific instruction constitutes as no instruction. If a product token appears multiple times in the field, or if rules are provided for both a specific product token and the global * identifier, the implementor MUST apply the union of all restrictive rules. A restriction applied to the global * token applies to all accessors regardless of whether a specific product token is present. For example, the following response header field specifies a global nosnippet rule for all accessors, and an additional noindex rule specifically for ExampleBot: The structured field in the examples is deserialized into the following objects: Implementors SHOULD impose a parsing limit on the field value to protect their systems. The parsing limit MUST be at least 8 kibibytes . For a robots-tag field that exceeds the implementor's parsing limit, the implementor MUST process the data up to that limit. Any complete and valid List members found within the processed bytes MUST be honored. Any partially transmitted or truncated List member at the limit, and all subsequent bytes in that field, MUST be ignored. This ensures consistency with the processing of robots.txt files as specified in Section 2.1.1 of .

HTML Meta Element For historical reasons the robots-tag header values may be specified by HTML service owners as an HTML meta tag. In case of the meta tag, the name attribute is used to specify the product token, and the content attribute to specify the comma separated robots-tag rules. As with the header, the product token may be a global token, robots, which signifies that the rules apply to all requestors, or a specific product token applicable to a single requestor. For example: ]]> Multiple robots meta elements may appear in a single HTML document. The implementor MUST apply the union of all rules found in all applicable elements. This includes rules specified for the global robots token and rules specified for the accessor's specific product token. Because all rules specified in this document are restrictive, they are inherently additive. An accessor cannot "opt-out" of a restriction defined in a global 'robots' tag by being mentioned in a specific tag without that restriction. If any applicable tag contains a restrictive rule, that rule MUST be honored.

Robots controls rules The possible values of the rules are:

• noindex - instructs the parser to not store the served data in its publicly accessible index.
• nosnippet - instructs the parser to not reproduce any stored data as an excerpt snippet.

The values are case insensitive. Implementors may support other rules as specified in Section 2.2.4 of .

Caching of values Implementors SHOULD link the validity of robots-tag rules to the freshness of the associated resource. Rules extracted from an HTTP response header or an HTML meta tag remain in effect until the resource is recrawled or the cached representation expires. Implementors SHOULD determine the freshness of the rules using standard HTTP cache directives, such as Cache-Control or Expires, as defined in . In the absence of explicit cache directives, implementors MAY use heuristics to determine the refresh interval, typically matching the scheduled recrawl frequency of the resource. If a resource is not recrawled due to a lack of perceived change, the last known robots-tag rules MUST continue to be honored.

Security Considerations The robots-tag is not a substitute for valid content security measures. To control access to the URL paths where the robots-tag appears, service owners SHOULD employ a valid security measure such as HTTP Authentication as defined in . The content of the robots-tag header field is not secure, private or integrity-guaranteed, and due caution should be exercised when using it. Use of Transport Layer Security () with HTTP () is currently the only end-to-end way to provide such protection. In case of a robots-tag specified in a HTML meta element, implementors should consider only the meta elements specified in the head element of the HTML document, which is generally only accessible to the service owner. Implementors who execute client-side code MUST NOT treat the post-execution DOM state as the exclusive source of truth. Instead, implementors MUST apply the union of all restrictive rules identified in both the initial HTML source and the final DOM state. A restriction present in either state MUST be honored. To protect against memory overflow attacks, implementers should enforce a limit on how much data they will parse; see [Section 3.1.1] for the lower limit.

IANA Considerations

HTTP Field Name Registration IANA is requested to register the following HTTP field name in the "Hypertext Transfer Protocol (HTTP) Field Name Registry" according to the procedures defined in Section 18.4 of : Field Name: Robots-Tag Template: None Status: permanent Reference: [This document] Comments: This field name supersedes the experimental X-Robots-Tag field.

Robots Control Rules Registry IANA is requested to create a new "Robots Control Rules" registry. This registry manages the tokens used as parameters within the Robots-Tag HTTP field and, by extension, the content attribute of the robots meta element. The registration policy for this registry is "IETF Review" or "Expert Review" as defined in . The initial entries for this registry are: Rule Name: noindex Description: Instructs the parser to not store the served data in its publicly accessible index. Reference: [This document, Section 3.1.3] Rule Name: nosnippet Description: Instructs the parser to not reproduce any stored data as an excerpt snippet. Reference: [This document, Section 3.1.3]

Deprecation of X-Robots-Tag The X-Robots-Tag field name was used prior to the standardization of the Robots-Tag field. New implementations MUST NOT use the X- prefix for this field, adhering to the principles in . While parsers SHOULD continue to support X-Robots-Tag for backward compatibility with legacy systems, it is formally deprecated by this document.

References Normative References The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields that wish to use a common syntax that is more restrictive than traditional HTTP field values. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This specification defines a model for the relationships between resources on the Web ("links") and the type of those relationships ("link relation types"). It also defines the serialisation of such links in HTTP headers with the Link header field. Historically, designers and implementers of application protocols have often distinguished between standardized and unstandardized parameters by prefixing the names of unstandardized parameters with the string "X-" or similar constructs. In practice, that convention causes more problems than it solves. Therefore, this document deprecates the convention for newly defined parameters with textual (as opposed to numerical) names in application protocols. This memo documents an Internet Best Current Practice. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References

Acknowledgments TODO acknowledge.