The file that decides who can enter
The robots.txt file for AI bots is configured by stating, for each recognised crawler, which sections of the website it may read and which it may not.
It is a text file in the root of the domain: the rules must be written carefully, because a mistake can close off access to pages you actually want to be readable. In Cantù, in the province of Como, we configure it as part of our development and hosting work on the servers managed by NewVisibility.
What to consider in the rules
For companies in Como, Brianza and Milan we start from goals for visibility and confidentiality of content.
Want your robots.txt checked?
Write to us: we will reply from Cantù, for companies in Como and Brianza.
The llms.txt file is a proposed text file that summarises a website's content for language models. It is not an established standard and guarantees nothing: we assess it case by case for companies in Como and Brianza, never in place of clear content.
A slow or JavaScript-dependent website may be read incompletely by crawlers, which do not always run scripts. Serving the main content in the HTML reduces the risk. We look after the speed and rendering of bespoke websites for companies in Como and Milan.
The website architecture for AI crawlers includes a dedicated page for each topic, clear headings, readable URLs and internal links between related content. In Cantù we design the page tree together with the content, for companies in Como, Brianza and Milan.
Organization and LocalBusiness structured data describe in the website's code the name, address, contacts and field of activity in schema.org format, readable by automated systems. For a company in Cantù or Como they help state the location and service area precisely, without guaranteeing citations.
In Google Analytics you create a grouping of sources that includes the domains of AI assistants, to compare with the standard channels and with the website's goals. For companies in Como, Milan and Lugano we also check cookie consent and privacy.