The SitemapBuilder bot

You are here because SitemapBuilder is in your server logs. This page tells you what it does and how to control it.

What it is

Sitemap Builder is a free tool that makes diagrams from an XML sitemap. A person types the address of a sitemap, and the tool downloads that sitemap. It is not a search engine and it keeps no copy of your data.

The browser of the person downloads the sitemap directly when your server permits that (CORS). If not, our server downloads it. Our server identifies itself with this user agent:

SitemapBuilder/1.0 (+https://www.sitemapbuilder.co.uk/bot)

What it downloads

• Your /robots.txt file, at most one time each hour.
• XML sitemap files and sitemap index files, only when a person asks for them. It refuses all other content.
• It does not download your pages, and it does not follow the links in your sitemap.
• It downloads at most three files at the same time, and it has rate limits for each person.

How to control it

Our server obeys robots.txt (RFC 9309). It uses the group for SitemapBuilder. If there is no such group, it uses the group for *. To block all downloads by our server:

User-agent: SitemapBuilder
Disallow: /

To permit your sitemap files when the * group denies them:

User-agent: SitemapBuilder
Allow: /sitemap.xml
Allow: /sitemaps/

Our server keeps your robots.txt data for one hour, thus a change can take one hour to have an effect.

A direct download from the browser of a person does not come from our server and does not use our user agent. If robots.txt denies a sitemap, the tool tells the person before the browser tries a direct download.

Contact

Report a problem on GitHub.

Back to Sitemap Builder