Sitemap
A file containing a list of all important URLs on a website that helps search engines index pages more effectively.
What is a sitemap?
A sitemap is a file, usually in XML format, that contains a list of all the important URLs on your website. It is primarily used by search engine robots (e.g., Google) to crawl and index your website more effectively.
Use in automation
In the context of RAG systems, a sitemap serves as a starting point for automatically downloading website content into a vector database.