Crawl website to create visual sitemap
Plans available to: All
Using this feature you can crawl a website to create a visual sitemap. Perfect for a website audit at the start of a redesign project!
It crawls the provided site and creates a structured visual sitemap based on the URL structure of the pages it finds.
- From your sitemap list page click on ‘New Project’ and select “Generate by Crawling a Website” which you will find in the dropdown.
OR You can also select “Sitemap Finder” from the navigation bar on the left side of the screen.
- Paste the URL to your website.
- Click ‘Create’ or ‘Find my sitemap’ depending on the option you selected above, and the crawling begins!
Please allow up to 5 minutes for a crawl to take place for a 250-page website. We are adhering to best practice crawling which means not crawling too fast which can overload smaller sites.
You can close the menu and continue to use WriteMaps as normal. Your sitemap will appear in your sitemap list when crawling has finished.
Troubleshooting:
Page Limit is 250 – As extra extra large visual sitemaps become unwieldy, we’ll create a sitemap with the first 250 web pages that we find. If you are looking to crawl larger sites than this, other tools would be better suited.
Website not found – This means the URL provided (and the variations we also attempted) returned 404 Page Not Found errors. Try and visit the website yourself, and if you can view it in a logged-out state then WriteMaps Crawler should be able to crawl it. Copy and Paste the URL to prevent typo errors.
Blocked by Robots.txt – adhering to best practice crawling means we respect what a robots.txt file says. If the site you are trying to crawl disallows a wide range of crawlers in it’s robots.txt file (normally found at yourwebsite.com/robots.txt) then this crawling feature may not work).
If your website has disallowed robots in your robots.txt file and you want to only allow WriteMaps, add ‘User-agent’: ‘WriteMaps Bot’ and ‘Disallow: ‘ to your robots.txt file.
My sitemap looks very flat – This crawler looks at your URLs which are non-structured data, and attempts to make a structured diagram out of them. If most of your webpages are not in sub-directories and are ‘flat’ e.g. yourwebsite.com/product1, yourwebsite.com/product2, then the sitemap will appear flat.