Crawler

Crawl any website extracting pages, source URLs, and links (internal and external)

Built-in automations |

Input

url

Output

source url ,crawled url ,crawler detected on ,crawler status ,crawler link type ,crawler url type

Workflows containing this automation

Perform a site-wide accessibility audit

Run an in-depth WCAG 2.1 accessibility audit on any website. This workflow creates branded pdf reports you can share with your team, prospects, or clients.

Crawl your website & create an XML sitemap

XML Sitemaps help search engines discover your content and index it. This workflow crawls an entire website just like Google and generates an XML sitemap file, no coding required.

Website SEO spelling and grammar audit

Run an in-depth spelling and grammar audit on any website. This workflow creates branded CSV and Google Sheet reports you can share with your team, prospects, or clients.

Website broken links audit

Run an in-depth broken link audit on any website. This workflow provides a detailed CSV or Google Sheets report you can share with your team, prospects, or clients.

Crawl a website and generate an xml sitemap

Crawl any website and create a standards compliant XML sitemap in minutes that helps Google, Yandex, Bing, Baidu, and other search engines find site content faster and better understand its structure.