icon_url

Crawler

Crawl any website extracting pages, source URLs, and links (internal and external)
Built-in automations |
Crawl any website extracting pages, source URLs, and links (internal and external)
Built-in automations |
Crawl any website extracting pages, source URLs, and links (internal and external).
This automation shows the response code for each URL and consumes an automation credit per URL.
Can be used as a starting point for other automations including the SEO meta tag, email scraper, screenshot, image or file downloader, keyword finder or audit automations. (1 automation credit per crawled URL)
Input

url

Output

source url ,crawled url ,crawler detected on ,crawler status ,crawler link type ,crawler url type

How to use Crawler automation
Our crawler automation will help you to crawl any website extracting pages, source URLs, and links (both internal and external).

This automation shows the response code for each URL and consumes an automation credit per URL.

Can be used as a starting point for other automations including the SEO meta tag, email scraper, screenshot, image or file downloader, keyword finder or audit automations.

Different ways to use our Crawler automation:

  • This automation can be used as a starting point of your Hexomatic workflow.

How to use our Crawler automation:

Step 1: Create a blank workflow.
Step 2: Add the Crawler automation. After enter the source URL, the URLs to be ignored, URL types, the proxy mode and set the URL limit.
Step 3: Click continue to run or schedule the workflow.
Workflows containing Crawler automation
Website broken links audit
Run an in-depth broken link audit on any website. This workflow provides a detailed CSV or Google Sheets report you can share with your team, prospects, or clients.
Marketing
Crawler
Crawl your website & create an XML sitemap
XML Sitemaps help search engines discover your content and index it. This workflow crawls an entire website just like Google and generates an XML sitemap file, no coding required.
Marketing
Crawler
XML sitemap generator
Perform a site-wide accessibility audit
Run an in-depth WCAG 2.1 accessibility audit on any website. This workflow creates branded pdf reports you can share with your team, prospects, or clients.
Marketing
Crawler
Accessibility audit
Website SEO spelling and grammar audit
Run an in-depth spelling and grammar audit on any website. This workflow creates branded CSV and Google Sheet reports you can share with your team, prospects, or clients.
Marketing
Crawler
Grammar & spelling audit