Skip to content
Home » AI Crawler and Templates

AI Crawler and Templates

AI Crawler Support efficiently crawls any web content you want. It also comes with built-in templates for various scenarios, eliminating the need for coding skills . With just simple input, it can automatically collect, organize information from the entire web, and generate professional analysis reports.

1. Use AI crawlers

Find [AI Crawler] in the left navigation bar,

  1. Enter the “URL” you want to crawl. One URL per line, you can press Enter to input multiple URLs;
  2. Enter the “Prompt” to describe the data structure you want to extract.

If the crawled website “Requires a profile with proxy or login configuration”, after enabling this feature, select the profile where you have completed the account and proxy configuration (if you don’t know how to configure the account and proxy, you can click “View Tutorial”). Finally, click “Confirm Add”.

Click “More Settings” to enable “Extract main content only”; and set “Timeout (s)”, and if the timeout is reached, stop crawling content. After completing all the above settings, click “Run”to start the crawler.

Scroll down to view the AI crawling results. Click “Preview” (preview is not supported when crawling multiple URLs) to preview the crawling results.

  1. Click the “Rerun” icon to reuse the settings of the corresponding task for recrawling.
  2. Click the “Download” icon to save the crawled content resultsin file formatto your computer.

2. Use the recommended AI crawler template

Currently, 11 AI crawler “recommended templates” are available. Select the template that meets your needs based on the comments for each template.

For example, after clicking the “Research Report Generator” template, enter the “Research Topic”, you can also enable “Requires a profile with proxy or login configuration”, and finally click “Run”.

For example, after clicking on the “Marketing Copy Generation Assistant” template,

  1. Enter the “URL” of the website you need to analyze,
  2. Enter “Marketing Theme” and “Marketing Goal”.

You can also enable “Requires a profile with proxy or login configuration”, and finally click “Run”.

After successful execution , click “Preview” to view the marketing copy obtained through crawling and analysis using the “Marketing Copy Generation Assistant” template.