34 lines
1.7 KiB
Plaintext
34 lines
1.7 KiB
Plaintext
---
|
|
id: crawl-all-links-on-website
|
|
title: Crawl all links on website
|
|
---
|
|
|
|
import ApiLink from '@site/src/components/ApiLink';
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
|
|
|
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_bs.py';
|
|
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_pw.py';
|
|
|
|
This example uses the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> helper to add new links to the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> as the crawler navigates from page to page. By automatically discovering and enqueuing all links on a given page, the crawler can systematically scrape an entire website. This approach is ideal for web scraping tasks where you need to collect data from multiple interconnected pages.
|
|
|
|
:::tip
|
|
|
|
If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the `strategy` option, which is an instance of the `EnqueueStrategy` type alias. You can find more info about this option in the [Crawl website with relative links](./crawl-website-with-relative-links) example.
|
|
|
|
:::
|
|
|
|
<Tabs groupId="main">
|
|
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
|
<RunnableCodeBlock className="language-python" language="python">
|
|
{BeautifulSoupExample}
|
|
</RunnableCodeBlock>
|
|
</TabItem>
|
|
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
|
<RunnableCodeBlock className="language-python" language="python">
|
|
{PlaywrightExample}
|
|
</RunnableCodeBlock>
|
|
</TabItem>
|
|
</Tabs>
|