同步完整源码 - 2026-05-24

This commit is contained in:
xiaoxue
2026-05-24 23:06:32 +08:00
commit 2b4682de8e
764 changed files with 98522 additions and 0 deletions
+40
View File
@@ -0,0 +1,40 @@
---
id: add-data-to-dataset
title: Add data to dataset
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_bs.py';
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_pw.py';
import DatasetExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_dataset.py';
This example demonstrates how to store extracted data into datasets using the <ApiLink to="class/PushDataFunction#open">`context.push_data`</ApiLink> helper function. If the specified dataset does not already exist, it will be created automatically. Additionally, you can save data to custom datasets by providing `dataset_id` or `dataset_name` parameters to the <ApiLink to="class/PushDataFunction#open">`push_data`</ApiLink> function.
<Tabs groupId="main">
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
Each item in the dataset will be stored in its own file within the following directory:
```text
{PROJECT_FOLDER}/storage/datasets/default/
```
For more control, you can also open a dataset manually using the asynchronous constructor <ApiLink to="class/Dataset#open">`Dataset.open`</ApiLink>
<RunnableCodeBlock className="language-python" language="python">
{DatasetExample}
</RunnableCodeBlock>
+15
View File
@@ -0,0 +1,15 @@
---
id: beautifulsoup-crawler
title: BeautifulSoup crawler
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler.py';
This example demonstrates how to use <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> to crawl a list of URLs, load each URL using a plain HTTP request, parse the HTML using the [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) library and extract some data from it - the page title and all `<h1>`, `<h2>` and `<h3>` tags. This setup is perfect for scraping specific elements from web pages. Thanks to the well-known BeautifulSoup, you can easily navigate the HTML structure and retrieve the data you need with minimal code. It also shows how you can add optional pre-navigation hook to the crawler. Pre-navigation hooks are user defined functions that execute before sending the request.
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
@@ -0,0 +1,19 @@
---
id: capture-screenshots-using-playwright
title: Capture screenshots using Playwright
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import CaptureScreenshotExample from '!!raw-loader!roa-loader!./code_examples/capture_screenshot_using_playwright.py';
This example demonstrates how to capture screenshots of web pages using <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and store them in the key-value store.
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> is configured to automate the browsing and interaction with web pages. It uses headless Chromium as the browser type to perform these tasks. Each web page specified in the initial list of URLs is visited sequentially, and a screenshot of the page is captured using Playwright's `page.screenshot()` method.
The captured screenshots are stored in the key-value store, which is suitable for managing and storing files in various formats. In this case, screenshots are stored as PNG images with a unique key generated from the URL of the page.
<RunnableCodeBlock className="language-python" language="python">
{CaptureScreenshotExample}
</RunnableCodeBlock>
@@ -0,0 +1,27 @@
---
id: capturing-page-snapshots-with-error-snapshotter
title: Capturing page snapshots with ErrorSnapshotter
description: How to capture page snapshots on errors.
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import ParselCrawlerWithErrorSnapshotter from '!!raw-loader!roa-loader!./code_examples/parsel_crawler_with_error_snapshotter.py';
import PlaywrightCrawlerWithErrorSnapshotter from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_error_snapshotter.py';
This example demonstrates how to capture page snapshots on first occurrence of each unique error. The capturing happens automatically if you set `save_error_snapshots=True` in the crawler's <ApiLink to="class/Statistics">`Statistics`</ApiLink>. The error snapshot can contain `html` file and `jpeg` file that are created from the page where the unhandled exception was raised. Captured error snapshot files are saved to the default key-value store. Both <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and [HTTP crawlers](../guides/http-crawlers) are capable of capturing the html file, but only <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> is able to capture page screenshot as well.
<Tabs>
<TabItem value="ParselCrawler" label="ParselCrawler">
<RunnableCodeBlock className="language-python" language="python">
{ ParselCrawlerWithErrorSnapshotter }
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{ PlaywrightCrawlerWithErrorSnapshotter }
</RunnableCodeBlock>
</TabItem>
</Tabs>
@@ -0,0 +1,66 @@
import asyncio
from datetime import timedelta
from playwright.async_api import Route
from crawlee.crawlers import (
AdaptivePlaywrightCrawler,
AdaptivePlaywrightCrawlingContext,
AdaptivePlaywrightPreNavCrawlingContext,
)
async def main() -> None:
# Crawler created by following factory method will use `beautifulsoup`
# for parsing static content.
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
max_requests_per_crawl=10, # Limit the max requests per crawl.
playwright_crawler_specific_kwargs={'headless': False},
)
@crawler.router.default_handler
async def request_handler_for_label(
context: AdaptivePlaywrightCrawlingContext,
) -> None:
# Do some processing using `parsed_content`
context.log.info(context.parsed_content.title)
# Locate element h2 within 5 seconds
h2 = await context.query_selector_one('h2', timedelta(milliseconds=5000))
# Do stuff with element found by the selector
context.log.info(h2)
# Find more links and enqueue them.
await context.enqueue_links()
# Save some data.
await context.push_data({'Visited url': context.request.url})
@crawler.pre_navigation_hook
async def hook(context: AdaptivePlaywrightPreNavCrawlingContext) -> None:
"""Hook executed both in static sub crawler and playwright sub crawler.
Trying to access `context.page` in this hook would raise `AdaptiveContextError`
for pages crawled without playwright."""
context.log.info(f'pre navigation hook for: {context.request.url} ...')
@crawler.pre_navigation_hook(playwright_only=True)
async def hook_playwright(context: AdaptivePlaywrightPreNavCrawlingContext) -> None:
"""Hook executed only in playwright sub crawler.
It is safe to access `page` object.
"""
async def some_routing_function(route: Route) -> None:
await route.continue_()
await context.page.route('*/**', some_routing_function)
context.log.info(
f'Playwright only pre navigation hook for: {context.request.url} ...'
)
# Run the crawler with the initial list of URLs.
await crawler.run(['https://warehouse-theme-metal.myshopify.com/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,35 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': context.soup.title.string if context.soup.title else None,
'html': str(context.soup)[:1000],
}
# Push the extracted data to the default dataset.
await context.push_data(data)
# Run the crawler with the initial list of requests.
await crawler.run(
[
'https://crawlee.dev',
'https://apify.com',
'https://example.com',
]
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,15 @@
import asyncio
from crawlee.storages import Dataset
async def main() -> None:
# Open dataset manually using asynchronous constructor open().
dataset = await Dataset.open()
# Interact with dataset directly.
await dataset.push_data({'key': 'value'})
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,35 @@
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
crawler = PlaywrightCrawler()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': await context.page.title(),
'html': str(await context.page.content())[:1000],
}
# Push the extracted data to the default dataset.
await context.push_data(data)
# Run the crawler with the initial list of requests.
await crawler.run(
[
'https://crawlee.dev',
'https://apify.com',
'https://example.com',
]
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,57 @@
import asyncio
from datetime import timedelta
from crawlee.crawlers import (
BasicCrawlingContext,
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)
async def main() -> None:
# Create an instance of the BeautifulSoupCrawler class, a crawler that automatically
# loads the URLs and parses their HTML using the BeautifulSoup library.
crawler = BeautifulSoupCrawler(
# On error, retry each page at most once.
max_request_retries=1,
# Increase the timeout for processing each page to 30 seconds.
request_handler_timeout=timedelta(seconds=30),
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
# The handler receives a context parameter, providing various properties and
# helper methods. Here are a few key ones we use for demonstration:
# - request: an instance of the Request class containing details such as the URL
# being crawled and the HTTP method used.
# - soup: the BeautifulSoup object containing the parsed HTML of the response.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': context.soup.title.string if context.soup.title else None,
'h1s': [h1.text for h1 in context.soup.find_all('h1')],
'h2s': [h2.text for h2 in context.soup.find_all('h2')],
'h3s': [h3.text for h3 in context.soup.find_all('h3')],
}
# Push the extracted data to the default dataset. In local configuration,
# the data will be stored as JSON files in ./storage/datasets/default.
await context.push_data(data)
# Register pre navigation hook which will be called before each request.
# This hook is optional and does not need to be defined at all.
@crawler.pre_navigation_hook
async def some_hook(context: BasicCrawlingContext) -> None:
pass
# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,56 @@
import asyncio
from crawlee._types import BasicCrawlingContext
from crawlee.crawlers import BeautifulSoupCrawler
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Keep the crawler alive even when there are no requests to be processed now.
keep_alive=True,
)
def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None:
"""Stop crawler once specific url is visited.
Example of guard condition to stop the crawler."""
if context.request.url == 'https://crawlee.dev/docs/examples':
crawler.stop(
'Stop crawler that was in keep_alive state after specific url was visite'
)
else:
context.log.info('keep_alive=True, waiting for more requests to come.')
async def add_request_later(url: str, after_s: int) -> None:
"""Add requests to the queue after some time. Can be done by external code."""
# Just an example of request being added to the crawler later,
# when it is waiting due to `keep_alive=True`.
await asyncio.sleep(after_s)
await crawler.add_requests([url])
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BasicCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Stop crawler if some guard condition has been met.
stop_crawler_if_url_visited(context)
# Start some tasks that will add some requests later to simulate real situation,
# where requests are added later by external code.
add_request_later_task1 = asyncio.create_task(
add_request_later(url='https://crawlee.dev', after_s=1)
)
add_request_later_task2 = asyncio.create_task(
add_request_later(url='https://crawlee.dev/docs/examples', after_s=5)
)
# Run the crawler without the initial list of requests.
# Wait for more requests to be added to the queue later due to `keep_alive=True`.
await crawler.run()
await asyncio.gather(add_request_later_task1, add_request_later_task2)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,41 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
# Create an instance of the BeautifulSoupCrawler class, a crawler that automatically
# loads the URLs and parses their HTML using the BeautifulSoup library.
crawler = BeautifulSoupCrawler()
# Define the default request handler, which will be called for every request.
# The handler receives a context parameter, providing various properties and
# helper methods. Here are a few key ones we use for demonstration:
# - request: an instance of the Request class containing details such as the URL
# being crawled and the HTTP method used.
# - soup: the BeautifulSoup object containing the parsed HTML of the response.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Create custom condition to stop crawler once it finds what it is looking for.
if 'crawlee' in context.request.url:
crawler.stop(
reason='Manual stop of crawler after finding `crawlee` in the url.'
)
# Extract data from the page.
data = {
'url': context.request.url,
}
# Push the extracted data to the default dataset. In local configuration,
# the data will be stored as JSON files in ./storage/datasets/default.
await context.push_data(data)
# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,47 @@
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.storages import KeyValueStore
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
# Headless mode, set to False to see the browser in action.
headless=False,
# Browser types supported by Playwright.
browser_type='chromium',
)
# Open the default key-value store.
kvs = await KeyValueStore.open()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Capture the screenshot of the page using Playwright's API.
screenshot = await context.page.screenshot()
name = context.request.url.split('/')[-1]
# Store the screenshot in the key-value store.
await kvs.set_value(
key=f'screenshot-{name}',
value=screenshot,
content_type='image/png',
)
# Run the crawler with the initial list of URLs.
await crawler.run(
[
'https://crawlee.dev',
'https://apify.com',
'https://example.com',
]
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,91 @@
from __future__ import annotations
import asyncio
import inspect
import logging
import sys
from typing import TYPE_CHECKING
from loguru import logger
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
if TYPE_CHECKING:
from loguru import Record
# Configure loguru interceptor to capture standard logging output
class InterceptHandler(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
# Get corresponding Loguru level if it exists
try:
level: str | int = logger.level(record.levelname).name
except ValueError:
level = record.levelno
# Find caller from where originated the logged message
frame, depth = inspect.currentframe(), 0
while frame:
filename = frame.f_code.co_filename
is_logging = filename == logging.__file__
is_frozen = 'importlib' in filename and '_bootstrap' in filename
if depth > 0 and not (is_logging | is_frozen):
break
frame = frame.f_back
depth += 1
dummy_record = logging.LogRecord('dummy', 0, 'dummy', 0, 'dummy', None, None)
standard_attrs = set(dummy_record.__dict__.keys())
extra_dict = {
key: value
for key, value in record.__dict__.items()
if key not in standard_attrs
}
(
logger.bind(**extra_dict)
.opt(depth=depth, exception=record.exc_info)
.patch(lambda loguru_record: loguru_record.update({'name': record.name}))
.log(level, record.getMessage())
)
# Configure loguru formatter
def formatter(record: Record) -> str:
basic_format = '[{name}] | <level>{level: ^8}</level> | - {message}'
if record['extra']:
basic_format = basic_format + ' {extra}'
return f'{basic_format}\n'
# Remove default loguru logger
logger.remove()
# Set up loguru with JSONL serialization in file `crawler.log`
logger.add('crawler.log', format=formatter, serialize=True, level='INFO')
# Set up loguru logger for console
logger.add(sys.stderr, format=formatter, colorize=True, level='INFO')
# Configure standard logging to use our interceptor
logging.basicConfig(handlers=[InterceptHandler()], level=logging.INFO, force=True)
async def main() -> None:
# Initialize crawler with disabled table logs
crawler = HttpCrawler(
configure_logging=False, # Disable default logging configuration
statistics_log_format='inline', # Set inline formatting for statistics logs
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Run the crawler
await crawler.run(['https://www.crawlee.dev/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,25 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Enqueue all links found on the page.
await context.enqueue_links()
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,25 @@
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Enqueue all links found on the page.
await context.enqueue_links()
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,25 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Run the crawler with the initial list of requests.
await crawler.run(
[
'https://crawlee.dev',
'https://apify.com',
'https://example.com',
]
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,25 @@
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
crawler = PlaywrightCrawler()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Run the crawler with the initial list of requests.
await crawler.run(
[
'https://crawlee.dev',
'https://apify.com',
'https://example.com',
]
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,29 @@
import asyncio
from crawlee import Glob
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Enqueue all the documentation links found on the page, except for the examples.
await context.enqueue_links(
include=[Glob('https://crawlee.dev/docs/**')],
exclude=[Glob('https://crawlee.dev/docs/examples')],
)
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,29 @@
import asyncio
from crawlee import Glob
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Enqueue all the documentation links found on the page, except for the examples.
await context.enqueue_links(
include=[Glob('https://crawlee.dev/docs/**')],
exclude=[Glob('https://crawlee.dev/docs/examples')],
)
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,26 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Enqueue all links found on the page. Any URLs found will be matched by
# this strategy, even if they go off the site you are currently crawling.
await context.enqueue_links(strategy='all')
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,26 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Setting the strategy to same domain will enqueue all links found that
# are on the same hostname as request.loaded_url or request.url.
await context.enqueue_links(strategy='same-domain')
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,26 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Setting the strategy to same hostname will enqueue all links found that are on
# the same hostname (including subdomains) as request.loaded_url or request.url.
await context.enqueue_links(strategy='same-hostname')
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,26 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Setting the strategy to same origin will enqueue all links found that are on
# the same origin as request.loaded_url or request.url.
await context.enqueue_links(strategy='same-origin')
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,39 @@
import asyncio
import csv
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': context.soup.title.string if context.soup.title else None,
}
# Enqueue all links found on the page.
await context.enqueue_links()
# Push the extracted data to the default dataset.
await context.push_data(data)
# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev'])
# Export the entire dataset to a CSV file.
# Use semicolon as delimiter and always quote strings.
await crawler.export_data(path='results.csv', delimiter=';', quoting=csv.QUOTE_ALL)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,38 @@
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': context.soup.title.string if context.soup.title else None,
}
# Enqueue all links found on the page.
await context.enqueue_links()
# Push the extracted data to the default dataset.
await context.push_data(data)
# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev'])
# Export the entire dataset to a JSON file.
# Set ensure_ascii=False to allow Unicode characters in the output.
await crawler.export_data(path='results.json', ensure_ascii=False)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,36 @@
import asyncio
from crawlee import Glob
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract all the documentation links found on the page, except for the examples.
extracted_links = await context.extract_links(
include=[Glob('https://crawlee.dev/docs/**')],
exclude=[Glob('https://crawlee.dev/docs/examples')],
)
# Some very custom filtering which can't be achieved by `extract_links` arguments.
max_link_length = 30
filtered_links = [
link for link in extracted_links if len(link.url) < max_link_length
]
# Add filtered links to the request queue.
await context.add_requests(filtered_links)
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,36 @@
import asyncio
from crawlee import Glob
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract all the documentation links found on the page, except for the examples.
extracted_links = await context.extract_links(
include=[Glob('https://crawlee.dev/docs/**')],
exclude=[Glob('https://crawlee.dev/docs/examples')],
)
# Some very custom filtering which can't be achieved by `extract_links` arguments.
max_link_length = 30
filtered_links = [
link for link in extracted_links if len(link.url) < max_link_length
]
# Add filtered links to the request queue.
await context.add_requests(filtered_links)
# Run the crawler with the initial list of requests.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,41 @@
import asyncio
from urllib.parse import urlencode
from crawlee import Request
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
async def main() -> None:
crawler = HttpCrawler()
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
response = (await context.http_response.read()).decode('utf-8')
context.log.info(f'Response: {response}') # To see the response in the logs.
# Prepare a POST request to the form endpoint.
request = Request.from_url(
url='https://httpbin.org/post',
method='POST',
headers={'content-type': 'application/x-www-form-urlencoded'},
payload=urlencode(
{
'custname': 'John Doe',
'custtel': '1234567890',
'custemail': 'johndoe@example.com',
'size': 'large',
'topping': ['bacon', 'cheese', 'mushroom'],
'delivery': '13:00',
'comments': 'Please ring the doorbell upon arrival.',
}
).encode(),
)
# Run the crawler with the initial list of requests.
await crawler.run([request])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,28 @@
import asyncio
from urllib.parse import urlencode
from crawlee import Request
async def main() -> None:
# Prepare a POST request to the form endpoint.
request = Request.from_url(
url='https://httpbin.org/post',
method='POST',
headers={'content-type': 'application/x-www-form-urlencoded'},
payload=urlencode(
{
'custname': 'John Doe',
'custtel': '1234567890',
'custemail': 'johndoe@example.com',
'size': 'large',
'topping': ['bacon', 'cheese', 'mushroom'],
'delivery': '13:00',
'comments': 'Please ring the doorbell upon arrival.',
}
).encode(),
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,47 @@
import asyncio
from crawlee.crawlers import BasicCrawlingContext, ParselCrawler, ParselCrawlingContext
# Regex for identifying email addresses on a webpage.
EMAIL_REGEX = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
async def main() -> None:
crawler = ParselCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: ParselCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page.
data = {
'url': context.request.url,
'title': context.selector.xpath('//title/text()').get(),
'email_address_list': context.selector.re(EMAIL_REGEX),
}
# Push the extracted data to the default dataset.
await context.push_data(data)
# Enqueue all links found on the page.
await context.enqueue_links()
# Register pre navigation hook which will be called before each request.
# This hook is optional and does not need to be defined at all.
@crawler.pre_navigation_hook
async def some_hook(context: BasicCrawlingContext) -> None:
pass
# Run the crawler with the initial list of URLs.
await crawler.run(['https://github.com'])
# Export the entire dataset to a JSON file.
await crawler.export_data(path='results.json')
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,31 @@
import asyncio
from random import choice
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.statistics import Statistics
async def main() -> None:
crawler = ParselCrawler(
statistics=Statistics.with_default_state(save_error_snapshots=True)
)
@crawler.router.default_handler
async def request_handler(context: ParselCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Simulate various errors to demonstrate `ErrorSnapshotter`
# saving only the first occurrence of unique error.
await context.enqueue_links()
random_number = choice(range(10))
if random_number == 1:
raise KeyError('Some KeyError')
if random_number == 2:
raise ValueError('Some ValueError')
if random_number == 3:
raise RuntimeError('Some RuntimeError')
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,36 @@
import asyncio
from crawlee.crawlers import (
PlaywrightCrawler,
PlaywrightCrawlingContext,
PlaywrightPreNavCrawlingContext,
)
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
await context.enqueue_links()
# Define the hook, which will be called before every request.
@crawler.pre_navigation_hook
async def navigation_hook(context: PlaywrightPreNavCrawlingContext) -> None:
context.log.info(f'Navigating to {context.request.url} ...')
# Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
await context.block_requests(extra_url_patterns=['adsbygoogle.js'])
# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,67 @@
import asyncio
from crawlee.crawlers import (
PlaywrightCrawler,
PlaywrightCrawlingContext,
PlaywrightPreNavCrawlingContext,
)
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
# Headless mode, set to False to see the browser in action.
headless=False,
# Browser types supported by Playwright.
browser_type='chromium',
)
# Define the default request handler, which will be called for every request.
# The handler receives a context parameter, providing various properties and
# helper methods. Here are a few key ones we use for demonstration:
# - request: an instance of the Request class containing details such as the URL
# being crawled and the HTTP method used.
# - page: Playwright's Page object, which allows interaction with the web page
# (see https://playwright.dev/python/docs/api/class-page for more details).
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract data from the page using Playwright's API.
posts = await context.page.query_selector_all('.athing')
data = []
for post in posts:
# Get the HTML elements for the title and rank within each post.
title_element = await post.query_selector('.title a')
rank_element = await post.query_selector('.rank')
# Extract the data we want from the elements.
title = await title_element.inner_text() if title_element else None
rank = await rank_element.inner_text() if rank_element else None
href = await title_element.get_attribute('href') if title_element else None
data.append({'title': title, 'rank': rank, 'href': href})
# Push the extracted data to the default dataset. In local configuration,
# the data will be stored as JSON files in ./storage/datasets/default.
await context.push_data(data)
# Find a link to the next page and enqueue it if it exists.
await context.enqueue_links(selector='.morelink')
# Define a hook that will be called each time before navigating to a new URL.
# The hook receives a context parameter, providing access to the request and
# browser page among other things. In this example, we log the URL being
# navigated to.
@crawler.pre_navigation_hook
async def log_navigation_url(context: PlaywrightPreNavCrawlingContext) -> None:
context.log.info(f'Navigating to {context.request.url} ...')
# Run the crawler with the initial list of URLs.
await crawler.run(['https://news.ycombinator.com/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,69 @@
import asyncio
# Camoufox is external package and needs to be installed. It is not included in crawlee.
from camoufox import AsyncNewBrowser
from typing_extensions import override
from crawlee.browsers import (
BrowserPool,
PlaywrightBrowserController,
PlaywrightBrowserPlugin,
)
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
class CamoufoxPlugin(PlaywrightBrowserPlugin):
"""Example browser plugin that uses Camoufox browser,
but otherwise keeps the functionality of PlaywrightBrowserPlugin.
"""
@override
async def new_browser(self) -> PlaywrightBrowserController:
if not self._playwright:
raise RuntimeError('Playwright browser plugin is not initialized.')
return PlaywrightBrowserController(
browser=await AsyncNewBrowser(
self._playwright, **self._browser_launch_options
),
# Increase, if camoufox can handle it in your use case.
max_open_pages_per_browser=1,
# This turns off the crawlee header_generation. Camoufox has its own.
header_generator=None,
)
async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
# Custom browser pool. Gives users full control over browsers used by the crawler.
browser_pool=BrowserPool(plugins=[CamoufoxPlugin()]),
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Extract some data from the page using Playwright's API.
posts = await context.page.query_selector_all('.athing')
for post in posts:
# Get the HTML elements for the title and rank within each post.
title_element = await post.query_selector('.title a')
# Extract the data we want from the elements.
title = await title_element.inner_text() if title_element else None
# Push the extracted data to the default dataset.
await context.push_data({'title': title})
# Find a link to the next page and enqueue it if it exists.
await context.enqueue_links(selector='.morelink')
# Run the crawler with the initial list of URLs.
await crawler.run(['https://news.ycombinator.com/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,31 @@
import asyncio
from random import choice
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.statistics import Statistics
async def main() -> None:
crawler = PlaywrightCrawler(
statistics=Statistics.with_default_state(save_error_snapshots=True)
)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Simulate various errors to demonstrate `ErrorSnapshotter`
# saving only the first occurrence of unique error.
await context.enqueue_links()
random_number = choice(range(10))
if random_number == 1:
raise KeyError('Some KeyError')
if random_number == 2:
raise ValueError('Some ValueError')
if random_number == 3:
raise RuntimeError('Some RuntimeError')
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,44 @@
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.fingerprint_suite import (
DefaultFingerprintGenerator,
HeaderGeneratorOptions,
ScreenOptions,
)
async def main() -> None:
# Use default fingerprint generator with desired fingerprint options.
# Generator will generate real looking browser fingerprint based on the options.
# Unspecified fingerprint options will be automatically selected by the generator.
fingerprint_generator = DefaultFingerprintGenerator(
header_options=HeaderGeneratorOptions(browsers=['chrome']),
screen_options=ScreenOptions(min_width=400),
)
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
# Headless mode, set to False to see the browser in action.
headless=False,
# Browser types supported by Playwright.
browser_type='chromium',
# Fingerprint generator to be used. By default no fingerprint generation is done.
fingerprint_generator=fingerprint_generator,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Find a link to the next page and enqueue it if it exists.
await context.enqueue_links(selector='.morelink')
# Run the crawler with the initial list of URLs.
await crawler.run(['https://news.ycombinator.com/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,36 @@
import asyncio
from crawlee import SkippedReason
from crawlee.crawlers import (
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)
async def main() -> None:
# Initialize the crawler with robots.txt compliance enabled
crawler = BeautifulSoupCrawler(respect_robots_txt_file=True)
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# highlight-start
# This handler is called when a request is skipped
@crawler.on_skipped_request
async def skipped_request_handler(url: str, reason: SkippedReason) -> None:
# Check if the request was skipped due to robots.txt rules
if reason == 'robots_txt':
crawler.log.info(f'Skipped {url} due to robots.txt rules.')
# highlight-end
# Start the crawler with the specified URLs
# The login URL will be skipped and handled by the skipped_request_handler
await crawler.run(
['https://news.ycombinator.com/', 'https://news.ycombinator.com/login']
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,27 @@
import asyncio
from crawlee.crawlers import (
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)
async def main() -> None:
# Initialize the crawler with robots.txt compliance enabled
crawler = BeautifulSoupCrawler(respect_robots_txt_file=True)
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# Start the crawler with the specified URLs
# The crawler will check the robots.txt file before making requests
# In this example, 'https://news.ycombinator.com/login' will be skipped
# because it's disallowed in the site's robots.txt file
await crawler.run(
['https://news.ycombinator.com/', 'https://news.ycombinator.com/login']
)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,40 @@
import asyncio
from crawlee import ConcurrencySettings, service_locator
from crawlee.crawlers import (
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)
# Disable clearing the `RequestQueue`, `KeyValueStore` and `Dataset` on each run.
# This makes the scraper continue from where it left off in the previous run.
# The recommended way to achieve this behavior is setting the environment variable
# `CRAWLEE_PURGE_ON_START=0`
configuration = service_locator.get_configuration()
configuration.purge_on_start = False
async def main() -> None:
crawler = BeautifulSoupCrawler(
# Let's slow down the crawler for a demonstration
concurrency_settings=ConcurrencySettings(max_tasks_per_minute=20)
)
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
# List of links for crawl
requests = [
'https://crawlee.dev',
'https://crawlee.dev/python/docs',
'https://crawlee.dev/python/docs/examples',
'https://crawlee.dev/python/docs/guides',
'https://crawlee.dev/python/docs/quick-start',
]
await crawler.run(requests)
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,94 @@
import asyncio
from crawlee import ConcurrencySettings
from crawlee.crawlers import (
ParselCrawler,
ParselCrawlingContext,
PlaywrightCrawler,
PlaywrightCrawlingContext,
)
from crawlee.sessions import SessionPool
from crawlee.storages import RequestQueue
async def main() -> None:
# Open request queues for both crawlers with different aliases
playwright_rq = await RequestQueue.open(alias='playwright-requests')
parsel_rq = await RequestQueue.open(alias='parsel-requests')
# Use a shared session pool between both crawlers
async with SessionPool() as session_pool:
playwright_crawler = PlaywrightCrawler(
# Set the request queue for Playwright crawler
request_manager=playwright_rq,
session_pool=session_pool,
# Configure concurrency settings for Playwright crawler
concurrency_settings=ConcurrencySettings(
max_concurrency=5, desired_concurrency=5
),
# Set `keep_alive`` so that the crawler does not stop working when there are
# no requests in the queue.
keep_alive=True,
)
parsel_crawler = ParselCrawler(
# Set the request queue for Parsel crawler
request_manager=parsel_rq,
session_pool=session_pool,
# Configure concurrency settings for Parsel crawler
concurrency_settings=ConcurrencySettings(
max_concurrency=10, desired_concurrency=10
),
# Set maximum requests per crawl for Parsel crawler
max_requests_per_crawl=50,
)
@playwright_crawler.router.default_handler
async def handle_playwright(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Playwright Processing {context.request.url}...')
title = await context.page.title()
# Push the extracted data to the dataset for Playwright crawler
await context.push_data(
{'title': title, 'url': context.request.url, 'source': 'playwright'},
dataset_name='playwright-data',
)
@parsel_crawler.router.default_handler
async def handle_parsel(context: ParselCrawlingContext) -> None:
context.log.info(f'Parsel Processing {context.request.url}...')
title = context.parsed_content.css('title::text').get()
# Push the extracted data to the dataset for Parsel crawler
await context.push_data(
{'title': title, 'url': context.request.url, 'source': 'parsel'},
dataset_name='parsel-data',
)
# Enqueue links to the Playwright request queue for blog pages
await context.enqueue_links(
selector='a[href*="/blog/"]', rq_alias='playwright-requests'
)
# Enqueue other links to the Parsel request queue
await context.enqueue_links(selector='a:not([href*="/blog/"])')
# Start the Playwright crawler in the background
background_crawler_task = asyncio.create_task(playwright_crawler.run([]))
# Run the Parsel crawler with the initial URL and wait for it to finish
await parsel_crawler.run(['https://crawlee.dev/blog'])
# Wait for the Playwright crawler to finish processing all requests
while not await playwright_rq.is_empty():
playwright_crawler.log.info('Waiting for Playwright crawler to finish...')
await asyncio.sleep(5)
# Stop the Playwright crawler after all requests are processed
playwright_crawler.stop()
# Wait for the background Playwright crawler task to complete
await background_crawler_task
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,54 @@
import asyncio
import shutil
from pathlib import Path
from tempfile import TemporaryDirectory
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
# Profile name to use (usually 'Default' for single profile setups)
PROFILE_NAME = 'Default'
# Paths to Chrome profiles in your system (example for Windows)
# Use `chrome://version/` to find your profile path
PROFILE_PATH = Path(Path.home(), 'AppData', 'Local', 'Google', 'Chrome', 'User Data')
async def main() -> None:
# Create a temporary folder to copy the profile to
with TemporaryDirectory(prefix='crawlee-') as tmpdirname:
tmp_profile_dir = Path(tmpdirname)
# Copy the profile to a temporary folder
shutil.copytree(
PROFILE_PATH / PROFILE_NAME,
tmp_profile_dir / PROFILE_NAME,
dirs_exist_ok=True,
)
crawler = PlaywrightCrawler(
headless=False,
# Use the installed Chrome browser
browser_type='chrome',
# Disable fingerprints to preserve profile identity
fingerprint_generator=None,
# Set user data directory to temp folder
user_data_dir=tmp_profile_dir,
browser_launch_options={
# Slow down actions to mimic human behavior
'slow_mo': 200,
'args': [
# Use the specified profile
f'--profile-directory={PROFILE_NAME}',
],
},
)
@crawler.router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Visiting {context.request.url}')
await crawler.run(['https://crawlee.dev/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,42 @@
import asyncio
from pathlib import Path
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
# Replace this with your actual Firefox profile name
# Find it at about:profiles in Firefox
PROFILE_NAME = 'your-profile-name-here'
# Paths to Firefox profiles in your system (example for Windows)
# Use `about:profiles` to find your profile path
PROFILE_PATH = Path(
Path.home(), 'AppData', 'Roaming', 'Mozilla', 'Firefox', 'Profiles', PROFILE_NAME
)
async def main() -> None:
crawler = PlaywrightCrawler(
# Use Firefox browser type
browser_type='firefox',
# Disable fingerprints to use the profile as is
fingerprint_generator=None,
headless=False,
# Path to your Firefox profile
user_data_dir=PROFILE_PATH,
browser_launch_options={
'args': [
# Required to avoid version conflicts
'--allow-downgrade'
]
},
)
@crawler.router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'Visiting {context.request.url}')
await crawler.run(['https://crawlee.dev/'])
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,101 @@
import asyncio
from collections.abc import Callable
from yarl import URL
from crawlee import RequestOptions, RequestTransformAction
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import ImpitHttpClient
from crawlee.request_loaders import SitemapRequestLoader
# Create a transform_request_function that maps request options based on the host in
# the URL
def create_transform_request(
data_mapper: dict[str, dict],
) -> Callable[[RequestOptions], RequestOptions | RequestTransformAction]:
def transform_request(
request_options: RequestOptions,
) -> RequestOptions | RequestTransformAction:
# According to the Sitemap protocol, all URLs in a Sitemap must be from a single
# host.
request_host = URL(request_options['url']).host
if request_host and (mapping_data := data_mapper.get(request_host)):
# Set properties from the mapping data
if 'label' in mapping_data:
request_options['label'] = mapping_data['label']
if 'user_data' in mapping_data:
request_options['user_data'] = mapping_data['user_data']
return request_options
return 'unchanged'
return transform_request
async def main() -> None:
# Prepare data mapping for hosts
apify_host = URL('https://apify.com/sitemap.xml').host
crawlee_host = URL('https://crawlee.dev/sitemap.xml').host
if not apify_host or not crawlee_host:
raise ValueError('Unable to extract host from URLs')
data_map = {
apify_host: {
'label': 'apify',
'user_data': {'source': 'apify'},
},
crawlee_host: {
'label': 'crawlee',
'user_data': {'source': 'crawlee'},
},
}
# Initialize the SitemapRequestLoader with the transform function
async with SitemapRequestLoader(
# Set the sitemap URLs and the HTTP client
sitemap_urls=['https://crawlee.dev/sitemap.xml', 'https://apify.com/sitemap.xml'],
http_client=ImpitHttpClient(),
transform_request_function=create_transform_request(data_map),
) as sitemap_loader:
# Convert the sitemap loader to a request manager
request_manager = await sitemap_loader.to_tandem()
# Create and configure the crawler
crawler = BeautifulSoupCrawler(
request_manager=request_manager,
max_requests_per_crawl=10,
)
# Create default handler for requests without a specific label
@crawler.router.default_handler
async def handler(context: BeautifulSoupCrawlingContext) -> None:
source = context.request.user_data.get('source', 'unknown')
context.log.info(
f'Processing request: {context.request.url} from source: {source}'
)
# Create handler for requests labeled 'apify'
@crawler.router.handler('apify')
async def apify_handler(context: BeautifulSoupCrawlingContext) -> None:
source = context.request.user_data.get('source', 'unknown')
context.log.info(
f'Apify handler processing: {context.request.url} from source: {source}'
)
# Create handler for requests labeled 'crawlee'
@crawler.router.handler('crawlee')
async def crawlee_handler(context: BeautifulSoupCrawlingContext) -> None:
source = context.request.user_data.get('source', 'unknown')
context.log.info(
f'Crawlee handler processing: {context.request.url} from source: {source}'
)
await crawler.run()
if __name__ == '__main__':
asyncio.run(main())
@@ -0,0 +1,33 @@
---
id: crawl-all-links-on-website
title: Crawl all links on website
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_bs.py';
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_pw.py';
This example uses the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> helper to add new links to the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> as the crawler navigates from page to page. By automatically discovering and enqueuing all links on a given page, the crawler can systematically scrape an entire website. This approach is ideal for web scraping tasks where you need to collect data from multiple interconnected pages.
:::tip
If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the `strategy` option, which is an instance of the `EnqueueStrategy` type alias. You can find more info about this option in the [Crawl website with relative links](./crawl-website-with-relative-links) example.
:::
<Tabs groupId="main">
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
+27
View File
@@ -0,0 +1,27 @@
---
id: crawl-multiple-urls
title: Crawl multiple URLs
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_multiple_urls_bs.py';
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_multiple_urls_pw.py';
This example demonstrates how to crawl a specified list of URLs using different crawlers. You'll learn how to set up the crawler, define a request handler, and run the crawler with multiple URLs. This setup is useful for scraping data from multiple pages or websites concurrently.
<Tabs groupId="main">
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
@@ -0,0 +1,47 @@
---
id: crawl-specific-links-on-website
title: Crawl specific links on website
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_specific_links_on_website_bs.py';
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_specific_links_on_website_pw.py';
import BeautifulSoupExampleExtractAndAdd from '!!raw-loader!roa-loader!./code_examples/extract_and_add_specific_links_on_website_bs.py';
import PlaywrightExampleExtractAndAdd from '!!raw-loader!roa-loader!./code_examples/extract_and_add_specific_links_on_website_pw.py';
This example demonstrates how to crawl a website while targeting specific patterns of links. By utilizing the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> helper, you can pass `include` or `exclude` parameters to improve your crawling strategy. This approach ensures that only the links matching the specified patterns are added to the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>. Both `include` and `exclude` support lists of globs or regular expressions. This functionality is great for focusing on relevant sections of a website and avoiding scraping unnecessary or irrelevant content.
<Tabs groupId="first-example">
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
## Even more control over the enqueued links
<ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> is a convenience helper and internally it calls <ApiLink to="class/ExtractLinksFunction">`extract_links`</ApiLink> to find the links and <ApiLink to="class/AddRequestsFunction">`add_requests`</ApiLink> to add them to the queue. If you need some additional custom filtering of the extracted links before enqueuing them, then consider using <ApiLink to="class/ExtractLinksFunction">`extract_links`</ApiLink> and <ApiLink to="class/AddRequestsFunction">`add_requests`</ApiLink> instead of the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink>
<Tabs groupId="second-example">
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExampleExtractAndAdd}
</RunnableCodeBlock>
</TabItem>
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightExampleExtractAndAdd}
</RunnableCodeBlock>
</TabItem>
</Tabs>
@@ -0,0 +1,52 @@
---
id: crawl-website-with-relative-links
title: Crawl website with relative links
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import AllLinksExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_all_links.py';
import SameDomainExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_domain.py';
import SameHostnameExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_hostname.py';
import SameOriginExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_origin.py';
When crawling a website, you may encounter various types of links that you wish to include in your crawl. To facilitate this, we provide the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> method on the crawler context, which will automatically find and add these links to the crawler's <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>. This method simplifies the process of handling different types of links, including relative links, by automatically resolving them based on the page's context.
:::note
For these examples, we are using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. However, the same method is available for other crawlers as well. You can use it in exactly the same way.
:::
`EnqueueStrategy` type alias provides four distinct strategies for crawling relative links:
- `all` - Enqueues all links found, regardless of the domain they point to. This strategy is useful when you want to follow every link, including those that navigate to external websites.
- `same-domain` - Enqueues all links found that share the same domain name, including any possible subdomains. This strategy ensures that all links within the same top-level and base domain are included.
- `same-hostname` - Enqueues all links found for the exact same hostname. This is the **default** strategy, and it restricts the crawl to links that have the same hostname as the current page, excluding subdomains.
- `same-origin` - Enqueues all links found that share the same origin. The same origin refers to URLs that share the same protocol, domain, and port, ensuring a strict scope for the crawl.
<Tabs groupId="main">
<TabItem value="all_links" label="All links">
<RunnableCodeBlock className="language-python" language="python">
{AllLinksExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="same-domain" label="Same domain">
<RunnableCodeBlock className="language-python" language="python">
{SameDomainExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="same-hostname" label="Same hostname">
<RunnableCodeBlock className="language-python" language="python">
{SameHostnameExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="same-origin" label="Same origin">
<RunnableCodeBlock className="language-python" language="python">
{SameOriginExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
+15
View File
@@ -0,0 +1,15 @@
---
id: crawler-keep-alive
title: Keep a Crawler alive waiting for more requests
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_keep_alive.py';
This example demonstrates how to keep crawler alive even when there are no requests at the moment by using `keep_alive=True` argument of <ApiLink to="class/BasicCrawler#__init__">`BasicCrawler.__init__`</ApiLink>. This is available to all crawlers that inherit from <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> and in the example below it is shown on <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. To stop the crawler that was started with `keep_alive=True` you can call `crawler.stop()`.
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
+15
View File
@@ -0,0 +1,15 @@
---
id: crawler-stop
title: Stopping a Crawler with stop method
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_stop.py';
This example demonstrates how to use `stop` method of <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> to stop crawler once the crawler finds what it is looking for. This method is available to all crawlers that inherit from <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> and in the example below it is shown on <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. Simply call `crawler.stop()` to stop the crawler. It will not continue to crawl through new requests. Requests that are already being concurrently processed are going to get finished. It is possible to call `stop` method with optional argument `reason` that is a string that will be used in logs and it can improve logs readability especially if you have multiple different conditions for triggering `stop`.
<RunnableCodeBlock className="language-python" language="python">
{BeautifulSoupExample}
</RunnableCodeBlock>
@@ -0,0 +1,33 @@
---
id: export-entire-dataset-to-file
title: Export entire dataset to file
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import JsonExample from '!!raw-loader!roa-loader!./code_examples/export_entire_dataset_to_file_json.py';
import CsvExample from '!!raw-loader!roa-loader!./code_examples/export_entire_dataset_to_file_csv.py';
This example demonstrates how to use the <ApiLink to="class/BasicCrawler#export_data">`BasicCrawler.export_data`</ApiLink> method of the crawler to export the entire default dataset to a single file. This method supports exporting data in either CSV or JSON format and also accepts additional keyword arguments so you can fine-tune the underlying `json.dump` or `csv.writer` behavior.
:::note
For these examples, we are using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. However, the same method is available for other crawlers as well. You can use it in exactly the same way.
:::
<Tabs groupId="main">
<TabItem value="json" label="JSON">
<RunnableCodeBlock className="language-python" language="python">
{JsonExample}
</RunnableCodeBlock>
</TabItem>
<TabItem value="csv" label="CSV">
<RunnableCodeBlock className="language-python" language="python">
{CsvExample}
</RunnableCodeBlock>
</TabItem>
</Tabs>
+113
View File
@@ -0,0 +1,113 @@
---
id: fill-and-submit-web-form
title: Fill and submit web form
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import RequestExample from '!!raw-loader!roa-loader!./code_examples/fill_and_submit_web_form_request.py';
import CrawlerExample from '!!raw-loader!roa-loader!./code_examples/fill_and_submit_web_form_crawler.py';
This example demonstrates how to fill and submit a web form using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink> crawler. The same approach applies to any crawler that inherits from it, such as the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> or <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>.
We are going to use the [httpbin.org](https://httpbin.org) website to demonstrate how it works.
## Investigate the form fields
First, we need to examine the form fields and the form's action URL. You can do this by opening the [httpbin.org/forms/post](https://httpbin.org/forms/post) page in a browser and inspecting the form fields.
In Chrome, right-click on the page and select "Inspect" or press `Ctrl+Shift+I`.
Use the element selector (`Ctrl+Shift+C`) to click on the form element you want to inspect.
![HTML input element name](/img/fill-and-submit-web-form/00.jpg 'HTML input element name.')
Identify the field names. For example, the customer name field is `custname`, the email field is `custemail`, and the phone field is `custtel`.
Now navigate to the "Network" tab in developer tools and submit the form by clicking the "Submit order" button.
![Submitting the form](/img/fill-and-submit-web-form/01.jpg 'Submitting the form.')
Find the form submission request and examine its details. The "Headers" tab will show the submission URL, in this case, it is `https://httpbin.org/post`.
![Network request investigation](/img/fill-and-submit-web-form/02.jpg 'Network request investigation.')
The "Payload" tab will display the form fields and their submitted values. This method could be an alternative to inspecting the HTML source code directly.
![Network payload investigation](/img/fill-and-submit-web-form/03.jpg 'Network payload investigation.')
## Preparing a POST request
Now, let's create a POST request with the form fields and their values using the <ApiLink to="class/Request">`Request`</ApiLink> class, specifically its <ApiLink to="class/Request#from_url">`Request.from_url`</ApiLink> constructor:
<RunnableCodeBlock className="language-python" language="python">
{RequestExample}
</RunnableCodeBlock>
Alternatively, you can send form data as URL parameters using the `url` argument. It depends on the form and how it is implemented. However, sending the data as a POST request body using the `payload` is generally a better approach.
## Implementing the crawler
Finally, let's implement the crawler and run it with the prepared request. Although we are using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>, the process is the same for any crawler that inherits from it.
<RunnableCodeBlock className="language-python" language="python">
{CrawlerExample}
</RunnableCodeBlock>
## Running the crawler
Finally, run your crawler. Your logs should show something like this:
```plaintext
...
[crawlee.http_crawler._http_crawler] INFO Processing https://httpbin.org/post ...
[crawlee.http_crawler._http_crawler] INFO Response: {
"args": {},
"data": "",
"files": {},
"form": {
"comments": "Please ring the doorbell upon arrival.",
"custemail": "johndoe@example.com",
"custname": "John Doe",
"custtel": "1234567890",
"delivery": "13:00",
"size": "large",
"topping": [
"bacon",
"cheese",
"mushroom"
]
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Content-Length": "190",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "python-httpx/0.27.0",
"X-Amzn-Trace-Id": "Root=1-66c849d6-1ae432fb7b4156e6149ff37f"
},
"json": null,
"origin": "78.80.81.196",
"url": "https://httpbin.org/post"
}
[crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
[crawlee.http_crawler._http_crawler] INFO Final request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished │ 1 │
│ requests_failed │ 0 │
│ retry_histogram │ [1] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ 0.678442 │
│ requests_finished_per_minute │ 85 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 0.678442 │
│ requests_total │ 1 │
│ crawler_runtime │ 0.707666 │
└───────────────────────────────┴──────────┘
```
This log output confirms that the crawler successfully submitted the form and processed the response. Congratulations! You have successfully filled and submitted a web form using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>.
+57
View File
@@ -0,0 +1,57 @@
---
id: configure-json-logging
title: Сonfigure JSON logging
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import JsonLoggingExample from '!!raw-loader!roa-loader!./code_examples/configure_json_logging.py';
This example demonstrates how to configure JSON line (JSONL) logging with Crawlee. By using the `use_table_logs=False` parameter, you can disable table-formatted statistics logs, which makes it easier to parse logs with external tools or to serialize them as JSON.
The example shows how to integrate with the popular [`loguru`](https://github.com/delgan/loguru) library to capture Crawlee logs and format them as JSONL (one JSON object per line). This approach works well when you need to collect logs for analysis, monitoring, or when integrating with logging platforms like ELK Stack, Grafana Loki, or similar systems.
<RunnableCodeBlock className="language-python" language="python">
{JsonLoggingExample}
</RunnableCodeBlock>
Here's an example of what a crawler statistics log entry in JSONL format.
```json
{
"text": "[HttpCrawler] | INFO | - Final request statistics: {'requests_finished': 1, 'requests_failed': 0, 'retry_histogram': [1], 'request_avg_failed_duration': None, 'request_avg_finished_duration': 3.57098, 'requests_finished_per_minute': 17, 'requests_failed_per_minute': 0, 'request_total_duration': 3.57098, 'requests_total': 1, 'crawler_runtime': 3.59165}\n",
"record": {
"elapsed": { "repr": "0:00:05.604568", "seconds": 5.604568 },
"exception": null,
"extra": {
"requests_finished": 1,
"requests_failed": 0,
"retry_histogram": [1],
"request_avg_failed_duration": null,
"request_avg_finished_duration": 3.57098,
"requests_finished_per_minute": 17,
"requests_failed_per_minute": 0,
"request_total_duration": 3.57098,
"requests_total": 1,
"crawler_runtime": 3.59165
},
"file": {
"name": "_basic_crawler.py",
"path": "/crawlers/_basic/_basic_crawler.py"
},
"function": "run",
"level": { "icon": "ℹ️", "name": "INFO", "no": 20 },
"line": 583,
"message": "Final request statistics:",
"module": "_basic_crawler",
"name": "HttpCrawler",
"process": { "id": 198383, "name": "MainProcess" },
"thread": { "id": 135312814966592, "name": "MainThread" },
"time": {
"repr": "2025-03-17 17:14:45.339150+00:00",
"timestamp": 1742231685.33915
}
}
}
```
+15
View File
@@ -0,0 +1,15 @@
---
id: parsel-crawler
title: Parsel crawler
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ParselCrawlerExample from '!!raw-loader!roa-loader!./code_examples/parsel_crawler.py';
This example shows how to use <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> to crawl a website or a list of URLs. Each URL is loaded using a plain HTTP request and the response is parsed using [Parsel](https://pypi.org/project/parsel/) library which supports CSS and XPath selectors for HTML responses and JMESPath for JSON responses. We can extract data from all kinds of complex HTML structures using XPath. In this example, we will use Parsel to crawl github.com and extract page title, URL and emails found in the webpage. The default handler will scrape data from the current webpage and enqueue all the links found in the webpage for continuous scraping. It also shows how you can add optional pre-navigation hook to the crawler. Pre-navigation hooks are user defined functions that execute before sending the request.
<RunnableCodeBlock className="language-python" language="python">
{ParselCrawlerExample}
</RunnableCodeBlock>
+19
View File
@@ -0,0 +1,19 @@
---
id: playwright-crawler
title: Playwright crawler
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler.py';
This example demonstrates how to use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to recursively scrape the Hacker news website using headless Chromium and Playwright.
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> manages the browser and page instances, simplifying the process of interacting with web pages. In the request handler, Playwright's API is used to extract data from each post on the page. Specifically, it retrieves the title, rank, and URL of each post. Additionally, the handler enqueues links to the next pages to ensure continuous scraping. This setup is ideal for scraping dynamic web pages where JavaScript execution is required to render the content.
A **pre-navigation hook** can be used to perform actions before navigating to the URL. This hook provides further flexibility in controlling environment and preparing for navigation.
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightCrawlerExample}
</RunnableCodeBlock>
@@ -0,0 +1,20 @@
---
id: adaptive-playwright-crawler
title: Adaptive Playwright crawler
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import AdaptivePlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/adaptive_playwright_crawler.py';
This example demonstrates how to use <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink>. An <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> is a combination of <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and some implementation of HTTP-based crawler such as <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> or <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>.
It uses a more limited crawling context interface so that it is able to switch to HTTP-only crawling when it detects that it may bring a performance benefit.
A [pre-navigation hook](/python/docs/guides/adaptive-playwright-crawler#page-configuration-with-pre-navigation-hooks) can be used to perform actions before navigating to the URL. This hook provides further flexibility in controlling environment and preparing for navigation. Hooks will be executed both for the pages crawled by HTTP-bases sub crawler and playwright based sub crawler. Use `playwright_only=True` to mark hooks that should be executed only for playwright sub crawler.
For more detailed description please see [Adaptive Playwright crawler guide](/python/docs/guides/adaptive-playwright-crawler)
<RunnableCodeBlock className="language-python" language="python">
{AdaptivePlaywrightCrawlerExample}
</RunnableCodeBlock>
@@ -0,0 +1,27 @@
---
id: playwright-crawler-with-block-requests
title: Playwright crawler with block requests
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import PlaywrightBlockRequests from '!!raw-loader!roa-loader!./code_examples/playwright_block_requests.py';
This example demonstrates how to optimize your <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> performance by blocking unnecessary network requests.
The primary use case is when you need to scrape or interact with web pages without loading non-essential resources like images, styles, or analytics scripts. This can significantly reduce bandwidth usage and improve crawling speed.
The <ApiLink to="class/BlockRequestsFunction">`block_requests`</ApiLink> helper provides the most efficient way to block requests as it operates directly in the browser.
By default, <ApiLink to="class/BlockRequestsFunction">`block_requests`</ApiLink> will block all URLs including the following patterns:
```python
['.css', '.webp', '.jpg', '.jpeg', '.png', '.svg', '.gif', '.woff', '.pdf', '.zip']
```
You can also replace the default patterns list with your own by providing `url_patterns`, or extend it by passing additional patterns in `extra_url_patterns`.
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightBlockRequests}
</RunnableCodeBlock>
@@ -0,0 +1,26 @@
---
id: playwright-crawler-with-camoufox
title: Playwright crawler with Camoufox
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import PlaywrightCrawlerExampleWithCamoufox from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_camoufox.py';
This example demonstrates how to integrate Camoufox into <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> using <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink> with custom <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>.
Camoufox is a stealthy minimalistic build of Firefox. For details please visit its homepage https://camoufox.com/ .
To be able to run this example you will need to install camoufox, as it is external tool, and it is not part of the crawlee. For installation please see https://pypi.org/project/camoufox/.
**Warning!** Camoufox is using custom build of firefox. This build can be hundreds of MB large.
You can either pre-download this file using following command `python3 -m camoufox fetch` or camoufox will download it automatically once you try to run it, and it does not find existing binary.
For more details please refer to: https://github.com/daijro/camoufox/tree/main/pythonlib#camoufox-python-interface
**Project template -** It is possible to generate project with Python code which includes Camoufox integration into crawlee through crawlee cli. Call `crawlee create` and pick `Playwright-camoufox` when asked for Crawler type.
The example code after PlayWrightCrawler instantiation is similar to example describing the use of Playwright Crawler. The main difference is that in this example Camoufox will be used as the browser through BrowserPool.
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightCrawlerExampleWithCamoufox}
</RunnableCodeBlock>
@@ -0,0 +1,17 @@
---
id: playwright-crawler-with-fingerprint-generator
title: Playwright crawler with fingerprint generator
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_fingerprint_generator.py';
This example demonstrates how to use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> together with <ApiLink to="class/FingerprintGenerator">`FingerprintGenerator`</ApiLink> that will populate several browser attributes to mimic real browser fingerprint. To read more about fingerprints please see: https://docs.apify.com/academy/anti-scraping/techniques/fingerprinting.
You can implement your own fingerprint generator or use <ApiLink to="class/BrowserforgeFingerprintGenerator">`DefaultFingerprintGenerator`</ApiLink>. To use the generator initialize it with the desired fingerprint options. The generator will try to create fingerprint based on those options. Unspecified options will be automatically selected by the generator from the set of reasonable values. If some option is important for you, do not rely on the default and explicitly define it.
<RunnableCodeBlock className="language-python" language="python">
{PlaywrightCrawlerExample}
</RunnableCodeBlock>
+32
View File
@@ -0,0 +1,32 @@
---
id: respect-robots-txt-file
title: Respect robots.txt file
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import RespectRobotsTxt from '!!raw-loader!roa-loader!./code_examples/respect_robots_txt_file.py';
import OnSkippedRequest from '!!raw-loader!roa-loader!./code_examples/respect_robots_on_skipped_request.py';
This example demonstrates how to configure your crawler to respect the rules established by websites for crawlers as described in the [robots.txt](https://www.robotstxt.org/robotstxt.html) file.
To configure `Crawlee` to follow the `robots.txt` file, set the parameter `respect_robots_txt_file=True` in <ApiLink to="class/BasicCrawlerOptions">`BasicCrawlerOptions`</ApiLink>. In this case, `Crawlee` will skip any URLs forbidden in the website's robots.txt file.
As an example, let's look at the website `https://news.ycombinator.com/` and its corresponding [robots.txt](https://news.ycombinator.com/robots.txt) file. Since the file has a rule `Disallow: /login`, the URL `https://news.ycombinator.com/login` will be automatically skipped.
The code below demonstrates this behavior using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>:
<RunnableCodeBlock className="language-python" language="python">
{RespectRobotsTxt}
</RunnableCodeBlock>
## Handle with `on_skipped_request`
If you want to process URLs skipped according to the `robots.txt` rules, for example for further analysis, you should use the `on_skipped_request` handler from <ApiLink to="class/BasicCrawler#on_skipped_request">`BasicCrawler`</ApiLink>.
Let's update the code by adding the `on_skipped_request` handler:
<RunnableCodeBlock className="language-python" language="python">
{OnSkippedRequest}
</RunnableCodeBlock>
+35
View File
@@ -0,0 +1,35 @@
---
id: resuming-paused-crawl
title: Resuming a paused crawl
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ResumeCrawl from '!!raw-loader!roa-loader!./code_examples/resuming_paused_crawl.py';
This example demonstrates how to resume crawling from its last state when running locally, if for some reason it was unexpectedly terminated.
If each run should continue crawling from the previous state, you can configure this using `purge_on_start` in <ApiLink to="class/Configuration">`Configuration`</ApiLink>.
Use the code below and perform 2 sequential runs. During the 1st run, stop the crawler by pressing `CTRL+C`, and the 2nd run will resume crawling from where it stopped.
<RunnableCodeBlock className="language-python" language="python">
{ResumeCrawl}
</RunnableCodeBlock>
Perform the 1st run, interrupting the crawler with `CTRL+C` after 2 links have been processed.
![Run with interruption](/img/resuming-paused-crawl/00.webp 'Run with interruption.')
Now resume crawling after the pause to process the remaining 3 links.
![Resuming crawling](/img/resuming-paused-crawl/01.webp 'Resuming crawling.')
Alternatively, use the environment variable `CRAWLEE_PURGE_ON_START=0` instead of using `configuration.purge_on_start = False`.
For example, when running code:
```bash
CRAWLEE_PURGE_ON_START=0 python -m best_crawler
```
+19
View File
@@ -0,0 +1,19 @@
---
id: run-parallel-crawlers
title: Run parallel crawlers
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import RunParallelCrawlersExample from '!!raw-loader!roa-loader!./code_examples/run_parallel_crawlers.py';
This example demonstrates how to run two parallel crawlers where one crawler processes links discovered by another crawler.
In some situations, you may need different approaches for scraping data from a website. For example, you might use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> for navigating JavaScript-heavy pages and a faster, more lightweight <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> for processing static pages. One way to solve this is to use <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink>, see the [Adaptive Playwright crawler example](./adaptive-playwright-crawler) to learn more.
The code below demonstrates an alternative approach using two separate crawlers. Links are passed between crawlers via <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> aliases. The `keep_alive` option allows the Playwright crawler to run in the background and wait for incoming links without stopping when its queue is empty. You can also use different storage clients for each crawler without losing the ability to pass links between queues. Learn more about available storage clients in this [guide](/python/docs/guides/storage-clients).
<RunnableCodeBlock className="language-python" language="python">
{RunParallelCrawlersExample}
</RunnableCodeBlock>
+39
View File
@@ -0,0 +1,39 @@
---
id: using_browser_profile
title: Using browser profile
---
import ApiLink from '@site/src/components/ApiLink';
import CodeBlock from '@theme/CodeBlock';
import ChromeProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_chrome.py';
import FirefoxProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_firefox.py';
This example demonstrates how to run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> using your local browser profile from [Chrome](https://www.google.com/intl/us/chrome/) or [Firefox](https://www.firefox.com/).
Using browser profiles allows you to leverage existing login sessions, saved passwords, bookmarks, and other personalized browser data during crawling. This can be particularly useful for testing scenarios or when you need to access content that requires authentication.
## Chrome browser
To run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> with your Chrome profile, you need to know the path to your profile files. You can find this information by entering `chrome://version/` as a URL in your Chrome browser. If you have multiple profiles, pay attention to the profile name - if you only have one profile, it's always `Default`.
:::warning Profile access limitation
Due to [Chrome's security policies](https://developer.chrome.com/blog/remote-debugging-port), automation cannot use your main browsing profile directly. The example copies your profile to a temporary location as a workaround.
:::
Make sure you don't have any running Chrome browser processes before running this code:
<CodeBlock className="language-python" language="python">
{ChromeProfileExample}
</CodeBlock>
## Firefox browser
To find the path to your Firefox profile, enter `about:profiles` as a URL in your Firefox browser. Unlike Chrome, you can use your standard profile path directly without copying it first.
Make sure you don't have any running Firefox browser processes before running this code:
<CodeBlock className="language-python" language="python">
{FirefoxProfileExample}
</CodeBlock>
@@ -0,0 +1,22 @@
---
id: using-sitemap-request-loader
title: Using sitemap request loader
---
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import SitemapRequestLoaderExample from '!!raw-loader!roa-loader!./code_examples/using_sitemap_request_loader.py';
This example demonstrates how to use <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> to crawl websites that provide `sitemap.xml` files following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> processes sitemaps in a streaming fashion without loading them entirely into memory, making it suitable for large sitemaps.
The example shows how to use the `transform_request_function` parameter to configure request options based on URL patterns. This allows you to modify request properties such as labels and user data based on the source URL, enabling different handling logic for different websites or sections.
The following code example implements processing of sitemaps from two different domains (Apify and Crawlee), with different labels assigned to requests based on their host. The `create_transform_request` function maps each host to the corresponding request configuration, while the crawler uses different handlers based on the assigned labels.
<RunnableCodeBlock className="language-python" language="python">
{SitemapRequestLoaderExample}
</RunnableCodeBlock>
For more information about request loaders, see the [Request loaders guide](../guides/request-loaders).