同步完整源码 - 2026-05-24
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
---
|
||||
id: add-data-to-dataset
|
||||
title: Add data to dataset
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_bs.py';
|
||||
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_pw.py';
|
||||
import DatasetExample from '!!raw-loader!roa-loader!./code_examples/add_data_to_dataset_dataset.py';
|
||||
|
||||
This example demonstrates how to store extracted data into datasets using the <ApiLink to="class/PushDataFunction#open">`context.push_data`</ApiLink> helper function. If the specified dataset does not already exist, it will be created automatically. Additionally, you can save data to custom datasets by providing `dataset_id` or `dataset_name` parameters to the <ApiLink to="class/PushDataFunction#open">`push_data`</ApiLink> function.
|
||||
|
||||
<Tabs groupId="main">
|
||||
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
Each item in the dataset will be stored in its own file within the following directory:
|
||||
|
||||
```text
|
||||
{PROJECT_FOLDER}/storage/datasets/default/
|
||||
```
|
||||
|
||||
For more control, you can also open a dataset manually using the asynchronous constructor <ApiLink to="class/Dataset#open">`Dataset.open`</ApiLink>
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{DatasetExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
id: beautifulsoup-crawler
|
||||
title: BeautifulSoup crawler
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler.py';
|
||||
|
||||
This example demonstrates how to use <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> to crawl a list of URLs, load each URL using a plain HTTP request, parse the HTML using the [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) library and extract some data from it - the page title and all `<h1>`, `<h2>` and `<h3>` tags. This setup is perfect for scraping specific elements from web pages. Thanks to the well-known BeautifulSoup, you can easily navigate the HTML structure and retrieve the data you need with minimal code. It also shows how you can add optional pre-navigation hook to the crawler. Pre-navigation hooks are user defined functions that execute before sending the request.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
id: capture-screenshots-using-playwright
|
||||
title: Capture screenshots using Playwright
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import CaptureScreenshotExample from '!!raw-loader!roa-loader!./code_examples/capture_screenshot_using_playwright.py';
|
||||
|
||||
This example demonstrates how to capture screenshots of web pages using <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and store them in the key-value store.
|
||||
|
||||
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> is configured to automate the browsing and interaction with web pages. It uses headless Chromium as the browser type to perform these tasks. Each web page specified in the initial list of URLs is visited sequentially, and a screenshot of the page is captured using Playwright's `page.screenshot()` method.
|
||||
|
||||
The captured screenshots are stored in the key-value store, which is suitable for managing and storing files in various formats. In this case, screenshots are stored as PNG images with a unique key generated from the URL of the page.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{CaptureScreenshotExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
id: capturing-page-snapshots-with-error-snapshotter
|
||||
title: Capturing page snapshots with ErrorSnapshotter
|
||||
description: How to capture page snapshots on errors.
|
||||
---
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import ParselCrawlerWithErrorSnapshotter from '!!raw-loader!roa-loader!./code_examples/parsel_crawler_with_error_snapshotter.py';
|
||||
import PlaywrightCrawlerWithErrorSnapshotter from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_error_snapshotter.py';
|
||||
|
||||
|
||||
This example demonstrates how to capture page snapshots on first occurrence of each unique error. The capturing happens automatically if you set `save_error_snapshots=True` in the crawler's <ApiLink to="class/Statistics">`Statistics`</ApiLink>. The error snapshot can contain `html` file and `jpeg` file that are created from the page where the unhandled exception was raised. Captured error snapshot files are saved to the default key-value store. Both <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and [HTTP crawlers](../guides/http-crawlers) are capable of capturing the html file, but only <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> is able to capture page screenshot as well.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="ParselCrawler" label="ParselCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{ ParselCrawlerWithErrorSnapshotter }
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{ PlaywrightCrawlerWithErrorSnapshotter }
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,66 @@
|
||||
import asyncio
|
||||
from datetime import timedelta
|
||||
|
||||
from playwright.async_api import Route
|
||||
|
||||
from crawlee.crawlers import (
|
||||
AdaptivePlaywrightCrawler,
|
||||
AdaptivePlaywrightCrawlingContext,
|
||||
AdaptivePlaywrightPreNavCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Crawler created by following factory method will use `beautifulsoup`
|
||||
# for parsing static content.
|
||||
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
|
||||
max_requests_per_crawl=10, # Limit the max requests per crawl.
|
||||
playwright_crawler_specific_kwargs={'headless': False},
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler_for_label(
|
||||
context: AdaptivePlaywrightCrawlingContext,
|
||||
) -> None:
|
||||
# Do some processing using `parsed_content`
|
||||
context.log.info(context.parsed_content.title)
|
||||
|
||||
# Locate element h2 within 5 seconds
|
||||
h2 = await context.query_selector_one('h2', timedelta(milliseconds=5000))
|
||||
# Do stuff with element found by the selector
|
||||
context.log.info(h2)
|
||||
|
||||
# Find more links and enqueue them.
|
||||
await context.enqueue_links()
|
||||
# Save some data.
|
||||
await context.push_data({'Visited url': context.request.url})
|
||||
|
||||
@crawler.pre_navigation_hook
|
||||
async def hook(context: AdaptivePlaywrightPreNavCrawlingContext) -> None:
|
||||
"""Hook executed both in static sub crawler and playwright sub crawler.
|
||||
|
||||
Trying to access `context.page` in this hook would raise `AdaptiveContextError`
|
||||
for pages crawled without playwright."""
|
||||
context.log.info(f'pre navigation hook for: {context.request.url} ...')
|
||||
|
||||
@crawler.pre_navigation_hook(playwright_only=True)
|
||||
async def hook_playwright(context: AdaptivePlaywrightPreNavCrawlingContext) -> None:
|
||||
"""Hook executed only in playwright sub crawler.
|
||||
|
||||
It is safe to access `page` object.
|
||||
"""
|
||||
|
||||
async def some_routing_function(route: Route) -> None:
|
||||
await route.continue_()
|
||||
|
||||
await context.page.route('*/**', some_routing_function)
|
||||
context.log.info(
|
||||
f'Playwright only pre navigation hook for: {context.request.url} ...'
|
||||
)
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://warehouse-theme-metal.myshopify.com/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,35 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': context.soup.title.string if context.soup.title else None,
|
||||
'html': str(context.soup)[:1000],
|
||||
}
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data(data)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(
|
||||
[
|
||||
'https://crawlee.dev',
|
||||
'https://apify.com',
|
||||
'https://example.com',
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,15 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.storages import Dataset
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Open dataset manually using asynchronous constructor open().
|
||||
dataset = await Dataset.open()
|
||||
|
||||
# Interact with dataset directly.
|
||||
await dataset.push_data({'key': 'value'})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,35 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': await context.page.title(),
|
||||
'html': str(await context.page.content())[:1000],
|
||||
}
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data(data)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(
|
||||
[
|
||||
'https://crawlee.dev',
|
||||
'https://apify.com',
|
||||
'https://example.com',
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,57 @@
|
||||
import asyncio
|
||||
from datetime import timedelta
|
||||
|
||||
from crawlee.crawlers import (
|
||||
BasicCrawlingContext,
|
||||
BeautifulSoupCrawler,
|
||||
BeautifulSoupCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Create an instance of the BeautifulSoupCrawler class, a crawler that automatically
|
||||
# loads the URLs and parses their HTML using the BeautifulSoup library.
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# On error, retry each page at most once.
|
||||
max_request_retries=1,
|
||||
# Increase the timeout for processing each page to 30 seconds.
|
||||
request_handler_timeout=timedelta(seconds=30),
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
# The handler receives a context parameter, providing various properties and
|
||||
# helper methods. Here are a few key ones we use for demonstration:
|
||||
# - request: an instance of the Request class containing details such as the URL
|
||||
# being crawled and the HTTP method used.
|
||||
# - soup: the BeautifulSoup object containing the parsed HTML of the response.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': context.soup.title.string if context.soup.title else None,
|
||||
'h1s': [h1.text for h1 in context.soup.find_all('h1')],
|
||||
'h2s': [h2.text for h2 in context.soup.find_all('h2')],
|
||||
'h3s': [h3.text for h3 in context.soup.find_all('h3')],
|
||||
}
|
||||
|
||||
# Push the extracted data to the default dataset. In local configuration,
|
||||
# the data will be stored as JSON files in ./storage/datasets/default.
|
||||
await context.push_data(data)
|
||||
|
||||
# Register pre navigation hook which will be called before each request.
|
||||
# This hook is optional and does not need to be defined at all.
|
||||
@crawler.pre_navigation_hook
|
||||
async def some_hook(context: BasicCrawlingContext) -> None:
|
||||
pass
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,56 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee._types import BasicCrawlingContext
|
||||
from crawlee.crawlers import BeautifulSoupCrawler
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Keep the crawler alive even when there are no requests to be processed now.
|
||||
keep_alive=True,
|
||||
)
|
||||
|
||||
def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None:
|
||||
"""Stop crawler once specific url is visited.
|
||||
|
||||
Example of guard condition to stop the crawler."""
|
||||
if context.request.url == 'https://crawlee.dev/docs/examples':
|
||||
crawler.stop(
|
||||
'Stop crawler that was in keep_alive state after specific url was visite'
|
||||
)
|
||||
else:
|
||||
context.log.info('keep_alive=True, waiting for more requests to come.')
|
||||
|
||||
async def add_request_later(url: str, after_s: int) -> None:
|
||||
"""Add requests to the queue after some time. Can be done by external code."""
|
||||
# Just an example of request being added to the crawler later,
|
||||
# when it is waiting due to `keep_alive=True`.
|
||||
await asyncio.sleep(after_s)
|
||||
await crawler.add_requests([url])
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BasicCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Stop crawler if some guard condition has been met.
|
||||
stop_crawler_if_url_visited(context)
|
||||
|
||||
# Start some tasks that will add some requests later to simulate real situation,
|
||||
# where requests are added later by external code.
|
||||
add_request_later_task1 = asyncio.create_task(
|
||||
add_request_later(url='https://crawlee.dev', after_s=1)
|
||||
)
|
||||
add_request_later_task2 = asyncio.create_task(
|
||||
add_request_later(url='https://crawlee.dev/docs/examples', after_s=5)
|
||||
)
|
||||
|
||||
# Run the crawler without the initial list of requests.
|
||||
# Wait for more requests to be added to the queue later due to `keep_alive=True`.
|
||||
await crawler.run()
|
||||
|
||||
await asyncio.gather(add_request_later_task1, add_request_later_task2)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,41 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Create an instance of the BeautifulSoupCrawler class, a crawler that automatically
|
||||
# loads the URLs and parses their HTML using the BeautifulSoup library.
|
||||
crawler = BeautifulSoupCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
# The handler receives a context parameter, providing various properties and
|
||||
# helper methods. Here are a few key ones we use for demonstration:
|
||||
# - request: an instance of the Request class containing details such as the URL
|
||||
# being crawled and the HTTP method used.
|
||||
# - soup: the BeautifulSoup object containing the parsed HTML of the response.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Create custom condition to stop crawler once it finds what it is looking for.
|
||||
if 'crawlee' in context.request.url:
|
||||
crawler.stop(
|
||||
reason='Manual stop of crawler after finding `crawlee` in the url.'
|
||||
)
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
}
|
||||
|
||||
# Push the extracted data to the default dataset. In local configuration,
|
||||
# the data will be stored as JSON files in ./storage/datasets/default.
|
||||
await context.push_data(data)
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,47 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
from crawlee.storages import KeyValueStore
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
# Headless mode, set to False to see the browser in action.
|
||||
headless=False,
|
||||
# Browser types supported by Playwright.
|
||||
browser_type='chromium',
|
||||
)
|
||||
|
||||
# Open the default key-value store.
|
||||
kvs = await KeyValueStore.open()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Capture the screenshot of the page using Playwright's API.
|
||||
screenshot = await context.page.screenshot()
|
||||
name = context.request.url.split('/')[-1]
|
||||
|
||||
# Store the screenshot in the key-value store.
|
||||
await kvs.set_value(
|
||||
key=f'screenshot-{name}',
|
||||
value=screenshot,
|
||||
content_type='image/png',
|
||||
)
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(
|
||||
[
|
||||
'https://crawlee.dev',
|
||||
'https://apify.com',
|
||||
'https://example.com',
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,91 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import inspect
|
||||
import logging
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from loguru import Record
|
||||
|
||||
|
||||
# Configure loguru interceptor to capture standard logging output
|
||||
class InterceptHandler(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
# Get corresponding Loguru level if it exists
|
||||
try:
|
||||
level: str | int = logger.level(record.levelname).name
|
||||
except ValueError:
|
||||
level = record.levelno
|
||||
|
||||
# Find caller from where originated the logged message
|
||||
frame, depth = inspect.currentframe(), 0
|
||||
while frame:
|
||||
filename = frame.f_code.co_filename
|
||||
is_logging = filename == logging.__file__
|
||||
is_frozen = 'importlib' in filename and '_bootstrap' in filename
|
||||
if depth > 0 and not (is_logging | is_frozen):
|
||||
break
|
||||
frame = frame.f_back
|
||||
depth += 1
|
||||
|
||||
dummy_record = logging.LogRecord('dummy', 0, 'dummy', 0, 'dummy', None, None)
|
||||
standard_attrs = set(dummy_record.__dict__.keys())
|
||||
extra_dict = {
|
||||
key: value
|
||||
for key, value in record.__dict__.items()
|
||||
if key not in standard_attrs
|
||||
}
|
||||
|
||||
(
|
||||
logger.bind(**extra_dict)
|
||||
.opt(depth=depth, exception=record.exc_info)
|
||||
.patch(lambda loguru_record: loguru_record.update({'name': record.name}))
|
||||
.log(level, record.getMessage())
|
||||
)
|
||||
|
||||
|
||||
# Configure loguru formatter
|
||||
def formatter(record: Record) -> str:
|
||||
basic_format = '[{name}] | <level>{level: ^8}</level> | - {message}'
|
||||
if record['extra']:
|
||||
basic_format = basic_format + ' {extra}'
|
||||
return f'{basic_format}\n'
|
||||
|
||||
|
||||
# Remove default loguru logger
|
||||
logger.remove()
|
||||
|
||||
# Set up loguru with JSONL serialization in file `crawler.log`
|
||||
logger.add('crawler.log', format=formatter, serialize=True, level='INFO')
|
||||
|
||||
# Set up loguru logger for console
|
||||
logger.add(sys.stderr, format=formatter, colorize=True, level='INFO')
|
||||
|
||||
# Configure standard logging to use our interceptor
|
||||
logging.basicConfig(handlers=[InterceptHandler()], level=logging.INFO, force=True)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Initialize crawler with disabled table logs
|
||||
crawler = HttpCrawler(
|
||||
configure_logging=False, # Disable default logging configuration
|
||||
statistics_log_format='inline', # Set inline formatting for statistics logs
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: HttpCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Run the crawler
|
||||
await crawler.run(['https://www.crawlee.dev/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,25 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Enqueue all links found on the page.
|
||||
await context.enqueue_links()
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,25 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Enqueue all links found on the page.
|
||||
await context.enqueue_links()
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,25 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(
|
||||
[
|
||||
'https://crawlee.dev',
|
||||
'https://apify.com',
|
||||
'https://example.com',
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,25 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(
|
||||
[
|
||||
'https://crawlee.dev',
|
||||
'https://apify.com',
|
||||
'https://example.com',
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,29 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import Glob
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Enqueue all the documentation links found on the page, except for the examples.
|
||||
await context.enqueue_links(
|
||||
include=[Glob('https://crawlee.dev/docs/**')],
|
||||
exclude=[Glob('https://crawlee.dev/docs/examples')],
|
||||
)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,29 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import Glob
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Enqueue all the documentation links found on the page, except for the examples.
|
||||
await context.enqueue_links(
|
||||
include=[Glob('https://crawlee.dev/docs/**')],
|
||||
exclude=[Glob('https://crawlee.dev/docs/examples')],
|
||||
)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Enqueue all links found on the page. Any URLs found will be matched by
|
||||
# this strategy, even if they go off the site you are currently crawling.
|
||||
await context.enqueue_links(strategy='all')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Setting the strategy to same domain will enqueue all links found that
|
||||
# are on the same hostname as request.loaded_url or request.url.
|
||||
await context.enqueue_links(strategy='same-domain')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Setting the strategy to same hostname will enqueue all links found that are on
|
||||
# the same hostname (including subdomains) as request.loaded_url or request.url.
|
||||
await context.enqueue_links(strategy='same-hostname')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Setting the strategy to same origin will enqueue all links found that are on
|
||||
# the same origin as request.loaded_url or request.url.
|
||||
await context.enqueue_links(strategy='same-origin')
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,39 @@
|
||||
import asyncio
|
||||
import csv
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': context.soup.title.string if context.soup.title else None,
|
||||
}
|
||||
|
||||
# Enqueue all links found on the page.
|
||||
await context.enqueue_links()
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data(data)
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
# Export the entire dataset to a CSV file.
|
||||
# Use semicolon as delimiter and always quote strings.
|
||||
await crawler.export_data(path='results.csv', delimiter=';', quoting=csv.QUOTE_ALL)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,38 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': context.soup.title.string if context.soup.title else None,
|
||||
}
|
||||
|
||||
# Enqueue all links found on the page.
|
||||
await context.enqueue_links()
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data(data)
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
# Export the entire dataset to a JSON file.
|
||||
# Set ensure_ascii=False to allow Unicode characters in the output.
|
||||
await crawler.export_data(path='results.json', ensure_ascii=False)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,36 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import Glob
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract all the documentation links found on the page, except for the examples.
|
||||
extracted_links = await context.extract_links(
|
||||
include=[Glob('https://crawlee.dev/docs/**')],
|
||||
exclude=[Glob('https://crawlee.dev/docs/examples')],
|
||||
)
|
||||
# Some very custom filtering which can't be achieved by `extract_links` arguments.
|
||||
max_link_length = 30
|
||||
filtered_links = [
|
||||
link for link in extracted_links if len(link.url) < max_link_length
|
||||
]
|
||||
# Add filtered links to the request queue.
|
||||
await context.add_requests(filtered_links)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,36 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import Glob
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract all the documentation links found on the page, except for the examples.
|
||||
extracted_links = await context.extract_links(
|
||||
include=[Glob('https://crawlee.dev/docs/**')],
|
||||
exclude=[Glob('https://crawlee.dev/docs/examples')],
|
||||
)
|
||||
# Some very custom filtering which can't be achieved by `extract_links` arguments.
|
||||
max_link_length = 30
|
||||
filtered_links = [
|
||||
link for link in extracted_links if len(link.url) < max_link_length
|
||||
]
|
||||
# Add filtered links to the request queue.
|
||||
await context.add_requests(filtered_links)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,41 @@
|
||||
import asyncio
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from crawlee import Request
|
||||
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = HttpCrawler()
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: HttpCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
response = (await context.http_response.read()).decode('utf-8')
|
||||
context.log.info(f'Response: {response}') # To see the response in the logs.
|
||||
|
||||
# Prepare a POST request to the form endpoint.
|
||||
request = Request.from_url(
|
||||
url='https://httpbin.org/post',
|
||||
method='POST',
|
||||
headers={'content-type': 'application/x-www-form-urlencoded'},
|
||||
payload=urlencode(
|
||||
{
|
||||
'custname': 'John Doe',
|
||||
'custtel': '1234567890',
|
||||
'custemail': 'johndoe@example.com',
|
||||
'size': 'large',
|
||||
'topping': ['bacon', 'cheese', 'mushroom'],
|
||||
'delivery': '13:00',
|
||||
'comments': 'Please ring the doorbell upon arrival.',
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
# Run the crawler with the initial list of requests.
|
||||
await crawler.run([request])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,28 @@
|
||||
import asyncio
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from crawlee import Request
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Prepare a POST request to the form endpoint.
|
||||
request = Request.from_url(
|
||||
url='https://httpbin.org/post',
|
||||
method='POST',
|
||||
headers={'content-type': 'application/x-www-form-urlencoded'},
|
||||
payload=urlencode(
|
||||
{
|
||||
'custname': 'John Doe',
|
||||
'custtel': '1234567890',
|
||||
'custemail': 'johndoe@example.com',
|
||||
'size': 'large',
|
||||
'topping': ['bacon', 'cheese', 'mushroom'],
|
||||
'delivery': '13:00',
|
||||
'comments': 'Please ring the doorbell upon arrival.',
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,47 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import BasicCrawlingContext, ParselCrawler, ParselCrawlingContext
|
||||
|
||||
# Regex for identifying email addresses on a webpage.
|
||||
EMAIL_REGEX = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = ParselCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: ParselCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page.
|
||||
data = {
|
||||
'url': context.request.url,
|
||||
'title': context.selector.xpath('//title/text()').get(),
|
||||
'email_address_list': context.selector.re(EMAIL_REGEX),
|
||||
}
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data(data)
|
||||
|
||||
# Enqueue all links found on the page.
|
||||
await context.enqueue_links()
|
||||
|
||||
# Register pre navigation hook which will be called before each request.
|
||||
# This hook is optional and does not need to be defined at all.
|
||||
@crawler.pre_navigation_hook
|
||||
async def some_hook(context: BasicCrawlingContext) -> None:
|
||||
pass
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://github.com'])
|
||||
|
||||
# Export the entire dataset to a JSON file.
|
||||
await crawler.export_data(path='results.json')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,31 @@
|
||||
import asyncio
|
||||
from random import choice
|
||||
|
||||
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
|
||||
from crawlee.statistics import Statistics
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = ParselCrawler(
|
||||
statistics=Statistics.with_default_state(save_error_snapshots=True)
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: ParselCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
# Simulate various errors to demonstrate `ErrorSnapshotter`
|
||||
# saving only the first occurrence of unique error.
|
||||
await context.enqueue_links()
|
||||
random_number = choice(range(10))
|
||||
if random_number == 1:
|
||||
raise KeyError('Some KeyError')
|
||||
if random_number == 2:
|
||||
raise ValueError('Some ValueError')
|
||||
if random_number == 3:
|
||||
raise RuntimeError('Some RuntimeError')
|
||||
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,36 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import (
|
||||
PlaywrightCrawler,
|
||||
PlaywrightCrawlingContext,
|
||||
PlaywrightPreNavCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
await context.enqueue_links()
|
||||
|
||||
# Define the hook, which will be called before every request.
|
||||
@crawler.pre_navigation_hook
|
||||
async def navigation_hook(context: PlaywrightPreNavCrawlingContext) -> None:
|
||||
context.log.info(f'Navigating to {context.request.url} ...')
|
||||
|
||||
# Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
|
||||
await context.block_requests(extra_url_patterns=['adsbygoogle.js'])
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://crawlee.dev/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,67 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import (
|
||||
PlaywrightCrawler,
|
||||
PlaywrightCrawlingContext,
|
||||
PlaywrightPreNavCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
# Headless mode, set to False to see the browser in action.
|
||||
headless=False,
|
||||
# Browser types supported by Playwright.
|
||||
browser_type='chromium',
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
# The handler receives a context parameter, providing various properties and
|
||||
# helper methods. Here are a few key ones we use for demonstration:
|
||||
# - request: an instance of the Request class containing details such as the URL
|
||||
# being crawled and the HTTP method used.
|
||||
# - page: Playwright's Page object, which allows interaction with the web page
|
||||
# (see https://playwright.dev/python/docs/api/class-page for more details).
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract data from the page using Playwright's API.
|
||||
posts = await context.page.query_selector_all('.athing')
|
||||
data = []
|
||||
|
||||
for post in posts:
|
||||
# Get the HTML elements for the title and rank within each post.
|
||||
title_element = await post.query_selector('.title a')
|
||||
rank_element = await post.query_selector('.rank')
|
||||
|
||||
# Extract the data we want from the elements.
|
||||
title = await title_element.inner_text() if title_element else None
|
||||
rank = await rank_element.inner_text() if rank_element else None
|
||||
href = await title_element.get_attribute('href') if title_element else None
|
||||
|
||||
data.append({'title': title, 'rank': rank, 'href': href})
|
||||
|
||||
# Push the extracted data to the default dataset. In local configuration,
|
||||
# the data will be stored as JSON files in ./storage/datasets/default.
|
||||
await context.push_data(data)
|
||||
|
||||
# Find a link to the next page and enqueue it if it exists.
|
||||
await context.enqueue_links(selector='.morelink')
|
||||
|
||||
# Define a hook that will be called each time before navigating to a new URL.
|
||||
# The hook receives a context parameter, providing access to the request and
|
||||
# browser page among other things. In this example, we log the URL being
|
||||
# navigated to.
|
||||
@crawler.pre_navigation_hook
|
||||
async def log_navigation_url(context: PlaywrightPreNavCrawlingContext) -> None:
|
||||
context.log.info(f'Navigating to {context.request.url} ...')
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://news.ycombinator.com/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,69 @@
|
||||
import asyncio
|
||||
|
||||
# Camoufox is external package and needs to be installed. It is not included in crawlee.
|
||||
from camoufox import AsyncNewBrowser
|
||||
from typing_extensions import override
|
||||
|
||||
from crawlee.browsers import (
|
||||
BrowserPool,
|
||||
PlaywrightBrowserController,
|
||||
PlaywrightBrowserPlugin,
|
||||
)
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
|
||||
class CamoufoxPlugin(PlaywrightBrowserPlugin):
|
||||
"""Example browser plugin that uses Camoufox browser,
|
||||
but otherwise keeps the functionality of PlaywrightBrowserPlugin.
|
||||
"""
|
||||
|
||||
@override
|
||||
async def new_browser(self) -> PlaywrightBrowserController:
|
||||
if not self._playwright:
|
||||
raise RuntimeError('Playwright browser plugin is not initialized.')
|
||||
|
||||
return PlaywrightBrowserController(
|
||||
browser=await AsyncNewBrowser(
|
||||
self._playwright, **self._browser_launch_options
|
||||
),
|
||||
# Increase, if camoufox can handle it in your use case.
|
||||
max_open_pages_per_browser=1,
|
||||
# This turns off the crawlee header_generation. Camoufox has its own.
|
||||
header_generator=None,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
# Custom browser pool. Gives users full control over browsers used by the crawler.
|
||||
browser_pool=BrowserPool(plugins=[CamoufoxPlugin()]),
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Extract some data from the page using Playwright's API.
|
||||
posts = await context.page.query_selector_all('.athing')
|
||||
for post in posts:
|
||||
# Get the HTML elements for the title and rank within each post.
|
||||
title_element = await post.query_selector('.title a')
|
||||
|
||||
# Extract the data we want from the elements.
|
||||
title = await title_element.inner_text() if title_element else None
|
||||
|
||||
# Push the extracted data to the default dataset.
|
||||
await context.push_data({'title': title})
|
||||
|
||||
# Find a link to the next page and enqueue it if it exists.
|
||||
await context.enqueue_links(selector='.morelink')
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://news.ycombinator.com/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,31 @@
|
||||
import asyncio
|
||||
from random import choice
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
from crawlee.statistics import Statistics
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
statistics=Statistics.with_default_state(save_error_snapshots=True)
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
# Simulate various errors to demonstrate `ErrorSnapshotter`
|
||||
# saving only the first occurrence of unique error.
|
||||
await context.enqueue_links()
|
||||
random_number = choice(range(10))
|
||||
if random_number == 1:
|
||||
raise KeyError('Some KeyError')
|
||||
if random_number == 2:
|
||||
raise ValueError('Some ValueError')
|
||||
if random_number == 3:
|
||||
raise RuntimeError('Some RuntimeError')
|
||||
|
||||
await crawler.run(['https://crawlee.dev'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,44 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
from crawlee.fingerprint_suite import (
|
||||
DefaultFingerprintGenerator,
|
||||
HeaderGeneratorOptions,
|
||||
ScreenOptions,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Use default fingerprint generator with desired fingerprint options.
|
||||
# Generator will generate real looking browser fingerprint based on the options.
|
||||
# Unspecified fingerprint options will be automatically selected by the generator.
|
||||
fingerprint_generator = DefaultFingerprintGenerator(
|
||||
header_options=HeaderGeneratorOptions(browsers=['chrome']),
|
||||
screen_options=ScreenOptions(min_width=400),
|
||||
)
|
||||
|
||||
crawler = PlaywrightCrawler(
|
||||
# Limit the crawl to max requests. Remove or increase it for crawling all links.
|
||||
max_requests_per_crawl=10,
|
||||
# Headless mode, set to False to see the browser in action.
|
||||
headless=False,
|
||||
# Browser types supported by Playwright.
|
||||
browser_type='chromium',
|
||||
# Fingerprint generator to be used. By default no fingerprint generation is done.
|
||||
fingerprint_generator=fingerprint_generator,
|
||||
)
|
||||
|
||||
# Define the default request handler, which will be called for every request.
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Find a link to the next page and enqueue it if it exists.
|
||||
await context.enqueue_links(selector='.morelink')
|
||||
|
||||
# Run the crawler with the initial list of URLs.
|
||||
await crawler.run(['https://news.ycombinator.com/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,36 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import SkippedReason
|
||||
from crawlee.crawlers import (
|
||||
BeautifulSoupCrawler,
|
||||
BeautifulSoupCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Initialize the crawler with robots.txt compliance enabled
|
||||
crawler = BeautifulSoupCrawler(respect_robots_txt_file=True)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# highlight-start
|
||||
# This handler is called when a request is skipped
|
||||
@crawler.on_skipped_request
|
||||
async def skipped_request_handler(url: str, reason: SkippedReason) -> None:
|
||||
# Check if the request was skipped due to robots.txt rules
|
||||
if reason == 'robots_txt':
|
||||
crawler.log.info(f'Skipped {url} due to robots.txt rules.')
|
||||
|
||||
# highlight-end
|
||||
|
||||
# Start the crawler with the specified URLs
|
||||
# The login URL will be skipped and handled by the skipped_request_handler
|
||||
await crawler.run(
|
||||
['https://news.ycombinator.com/', 'https://news.ycombinator.com/login']
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,27 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee.crawlers import (
|
||||
BeautifulSoupCrawler,
|
||||
BeautifulSoupCrawlingContext,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Initialize the crawler with robots.txt compliance enabled
|
||||
crawler = BeautifulSoupCrawler(respect_robots_txt_file=True)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# Start the crawler with the specified URLs
|
||||
# The crawler will check the robots.txt file before making requests
|
||||
# In this example, 'https://news.ycombinator.com/login' will be skipped
|
||||
# because it's disallowed in the site's robots.txt file
|
||||
await crawler.run(
|
||||
['https://news.ycombinator.com/', 'https://news.ycombinator.com/login']
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,40 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import ConcurrencySettings, service_locator
|
||||
from crawlee.crawlers import (
|
||||
BeautifulSoupCrawler,
|
||||
BeautifulSoupCrawlingContext,
|
||||
)
|
||||
|
||||
# Disable clearing the `RequestQueue`, `KeyValueStore` and `Dataset` on each run.
|
||||
# This makes the scraper continue from where it left off in the previous run.
|
||||
# The recommended way to achieve this behavior is setting the environment variable
|
||||
# `CRAWLEE_PURGE_ON_START=0`
|
||||
configuration = service_locator.get_configuration()
|
||||
configuration.purge_on_start = False
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = BeautifulSoupCrawler(
|
||||
# Let's slow down the crawler for a demonstration
|
||||
concurrency_settings=ConcurrencySettings(max_tasks_per_minute=20)
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
context.log.info(f'Processing {context.request.url} ...')
|
||||
|
||||
# List of links for crawl
|
||||
requests = [
|
||||
'https://crawlee.dev',
|
||||
'https://crawlee.dev/python/docs',
|
||||
'https://crawlee.dev/python/docs/examples',
|
||||
'https://crawlee.dev/python/docs/guides',
|
||||
'https://crawlee.dev/python/docs/quick-start',
|
||||
]
|
||||
|
||||
await crawler.run(requests)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,94 @@
|
||||
import asyncio
|
||||
|
||||
from crawlee import ConcurrencySettings
|
||||
from crawlee.crawlers import (
|
||||
ParselCrawler,
|
||||
ParselCrawlingContext,
|
||||
PlaywrightCrawler,
|
||||
PlaywrightCrawlingContext,
|
||||
)
|
||||
from crawlee.sessions import SessionPool
|
||||
from crawlee.storages import RequestQueue
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Open request queues for both crawlers with different aliases
|
||||
playwright_rq = await RequestQueue.open(alias='playwright-requests')
|
||||
parsel_rq = await RequestQueue.open(alias='parsel-requests')
|
||||
|
||||
# Use a shared session pool between both crawlers
|
||||
async with SessionPool() as session_pool:
|
||||
playwright_crawler = PlaywrightCrawler(
|
||||
# Set the request queue for Playwright crawler
|
||||
request_manager=playwright_rq,
|
||||
session_pool=session_pool,
|
||||
# Configure concurrency settings for Playwright crawler
|
||||
concurrency_settings=ConcurrencySettings(
|
||||
max_concurrency=5, desired_concurrency=5
|
||||
),
|
||||
# Set `keep_alive`` so that the crawler does not stop working when there are
|
||||
# no requests in the queue.
|
||||
keep_alive=True,
|
||||
)
|
||||
|
||||
parsel_crawler = ParselCrawler(
|
||||
# Set the request queue for Parsel crawler
|
||||
request_manager=parsel_rq,
|
||||
session_pool=session_pool,
|
||||
# Configure concurrency settings for Parsel crawler
|
||||
concurrency_settings=ConcurrencySettings(
|
||||
max_concurrency=10, desired_concurrency=10
|
||||
),
|
||||
# Set maximum requests per crawl for Parsel crawler
|
||||
max_requests_per_crawl=50,
|
||||
)
|
||||
|
||||
@playwright_crawler.router.default_handler
|
||||
async def handle_playwright(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Playwright Processing {context.request.url}...')
|
||||
|
||||
title = await context.page.title()
|
||||
# Push the extracted data to the dataset for Playwright crawler
|
||||
await context.push_data(
|
||||
{'title': title, 'url': context.request.url, 'source': 'playwright'},
|
||||
dataset_name='playwright-data',
|
||||
)
|
||||
|
||||
@parsel_crawler.router.default_handler
|
||||
async def handle_parsel(context: ParselCrawlingContext) -> None:
|
||||
context.log.info(f'Parsel Processing {context.request.url}...')
|
||||
|
||||
title = context.parsed_content.css('title::text').get()
|
||||
# Push the extracted data to the dataset for Parsel crawler
|
||||
await context.push_data(
|
||||
{'title': title, 'url': context.request.url, 'source': 'parsel'},
|
||||
dataset_name='parsel-data',
|
||||
)
|
||||
|
||||
# Enqueue links to the Playwright request queue for blog pages
|
||||
await context.enqueue_links(
|
||||
selector='a[href*="/blog/"]', rq_alias='playwright-requests'
|
||||
)
|
||||
# Enqueue other links to the Parsel request queue
|
||||
await context.enqueue_links(selector='a:not([href*="/blog/"])')
|
||||
|
||||
# Start the Playwright crawler in the background
|
||||
background_crawler_task = asyncio.create_task(playwright_crawler.run([]))
|
||||
|
||||
# Run the Parsel crawler with the initial URL and wait for it to finish
|
||||
await parsel_crawler.run(['https://crawlee.dev/blog'])
|
||||
|
||||
# Wait for the Playwright crawler to finish processing all requests
|
||||
while not await playwright_rq.is_empty():
|
||||
playwright_crawler.log.info('Waiting for Playwright crawler to finish...')
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# Stop the Playwright crawler after all requests are processed
|
||||
playwright_crawler.stop()
|
||||
|
||||
# Wait for the background Playwright crawler task to complete
|
||||
await background_crawler_task
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,54 @@
|
||||
import asyncio
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
# Profile name to use (usually 'Default' for single profile setups)
|
||||
PROFILE_NAME = 'Default'
|
||||
|
||||
# Paths to Chrome profiles in your system (example for Windows)
|
||||
# Use `chrome://version/` to find your profile path
|
||||
PROFILE_PATH = Path(Path.home(), 'AppData', 'Local', 'Google', 'Chrome', 'User Data')
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Create a temporary folder to copy the profile to
|
||||
with TemporaryDirectory(prefix='crawlee-') as tmpdirname:
|
||||
tmp_profile_dir = Path(tmpdirname)
|
||||
|
||||
# Copy the profile to a temporary folder
|
||||
shutil.copytree(
|
||||
PROFILE_PATH / PROFILE_NAME,
|
||||
tmp_profile_dir / PROFILE_NAME,
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
|
||||
crawler = PlaywrightCrawler(
|
||||
headless=False,
|
||||
# Use the installed Chrome browser
|
||||
browser_type='chrome',
|
||||
# Disable fingerprints to preserve profile identity
|
||||
fingerprint_generator=None,
|
||||
# Set user data directory to temp folder
|
||||
user_data_dir=tmp_profile_dir,
|
||||
browser_launch_options={
|
||||
# Slow down actions to mimic human behavior
|
||||
'slow_mo': 200,
|
||||
'args': [
|
||||
# Use the specified profile
|
||||
f'--profile-directory={PROFILE_NAME}',
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def default_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Visiting {context.request.url}')
|
||||
|
||||
await crawler.run(['https://crawlee.dev/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,42 @@
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
|
||||
|
||||
# Replace this with your actual Firefox profile name
|
||||
# Find it at about:profiles in Firefox
|
||||
PROFILE_NAME = 'your-profile-name-here'
|
||||
|
||||
# Paths to Firefox profiles in your system (example for Windows)
|
||||
# Use `about:profiles` to find your profile path
|
||||
PROFILE_PATH = Path(
|
||||
Path.home(), 'AppData', 'Roaming', 'Mozilla', 'Firefox', 'Profiles', PROFILE_NAME
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
crawler = PlaywrightCrawler(
|
||||
# Use Firefox browser type
|
||||
browser_type='firefox',
|
||||
# Disable fingerprints to use the profile as is
|
||||
fingerprint_generator=None,
|
||||
headless=False,
|
||||
# Path to your Firefox profile
|
||||
user_data_dir=PROFILE_PATH,
|
||||
browser_launch_options={
|
||||
'args': [
|
||||
# Required to avoid version conflicts
|
||||
'--allow-downgrade'
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
@crawler.router.default_handler
|
||||
async def default_handler(context: PlaywrightCrawlingContext) -> None:
|
||||
context.log.info(f'Visiting {context.request.url}')
|
||||
|
||||
await crawler.run(['https://crawlee.dev/'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,101 @@
|
||||
import asyncio
|
||||
from collections.abc import Callable
|
||||
|
||||
from yarl import URL
|
||||
|
||||
from crawlee import RequestOptions, RequestTransformAction
|
||||
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
|
||||
from crawlee.http_clients import ImpitHttpClient
|
||||
from crawlee.request_loaders import SitemapRequestLoader
|
||||
|
||||
|
||||
# Create a transform_request_function that maps request options based on the host in
|
||||
# the URL
|
||||
def create_transform_request(
|
||||
data_mapper: dict[str, dict],
|
||||
) -> Callable[[RequestOptions], RequestOptions | RequestTransformAction]:
|
||||
def transform_request(
|
||||
request_options: RequestOptions,
|
||||
) -> RequestOptions | RequestTransformAction:
|
||||
# According to the Sitemap protocol, all URLs in a Sitemap must be from a single
|
||||
# host.
|
||||
request_host = URL(request_options['url']).host
|
||||
|
||||
if request_host and (mapping_data := data_mapper.get(request_host)):
|
||||
# Set properties from the mapping data
|
||||
if 'label' in mapping_data:
|
||||
request_options['label'] = mapping_data['label']
|
||||
if 'user_data' in mapping_data:
|
||||
request_options['user_data'] = mapping_data['user_data']
|
||||
|
||||
return request_options
|
||||
|
||||
return 'unchanged'
|
||||
|
||||
return transform_request
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# Prepare data mapping for hosts
|
||||
apify_host = URL('https://apify.com/sitemap.xml').host
|
||||
crawlee_host = URL('https://crawlee.dev/sitemap.xml').host
|
||||
|
||||
if not apify_host or not crawlee_host:
|
||||
raise ValueError('Unable to extract host from URLs')
|
||||
|
||||
data_map = {
|
||||
apify_host: {
|
||||
'label': 'apify',
|
||||
'user_data': {'source': 'apify'},
|
||||
},
|
||||
crawlee_host: {
|
||||
'label': 'crawlee',
|
||||
'user_data': {'source': 'crawlee'},
|
||||
},
|
||||
}
|
||||
|
||||
# Initialize the SitemapRequestLoader with the transform function
|
||||
async with SitemapRequestLoader(
|
||||
# Set the sitemap URLs and the HTTP client
|
||||
sitemap_urls=['https://crawlee.dev/sitemap.xml', 'https://apify.com/sitemap.xml'],
|
||||
http_client=ImpitHttpClient(),
|
||||
transform_request_function=create_transform_request(data_map),
|
||||
) as sitemap_loader:
|
||||
# Convert the sitemap loader to a request manager
|
||||
request_manager = await sitemap_loader.to_tandem()
|
||||
|
||||
# Create and configure the crawler
|
||||
crawler = BeautifulSoupCrawler(
|
||||
request_manager=request_manager,
|
||||
max_requests_per_crawl=10,
|
||||
)
|
||||
|
||||
# Create default handler for requests without a specific label
|
||||
@crawler.router.default_handler
|
||||
async def handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
source = context.request.user_data.get('source', 'unknown')
|
||||
context.log.info(
|
||||
f'Processing request: {context.request.url} from source: {source}'
|
||||
)
|
||||
|
||||
# Create handler for requests labeled 'apify'
|
||||
@crawler.router.handler('apify')
|
||||
async def apify_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
source = context.request.user_data.get('source', 'unknown')
|
||||
context.log.info(
|
||||
f'Apify handler processing: {context.request.url} from source: {source}'
|
||||
)
|
||||
|
||||
# Create handler for requests labeled 'crawlee'
|
||||
@crawler.router.handler('crawlee')
|
||||
async def crawlee_handler(context: BeautifulSoupCrawlingContext) -> None:
|
||||
source = context.request.user_data.get('source', 'unknown')
|
||||
context.log.info(
|
||||
f'Crawlee handler processing: {context.request.url} from source: {source}'
|
||||
)
|
||||
|
||||
await crawler.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: crawl-all-links-on-website
|
||||
title: Crawl all links on website
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_bs.py';
|
||||
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_all_links_on_website_pw.py';
|
||||
|
||||
This example uses the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> helper to add new links to the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> as the crawler navigates from page to page. By automatically discovering and enqueuing all links on a given page, the crawler can systematically scrape an entire website. This approach is ideal for web scraping tasks where you need to collect data from multiple interconnected pages.
|
||||
|
||||
:::tip
|
||||
|
||||
If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the `strategy` option, which is an instance of the `EnqueueStrategy` type alias. You can find more info about this option in the [Crawl website with relative links](./crawl-website-with-relative-links) example.
|
||||
|
||||
:::
|
||||
|
||||
<Tabs groupId="main">
|
||||
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
id: crawl-multiple-urls
|
||||
title: Crawl multiple URLs
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_multiple_urls_bs.py';
|
||||
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_multiple_urls_pw.py';
|
||||
|
||||
This example demonstrates how to crawl a specified list of URLs using different crawlers. You'll learn how to set up the crawler, define a request handler, and run the crawler with multiple URLs. This setup is useful for scraping data from multiple pages or websites concurrently.
|
||||
|
||||
<Tabs groupId="main">
|
||||
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
id: crawl-specific-links-on-website
|
||||
title: Crawl specific links on website
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/crawl_specific_links_on_website_bs.py';
|
||||
import PlaywrightExample from '!!raw-loader!roa-loader!./code_examples/crawl_specific_links_on_website_pw.py';
|
||||
|
||||
import BeautifulSoupExampleExtractAndAdd from '!!raw-loader!roa-loader!./code_examples/extract_and_add_specific_links_on_website_bs.py';
|
||||
import PlaywrightExampleExtractAndAdd from '!!raw-loader!roa-loader!./code_examples/extract_and_add_specific_links_on_website_pw.py';
|
||||
|
||||
This example demonstrates how to crawl a website while targeting specific patterns of links. By utilizing the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> helper, you can pass `include` or `exclude` parameters to improve your crawling strategy. This approach ensures that only the links matching the specified patterns are added to the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>. Both `include` and `exclude` support lists of globs or regular expressions. This functionality is great for focusing on relevant sections of a website and avoiding scraping unnecessary or irrelevant content.
|
||||
|
||||
<Tabs groupId="first-example">
|
||||
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Even more control over the enqueued links
|
||||
|
||||
<ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> is a convenience helper and internally it calls <ApiLink to="class/ExtractLinksFunction">`extract_links`</ApiLink> to find the links and <ApiLink to="class/AddRequestsFunction">`add_requests`</ApiLink> to add them to the queue. If you need some additional custom filtering of the extracted links before enqueuing them, then consider using <ApiLink to="class/ExtractLinksFunction">`extract_links`</ApiLink> and <ApiLink to="class/AddRequestsFunction">`add_requests`</ApiLink> instead of the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink>
|
||||
|
||||
<Tabs groupId="second-example">
|
||||
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExampleExtractAndAdd}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightExampleExtractAndAdd}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
id: crawl-website-with-relative-links
|
||||
title: Crawl website with relative links
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import AllLinksExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_all_links.py';
|
||||
import SameDomainExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_domain.py';
|
||||
import SameHostnameExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_hostname.py';
|
||||
import SameOriginExample from '!!raw-loader!roa-loader!./code_examples/crawl_website_with_relative_links_same_origin.py';
|
||||
|
||||
When crawling a website, you may encounter various types of links that you wish to include in your crawl. To facilitate this, we provide the <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink> method on the crawler context, which will automatically find and add these links to the crawler's <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>. This method simplifies the process of handling different types of links, including relative links, by automatically resolving them based on the page's context.
|
||||
|
||||
:::note
|
||||
|
||||
For these examples, we are using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. However, the same method is available for other crawlers as well. You can use it in exactly the same way.
|
||||
|
||||
:::
|
||||
|
||||
`EnqueueStrategy` type alias provides four distinct strategies for crawling relative links:
|
||||
|
||||
- `all` - Enqueues all links found, regardless of the domain they point to. This strategy is useful when you want to follow every link, including those that navigate to external websites.
|
||||
- `same-domain` - Enqueues all links found that share the same domain name, including any possible subdomains. This strategy ensures that all links within the same top-level and base domain are included.
|
||||
- `same-hostname` - Enqueues all links found for the exact same hostname. This is the **default** strategy, and it restricts the crawl to links that have the same hostname as the current page, excluding subdomains.
|
||||
- `same-origin` - Enqueues all links found that share the same origin. The same origin refers to URLs that share the same protocol, domain, and port, ensuring a strict scope for the crawl.
|
||||
|
||||
<Tabs groupId="main">
|
||||
<TabItem value="all_links" label="All links">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{AllLinksExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="same-domain" label="Same domain">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{SameDomainExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="same-hostname" label="Same hostname">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{SameHostnameExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="same-origin" label="Same origin">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{SameOriginExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
id: crawler-keep-alive
|
||||
title: Keep a Crawler alive waiting for more requests
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_keep_alive.py';
|
||||
|
||||
This example demonstrates how to keep crawler alive even when there are no requests at the moment by using `keep_alive=True` argument of <ApiLink to="class/BasicCrawler#__init__">`BasicCrawler.__init__`</ApiLink>. This is available to all crawlers that inherit from <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> and in the example below it is shown on <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. To stop the crawler that was started with `keep_alive=True` you can call `crawler.stop()`.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
id: crawler-stop
|
||||
title: Stopping a Crawler with stop method
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import BeautifulSoupExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_stop.py';
|
||||
|
||||
This example demonstrates how to use `stop` method of <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> to stop crawler once the crawler finds what it is looking for. This method is available to all crawlers that inherit from <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> and in the example below it is shown on <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. Simply call `crawler.stop()` to stop the crawler. It will not continue to crawl through new requests. Requests that are already being concurrently processed are going to get finished. It is possible to call `stop` method with optional argument `reason` that is a string that will be used in logs and it can improve logs readability especially if you have multiple different conditions for triggering `stop`.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{BeautifulSoupExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: export-entire-dataset-to-file
|
||||
title: Export entire dataset to file
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import JsonExample from '!!raw-loader!roa-loader!./code_examples/export_entire_dataset_to_file_json.py';
|
||||
import CsvExample from '!!raw-loader!roa-loader!./code_examples/export_entire_dataset_to_file_csv.py';
|
||||
|
||||
This example demonstrates how to use the <ApiLink to="class/BasicCrawler#export_data">`BasicCrawler.export_data`</ApiLink> method of the crawler to export the entire default dataset to a single file. This method supports exporting data in either CSV or JSON format and also accepts additional keyword arguments so you can fine-tune the underlying `json.dump` or `csv.writer` behavior.
|
||||
|
||||
:::note
|
||||
|
||||
For these examples, we are using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>. However, the same method is available for other crawlers as well. You can use it in exactly the same way.
|
||||
|
||||
:::
|
||||
|
||||
<Tabs groupId="main">
|
||||
<TabItem value="json" label="JSON">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{JsonExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
<TabItem value="csv" label="CSV">
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{CsvExample}
|
||||
</RunnableCodeBlock>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
id: fill-and-submit-web-form
|
||||
title: Fill and submit web form
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import RequestExample from '!!raw-loader!roa-loader!./code_examples/fill_and_submit_web_form_request.py';
|
||||
import CrawlerExample from '!!raw-loader!roa-loader!./code_examples/fill_and_submit_web_form_crawler.py';
|
||||
|
||||
This example demonstrates how to fill and submit a web form using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink> crawler. The same approach applies to any crawler that inherits from it, such as the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> or <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>.
|
||||
|
||||
We are going to use the [httpbin.org](https://httpbin.org) website to demonstrate how it works.
|
||||
|
||||
## Investigate the form fields
|
||||
|
||||
First, we need to examine the form fields and the form's action URL. You can do this by opening the [httpbin.org/forms/post](https://httpbin.org/forms/post) page in a browser and inspecting the form fields.
|
||||
|
||||
In Chrome, right-click on the page and select "Inspect" or press `Ctrl+Shift+I`.
|
||||
Use the element selector (`Ctrl+Shift+C`) to click on the form element you want to inspect.
|
||||
|
||||

|
||||
|
||||
Identify the field names. For example, the customer name field is `custname`, the email field is `custemail`, and the phone field is `custtel`.
|
||||
|
||||
Now navigate to the "Network" tab in developer tools and submit the form by clicking the "Submit order" button.
|
||||
|
||||

|
||||
|
||||
Find the form submission request and examine its details. The "Headers" tab will show the submission URL, in this case, it is `https://httpbin.org/post`.
|
||||
|
||||

|
||||
|
||||
The "Payload" tab will display the form fields and their submitted values. This method could be an alternative to inspecting the HTML source code directly.
|
||||
|
||||

|
||||
|
||||
## Preparing a POST request
|
||||
|
||||
Now, let's create a POST request with the form fields and their values using the <ApiLink to="class/Request">`Request`</ApiLink> class, specifically its <ApiLink to="class/Request#from_url">`Request.from_url`</ApiLink> constructor:
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{RequestExample}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
Alternatively, you can send form data as URL parameters using the `url` argument. It depends on the form and how it is implemented. However, sending the data as a POST request body using the `payload` is generally a better approach.
|
||||
|
||||
## Implementing the crawler
|
||||
|
||||
Finally, let's implement the crawler and run it with the prepared request. Although we are using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>, the process is the same for any crawler that inherits from it.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{CrawlerExample}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
## Running the crawler
|
||||
|
||||
Finally, run your crawler. Your logs should show something like this:
|
||||
|
||||
```plaintext
|
||||
...
|
||||
[crawlee.http_crawler._http_crawler] INFO Processing https://httpbin.org/post ...
|
||||
[crawlee.http_crawler._http_crawler] INFO Response: {
|
||||
"args": {},
|
||||
"data": "",
|
||||
"files": {},
|
||||
"form": {
|
||||
"comments": "Please ring the doorbell upon arrival.",
|
||||
"custemail": "johndoe@example.com",
|
||||
"custname": "John Doe",
|
||||
"custtel": "1234567890",
|
||||
"delivery": "13:00",
|
||||
"size": "large",
|
||||
"topping": [
|
||||
"bacon",
|
||||
"cheese",
|
||||
"mushroom"
|
||||
]
|
||||
},
|
||||
"headers": {
|
||||
"Accept": "*/*",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Content-Length": "190",
|
||||
"Content-Type": "application/x-www-form-urlencoded",
|
||||
"Host": "httpbin.org",
|
||||
"User-Agent": "python-httpx/0.27.0",
|
||||
"X-Amzn-Trace-Id": "Root=1-66c849d6-1ae432fb7b4156e6149ff37f"
|
||||
},
|
||||
"json": null,
|
||||
"origin": "78.80.81.196",
|
||||
"url": "https://httpbin.org/post"
|
||||
}
|
||||
|
||||
[crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
|
||||
[crawlee.http_crawler._http_crawler] INFO Final request statistics:
|
||||
┌───────────────────────────────┬──────────┐
|
||||
│ requests_finished │ 1 │
|
||||
│ requests_failed │ 0 │
|
||||
│ retry_histogram │ [1] │
|
||||
│ request_avg_failed_duration │ None │
|
||||
│ request_avg_finished_duration │ 0.678442 │
|
||||
│ requests_finished_per_minute │ 85 │
|
||||
│ requests_failed_per_minute │ 0 │
|
||||
│ request_total_duration │ 0.678442 │
|
||||
│ requests_total │ 1 │
|
||||
│ crawler_runtime │ 0.707666 │
|
||||
└───────────────────────────────┴──────────┘
|
||||
```
|
||||
|
||||
This log output confirms that the crawler successfully submitted the form and processed the response. Congratulations! You have successfully filled and submitted a web form using the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
id: configure-json-logging
|
||||
title: Сonfigure JSON logging
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import JsonLoggingExample from '!!raw-loader!roa-loader!./code_examples/configure_json_logging.py';
|
||||
|
||||
This example demonstrates how to configure JSON line (JSONL) logging with Crawlee. By using the `use_table_logs=False` parameter, you can disable table-formatted statistics logs, which makes it easier to parse logs with external tools or to serialize them as JSON.
|
||||
|
||||
The example shows how to integrate with the popular [`loguru`](https://github.com/delgan/loguru) library to capture Crawlee logs and format them as JSONL (one JSON object per line). This approach works well when you need to collect logs for analysis, monitoring, or when integrating with logging platforms like ELK Stack, Grafana Loki, or similar systems.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{JsonLoggingExample}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
Here's an example of what a crawler statistics log entry in JSONL format.
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "[HttpCrawler] | INFO | - Final request statistics: {'requests_finished': 1, 'requests_failed': 0, 'retry_histogram': [1], 'request_avg_failed_duration': None, 'request_avg_finished_duration': 3.57098, 'requests_finished_per_minute': 17, 'requests_failed_per_minute': 0, 'request_total_duration': 3.57098, 'requests_total': 1, 'crawler_runtime': 3.59165}\n",
|
||||
"record": {
|
||||
"elapsed": { "repr": "0:00:05.604568", "seconds": 5.604568 },
|
||||
"exception": null,
|
||||
"extra": {
|
||||
"requests_finished": 1,
|
||||
"requests_failed": 0,
|
||||
"retry_histogram": [1],
|
||||
"request_avg_failed_duration": null,
|
||||
"request_avg_finished_duration": 3.57098,
|
||||
"requests_finished_per_minute": 17,
|
||||
"requests_failed_per_minute": 0,
|
||||
"request_total_duration": 3.57098,
|
||||
"requests_total": 1,
|
||||
"crawler_runtime": 3.59165
|
||||
},
|
||||
"file": {
|
||||
"name": "_basic_crawler.py",
|
||||
"path": "/crawlers/_basic/_basic_crawler.py"
|
||||
},
|
||||
"function": "run",
|
||||
"level": { "icon": "ℹ️", "name": "INFO", "no": 20 },
|
||||
"line": 583,
|
||||
"message": "Final request statistics:",
|
||||
"module": "_basic_crawler",
|
||||
"name": "HttpCrawler",
|
||||
"process": { "id": 198383, "name": "MainProcess" },
|
||||
"thread": { "id": 135312814966592, "name": "MainThread" },
|
||||
"time": {
|
||||
"repr": "2025-03-17 17:14:45.339150+00:00",
|
||||
"timestamp": 1742231685.33915
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
id: parsel-crawler
|
||||
title: Parsel crawler
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import ParselCrawlerExample from '!!raw-loader!roa-loader!./code_examples/parsel_crawler.py';
|
||||
|
||||
This example shows how to use <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> to crawl a website or a list of URLs. Each URL is loaded using a plain HTTP request and the response is parsed using [Parsel](https://pypi.org/project/parsel/) library which supports CSS and XPath selectors for HTML responses and JMESPath for JSON responses. We can extract data from all kinds of complex HTML structures using XPath. In this example, we will use Parsel to crawl github.com and extract page title, URL and emails found in the webpage. The default handler will scrape data from the current webpage and enqueue all the links found in the webpage for continuous scraping. It also shows how you can add optional pre-navigation hook to the crawler. Pre-navigation hooks are user defined functions that execute before sending the request.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{ParselCrawlerExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
id: playwright-crawler
|
||||
title: Playwright crawler
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler.py';
|
||||
|
||||
This example demonstrates how to use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to recursively scrape the Hacker news website using headless Chromium and Playwright.
|
||||
|
||||
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> manages the browser and page instances, simplifying the process of interacting with web pages. In the request handler, Playwright's API is used to extract data from each post on the page. Specifically, it retrieves the title, rank, and URL of each post. Additionally, the handler enqueues links to the next pages to ensure continuous scraping. This setup is ideal for scraping dynamic web pages where JavaScript execution is required to render the content.
|
||||
|
||||
A **pre-navigation hook** can be used to perform actions before navigating to the URL. This hook provides further flexibility in controlling environment and preparing for navigation.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightCrawlerExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: adaptive-playwright-crawler
|
||||
title: Adaptive Playwright crawler
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import AdaptivePlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/adaptive_playwright_crawler.py';
|
||||
|
||||
This example demonstrates how to use <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink>. An <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> is a combination of <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and some implementation of HTTP-based crawler such as <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> or <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>.
|
||||
It uses a more limited crawling context interface so that it is able to switch to HTTP-only crawling when it detects that it may bring a performance benefit.
|
||||
|
||||
A [pre-navigation hook](/python/docs/guides/adaptive-playwright-crawler#page-configuration-with-pre-navigation-hooks) can be used to perform actions before navigating to the URL. This hook provides further flexibility in controlling environment and preparing for navigation. Hooks will be executed both for the pages crawled by HTTP-bases sub crawler and playwright based sub crawler. Use `playwright_only=True` to mark hooks that should be executed only for playwright sub crawler.
|
||||
|
||||
For more detailed description please see [Adaptive Playwright crawler guide](/python/docs/guides/adaptive-playwright-crawler)
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{AdaptivePlaywrightCrawlerExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
id: playwright-crawler-with-block-requests
|
||||
title: Playwright crawler with block requests
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import PlaywrightBlockRequests from '!!raw-loader!roa-loader!./code_examples/playwright_block_requests.py';
|
||||
|
||||
This example demonstrates how to optimize your <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> performance by blocking unnecessary network requests.
|
||||
|
||||
The primary use case is when you need to scrape or interact with web pages without loading non-essential resources like images, styles, or analytics scripts. This can significantly reduce bandwidth usage and improve crawling speed.
|
||||
|
||||
The <ApiLink to="class/BlockRequestsFunction">`block_requests`</ApiLink> helper provides the most efficient way to block requests as it operates directly in the browser.
|
||||
|
||||
By default, <ApiLink to="class/BlockRequestsFunction">`block_requests`</ApiLink> will block all URLs including the following patterns:
|
||||
|
||||
```python
|
||||
['.css', '.webp', '.jpg', '.jpeg', '.png', '.svg', '.gif', '.woff', '.pdf', '.zip']
|
||||
```
|
||||
|
||||
You can also replace the default patterns list with your own by providing `url_patterns`, or extend it by passing additional patterns in `extra_url_patterns`.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightBlockRequests}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
id: playwright-crawler-with-camoufox
|
||||
title: Playwright crawler with Camoufox
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import PlaywrightCrawlerExampleWithCamoufox from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_camoufox.py';
|
||||
|
||||
This example demonstrates how to integrate Camoufox into <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> using <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink> with custom <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>.
|
||||
|
||||
Camoufox is a stealthy minimalistic build of Firefox. For details please visit its homepage https://camoufox.com/ .
|
||||
To be able to run this example you will need to install camoufox, as it is external tool, and it is not part of the crawlee. For installation please see https://pypi.org/project/camoufox/.
|
||||
|
||||
**Warning!** Camoufox is using custom build of firefox. This build can be hundreds of MB large.
|
||||
You can either pre-download this file using following command `python3 -m camoufox fetch` or camoufox will download it automatically once you try to run it, and it does not find existing binary.
|
||||
For more details please refer to: https://github.com/daijro/camoufox/tree/main/pythonlib#camoufox-python-interface
|
||||
|
||||
**Project template -** It is possible to generate project with Python code which includes Camoufox integration into crawlee through crawlee cli. Call `crawlee create` and pick `Playwright-camoufox` when asked for Crawler type.
|
||||
|
||||
The example code after PlayWrightCrawler instantiation is similar to example describing the use of Playwright Crawler. The main difference is that in this example Camoufox will be used as the browser through BrowserPool.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightCrawlerExampleWithCamoufox}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
id: playwright-crawler-with-fingerprint-generator
|
||||
title: Playwright crawler with fingerprint generator
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_with_fingerprint_generator.py';
|
||||
|
||||
This example demonstrates how to use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> together with <ApiLink to="class/FingerprintGenerator">`FingerprintGenerator`</ApiLink> that will populate several browser attributes to mimic real browser fingerprint. To read more about fingerprints please see: https://docs.apify.com/academy/anti-scraping/techniques/fingerprinting.
|
||||
|
||||
You can implement your own fingerprint generator or use <ApiLink to="class/BrowserforgeFingerprintGenerator">`DefaultFingerprintGenerator`</ApiLink>. To use the generator initialize it with the desired fingerprint options. The generator will try to create fingerprint based on those options. Unspecified options will be automatically selected by the generator from the set of reasonable values. If some option is important for you, do not rely on the default and explicitly define it.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{PlaywrightCrawlerExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
id: respect-robots-txt-file
|
||||
title: Respect robots.txt file
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import RespectRobotsTxt from '!!raw-loader!roa-loader!./code_examples/respect_robots_txt_file.py';
|
||||
import OnSkippedRequest from '!!raw-loader!roa-loader!./code_examples/respect_robots_on_skipped_request.py';
|
||||
|
||||
This example demonstrates how to configure your crawler to respect the rules established by websites for crawlers as described in the [robots.txt](https://www.robotstxt.org/robotstxt.html) file.
|
||||
|
||||
To configure `Crawlee` to follow the `robots.txt` file, set the parameter `respect_robots_txt_file=True` in <ApiLink to="class/BasicCrawlerOptions">`BasicCrawlerOptions`</ApiLink>. In this case, `Crawlee` will skip any URLs forbidden in the website's robots.txt file.
|
||||
|
||||
As an example, let's look at the website `https://news.ycombinator.com/` and its corresponding [robots.txt](https://news.ycombinator.com/robots.txt) file. Since the file has a rule `Disallow: /login`, the URL `https://news.ycombinator.com/login` will be automatically skipped.
|
||||
|
||||
The code below demonstrates this behavior using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>:
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{RespectRobotsTxt}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
## Handle with `on_skipped_request`
|
||||
|
||||
If you want to process URLs skipped according to the `robots.txt` rules, for example for further analysis, you should use the `on_skipped_request` handler from <ApiLink to="class/BasicCrawler#on_skipped_request">`BasicCrawler`</ApiLink>.
|
||||
|
||||
Let's update the code by adding the `on_skipped_request` handler:
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{OnSkippedRequest}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
id: resuming-paused-crawl
|
||||
title: Resuming a paused crawl
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import ResumeCrawl from '!!raw-loader!roa-loader!./code_examples/resuming_paused_crawl.py';
|
||||
|
||||
This example demonstrates how to resume crawling from its last state when running locally, if for some reason it was unexpectedly terminated.
|
||||
|
||||
If each run should continue crawling from the previous state, you can configure this using `purge_on_start` in <ApiLink to="class/Configuration">`Configuration`</ApiLink>.
|
||||
|
||||
Use the code below and perform 2 sequential runs. During the 1st run, stop the crawler by pressing `CTRL+C`, and the 2nd run will resume crawling from where it stopped.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{ResumeCrawl}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
Perform the 1st run, interrupting the crawler with `CTRL+C` after 2 links have been processed.
|
||||
|
||||

|
||||
|
||||
Now resume crawling after the pause to process the remaining 3 links.
|
||||
|
||||

|
||||
|
||||
Alternatively, use the environment variable `CRAWLEE_PURGE_ON_START=0` instead of using `configuration.purge_on_start = False`.
|
||||
|
||||
For example, when running code:
|
||||
|
||||
```bash
|
||||
CRAWLEE_PURGE_ON_START=0 python -m best_crawler
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
id: run-parallel-crawlers
|
||||
title: Run parallel crawlers
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import RunParallelCrawlersExample from '!!raw-loader!roa-loader!./code_examples/run_parallel_crawlers.py';
|
||||
|
||||
This example demonstrates how to run two parallel crawlers where one crawler processes links discovered by another crawler.
|
||||
|
||||
In some situations, you may need different approaches for scraping data from a website. For example, you might use <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> for navigating JavaScript-heavy pages and a faster, more lightweight <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> for processing static pages. One way to solve this is to use <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink>, see the [Adaptive Playwright crawler example](./adaptive-playwright-crawler) to learn more.
|
||||
|
||||
The code below demonstrates an alternative approach using two separate crawlers. Links are passed between crawlers via <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> aliases. The `keep_alive` option allows the Playwright crawler to run in the background and wait for incoming links without stopping when its queue is empty. You can also use different storage clients for each crawler without losing the ability to pass links between queues. Learn more about available storage clients in this [guide](/python/docs/guides/storage-clients).
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{RunParallelCrawlersExample}
|
||||
</RunnableCodeBlock>
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
id: using_browser_profile
|
||||
title: Using browser profile
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
|
||||
import CodeBlock from '@theme/CodeBlock';
|
||||
|
||||
import ChromeProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_chrome.py';
|
||||
import FirefoxProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_firefox.py';
|
||||
|
||||
This example demonstrates how to run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> using your local browser profile from [Chrome](https://www.google.com/intl/us/chrome/) or [Firefox](https://www.firefox.com/).
|
||||
|
||||
Using browser profiles allows you to leverage existing login sessions, saved passwords, bookmarks, and other personalized browser data during crawling. This can be particularly useful for testing scenarios or when you need to access content that requires authentication.
|
||||
|
||||
## Chrome browser
|
||||
|
||||
To run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> with your Chrome profile, you need to know the path to your profile files. You can find this information by entering `chrome://version/` as a URL in your Chrome browser. If you have multiple profiles, pay attention to the profile name - if you only have one profile, it's always `Default`.
|
||||
|
||||
:::warning Profile access limitation
|
||||
Due to [Chrome's security policies](https://developer.chrome.com/blog/remote-debugging-port), automation cannot use your main browsing profile directly. The example copies your profile to a temporary location as a workaround.
|
||||
:::
|
||||
|
||||
Make sure you don't have any running Chrome browser processes before running this code:
|
||||
|
||||
<CodeBlock className="language-python" language="python">
|
||||
{ChromeProfileExample}
|
||||
</CodeBlock>
|
||||
|
||||
## Firefox browser
|
||||
|
||||
To find the path to your Firefox profile, enter `about:profiles` as a URL in your Firefox browser. Unlike Chrome, you can use your standard profile path directly without copying it first.
|
||||
|
||||
Make sure you don't have any running Firefox browser processes before running this code:
|
||||
|
||||
<CodeBlock className="language-python" language="python">
|
||||
{FirefoxProfileExample}
|
||||
</CodeBlock>
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
id: using-sitemap-request-loader
|
||||
title: Using sitemap request loader
|
||||
---
|
||||
|
||||
import ApiLink from '@site/src/components/ApiLink';
|
||||
|
||||
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
||||
|
||||
import SitemapRequestLoaderExample from '!!raw-loader!roa-loader!./code_examples/using_sitemap_request_loader.py';
|
||||
|
||||
This example demonstrates how to use <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> to crawl websites that provide `sitemap.xml` files following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> processes sitemaps in a streaming fashion without loading them entirely into memory, making it suitable for large sitemaps.
|
||||
|
||||
The example shows how to use the `transform_request_function` parameter to configure request options based on URL patterns. This allows you to modify request properties such as labels and user data based on the source URL, enabling different handling logic for different websites or sections.
|
||||
|
||||
The following code example implements processing of sitemaps from two different domains (Apify and Crawlee), with different labels assigned to requests based on their host. The `create_transform_request` function maps each host to the corresponding request configuration, while the crawler uses different handlers based on the assigned labels.
|
||||
|
||||
<RunnableCodeBlock className="language-python" language="python">
|
||||
{SitemapRequestLoaderExample}
|
||||
</RunnableCodeBlock>
|
||||
|
||||
For more information about request loaders, see the [Request loaders guide](../guides/request-loaders).
|
||||
Reference in New Issue
Block a user