156 lines
8.2 KiB
Plaintext
156 lines
8.2 KiB
Plaintext
---
|
|
id: scraping
|
|
title: Scraping
|
|
---
|
|
|
|
import ApiLink from '@site/src/components/ApiLink';
|
|
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
|
|
|
import ScrapingExample from '!!raw-loader!roa-loader!./code_examples/06_scraping.py';
|
|
|
|
In the [Real-world project](./real-world-project#choosing-the-data-you-need) chapter, you've created a list of the information you wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
|
|
|
|
- URL
|
|
- Manufacturer
|
|
- SKU
|
|
- Title
|
|
- Current price
|
|
- Stock available
|
|
|
|
|
|
|
|
## Scraping the URL and manufacturer
|
|
|
|
Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `context.request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
|
|
|
|
:::info url vs loaded url
|
|
|
|
You can use `request.loaded_url` as well. Remember the difference: `request.url` is what you enqueue, `request.loaded_url` is what gets processed (after possible redirects).
|
|
|
|
:::
|
|
|
|
By splitting the `request.url`, we can extract the manufacturer name directly from the URL. This is done by first splitting the URL to get the product identifier and then splitting that identifier to get the manufacturer name.
|
|
|
|
```python
|
|
# context.request.url:
|
|
# https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
|
|
|
|
# Split the URL and get the last part.
|
|
url_part = context.request.url.split('/').pop()
|
|
# url_part: sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
|
|
|
|
# Split the last part by '-' and get the first element.
|
|
manufacturer = url_part.split('-')[0]
|
|
# manufacturer: 'sennheiser'
|
|
```
|
|
|
|
:::tip Storing information
|
|
|
|
It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
|
|
|
|
:::
|
|
|
|
:::caution Adapt and extract
|
|
|
|
One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
|
|
|
|
:::
|
|
|
|
Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [Sony XBR-950G](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
|
|
|
|
## Scraping title
|
|
|
|
To scrape the product title from a webpage, you need to identify its location in the HTML structure. By using the element selector tool in your browser's DevTools, you can see that the title is within an `<h1>` tag, which is a common practice for important headers. This `<h1>` tag is enclosed in a `<div>` with the class product-meta. We can leverage this structure to create a combined selector `.product-meta h1`. This selector targets any `<h1>` element that is a child of an element with the class `product-meta`.
|
|
|
|
|
|
|
|
:::tip Verifying selectors with DevTools
|
|
|
|
Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of DevTools to open the search bar where you can quickly search for elements using their selectors. Always verify your scraping process and assumptions using the DevTools. It's faster than changing the crawler code all the time.
|
|
|
|
:::
|
|
|
|
To get the title, you need to locate it using Playwright with the `.product-meta h1` selector. This selector specifically targets the `<h1>` element you need. If multiple elements match, it will throw an error, which is beneficial as it prevents returning incorrect data silently. Ensuring the accuracy of your selectors is crucial for reliable data extraction.
|
|
|
|
```python
|
|
title = await context.page.locator('.product-meta h1').text_content()
|
|
```
|
|
|
|
## Scraping SKU
|
|
|
|
Using the DevTools, you can find that the product SKU is inside a `<span>` tag with the class `product-meta__sku-number`. Since there is no other `<span>` with that class on the page, you can safely use this selector to extract the SKU.
|
|
|
|
|
|
|
|
```python
|
|
# Find the SKU element using the selector and get its text content.
|
|
sku = await context.page.locator('span.product-meta__sku-number').text_content()
|
|
```
|
|
|
|
## Scraping current price
|
|
|
|
Using DevTools, you can find that the current price is within a `<span>` element tagged with the `price` class. However, it is nested alongside another `<span>` element with the `visually-hidden` class. To avoid extracting the wrong text, you can filter the elements to get the correct one using the `has_text` helper.
|
|
|
|
|
|
|
|
```python
|
|
# Locate the price element and filter out the visually hidden elements.
|
|
price_element = context.page.locator('span.price', has_text='$').first
|
|
|
|
# Extract the text content of the price element.
|
|
current_price_string = await price_element.text_content() or ''
|
|
# current_price_string: 'Sale price$1,398.00'
|
|
|
|
# Split the string by the '$' sign to get the numeric part.
|
|
raw_price = current_price_string.split('$')[1]
|
|
# raw_price: '1,398.00'
|
|
|
|
# Convert the raw price string to a float after removing commas.
|
|
price = float(raw_price.replace(',', ''))
|
|
# price: 1398.00
|
|
```
|
|
|
|
It might look a little complex at first glance, but let's walk through what you did. First, you locate the correct part of the `price` span by filtering for elements containing the `$` sign. This ensures that you get the actual price element. Once you have the right element, you extract its text content, which gives you a string similar to `Sale price$1,398.00`. To get the numeric value, you split this string by the `$` sign. Next, you remove any commas from the resulting numeric string and convert it to a float, allowing you to work with the price as a number. This process ensures that you accurately extract and convert the current price from the product page.
|
|
|
|
## Scraping stock availability
|
|
|
|
The final step is to scrape the stock availability information. There is a `<span>` with the class `product-form__inventory`, which contains the text `In stock` if the product is available. You can use the `has_text` helper to filter out the correct element.
|
|
|
|
```python
|
|
# Locate the element that contains the text 'In stock' and filter out other elements.
|
|
in_stock_element = context.page.locator(
|
|
selector='span.product-form__inventory',
|
|
has_text='In stock',
|
|
).first
|
|
|
|
# Check if the element exists by counting the matching elements.
|
|
in_stock = await in_stock_element.count() > 0
|
|
```
|
|
|
|
For this, all that matters is whether the element exists or not. You can use the `count()` method to check if any elements match the selector. If there are, it means the product is in stock.
|
|
|
|
## Trying it out
|
|
|
|
You have everything that is needed, so grab your newly created scraping logic, dump it into your original request handler and see the magic happen!
|
|
|
|
<RunnableCodeBlock className="language-python" language="python">
|
|
{ScrapingExample}
|
|
</RunnableCodeBlock>
|
|
|
|
When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
|
|
|
|
```json
|
|
{
|
|
"url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
|
|
"manufacturer": "sony",
|
|
"title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
|
|
"sku": "SON-692802-STR-DE",
|
|
"price": 698,
|
|
"in_stock": true
|
|
}
|
|
```
|
|
|
|
## Next steps
|
|
|
|
Next, you'll see how to save the data you scraped to the disk for further processing.
|