Purpose: Use Scrapy framework to crawl the cover image of Huya anchor live broadcast

Scrapy (install the Scrapy framework through pip pip install Scrapy) and Python3.x installation tutorial can find the tutorial to install by yourself, here by default it has been successfully configured in the Windows environment.

1. Create a new project (scrapy startproject)

Before starting crawling, a new Scrapy project must be created. Win+R open the cmd command window and run the following command:

cd desktop
scrapy startproject huya

At this point, the huya project folder has been created on the desktop, and the files in the folder are explained as follows:
scrapy.cfg: project configuration file
huya/: The Python module of the project, the code will be quoted from here
huya/items.py: the target file of the project
huya/pipelines.py: the pipeline file of the project
huya/settings.py: project settings file
huya/spiders/: Store crawler code directory
Then, enter the huya/spiders/ directory, and enter the cmd command as follows:

cd huya
cd huya
cd spiders

Enter the command in the current directory, the file huyaspider.py will be created in the huya/spiders/ directory, and the crawling domain range will be specified: huya.com

scrapy genspider huyaspider "huya.com"

2. Code section

1）items.py

Open the file items.py in the huya/huya/ directory, the code is as follows:

# -*- coding: utf-8 -*-
import scrapy

class HuyaItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # Room theme
    nickname = scrapy.Field()
    # Link
    imagelink = scrapy.Field()
    # Storage path
    imagePath = scrapy.Field()

2）huyaspider.py

Open the file huyaspider.py in the huya/huya/spiders/ directory and modify the code as follows:

# -*- coding: utf-8 -*-
import scrapy
from huya.items import HuyaItem

class HuyaspiderSpider(scrapy.Spider):
    name = 'huyaspider'
    allowed_domains = ['huya.com']
    start_urls = ['https://www.huya.com/g/2168']

    def parse(self, response):
        # Match all root node list collections through scrapy's own xpath
        image_list = response.xpath('//div[@class="box-bd"]/ul/li')


        for img_each in image_list:
            huyaItem=HuyaItem()
            huyaItem["nickname"] = img_each.xpath("./a/img[@class='pic']/@title").extract()[0]
            huyaItem["imagelink"] = img_each.xpath("./a/img[@class='pic']/@data-original").extract()[0]
            yield huyaItem

3）pipelines.py

Open the file pipelines.py in the huya/huya/ directory and modify the code as follows:

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import scrapy
from scrapy.utils.project import get_project_settings
from scrapy.pipelines.images import ImagesPipeline
import os

class HuyaPipeline(ImagesPipeline):
    # def process_item(self, item, spider):
    #     return item
    # Get the variable value set in the settings file
    IMAGES_STORE = get_project_settings().get("IMAGES_STORE")
    headers = get_project_settings().get("DEFAULT_REQUEST_HEADERS")

    def get_media_requests(self, item, info):
        image_url = item["imagelink"]
        # headers is the request header is mainly anti-crawler
        yield scrapy.Request(image_url, headers=self.headers)

    def item_completed(self, result, item, info):
        image_path = [x["path"] for ok, x in result if ok]
        # # Create a directory if the directory does not exist
        if os.path.exists(self.IMAGES_STORE) == False:
            os.mkdir(self.image_path)
        os.rename(self.IMAGES_STORE + "/" + image_path[0], self.IMAGES_STORE + "/" + item["nickname"] + ".jpg")
        item["imagePath"] = self.IMAGES_STORE + "/" + item["nickname"]
        return item

3）setting.py

Open the file setting.py in the huya/huya/ directory,Add tocode show as below:

IMAGES_STORE = "C:/Users/**Username**/Desktop/huya/Images"

Open the file setting.py in the huya/huya/ directory,modifycode show as below:

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
   'User-Agent' : 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;',
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   #'Accept-Language': 'en',
}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'huya.pipelines.HuyaPipeline': 300,
}

3. Crawl data

cmd command window:

scrapy crawl huyaspider

Start pycharm, create a start_huya.py file in the huya/ directory, and add the code as follows:

#!/usr/bin/env python
# -*- coding:utf-8 -*-

from scrapy import cmdline
cmdline.execute("scrapy crawl huyaspider".split())

4. Results