Python

LeetCode Archiver (3): Authentication Originally published in Chinese on 2019-01-11; this English edition preserves the original scope and technical context. Cookies and Sessions In order to obtain our own submission records, we must first log in. But we all know that HTTP is a stateless protocol, and each of its requests is independent. Whether it is a GET or POST request, it contains all the information for processing the current request, but it does not involve changes in status. Therefore, in order to maintain a persistent state on the stateless HTTP protocol, the concepts of Cookie and Session are introduced. Both are encrypted data stored in memory or hard disk to identify user-related information. ...

Create a crawler Originally published in Chinese on 2018-12-21; this English edition preserves the original scope and technical context. After creating a project, open it with PyCharm or any other IDE. Navigate to the project folder and use the command genspider to create a spider: cd scrapy_project scrapy genspider QuestionSetSpider leetcode.com Among them, QuestionSetSpider is the name of the crawler, and leetcode.com is the domain name of the website we intend to crawl. ...

LeetCode Archiver (1): Scrapy and Requests Originally published in Chinese on 2018-12-04; this English edition preserves the original scope and technical context. Introduction The official Scrapy documentation introduces Scrapy as follows: Scrapy is an application framework written to crawl website data and extract structured data. It can be used in a series of programs including data mining, information processing or storing historical data. It was originally designed for page scraping (more specifically, web scraping), but can also be used to obtain data returned by APIs (such as Amazon Associates Web Services) or general web crawlers. ...