Different websites have different anti-crawler strategies and different restrictions on crawlers. Generally, they can be divided into the following three categories:
1. Set not to return web pages or delay return time
The traditional anti-crawler method is not to return web pages, that is, the crawler sends the request to the corresponding website, and the website returns a 404 page, indicating that the server cannot provide information normally, or the server has no response; the website may also not return data for a long time, which means that the crawler is banned.
2. The returned web page is not the target web page
In addition to not returning pages, there are also some crawlers that return non-target pages, that is, the website returns false data, for example, when returning a blank page or crawling back multiple pages, the same page is returned. If your crawler runs smoothly, you will be happy to do other things. After searching for half an hour, you will find that the search results for each page are the same, that is, fake websites.
For example, the price page of Qunar.com, the price marked online is different from the html source code. For example, the air ticket price marked online is 530 yuan, and the ticket price in the html source code is 538 yuan. In addition to Qunar.com, Maoyan Movies and Douyu Live also adopted this method, and the numbers crawled down are different from the real numbers.
3. Increase the difficulty of access
The website will also prevent crawlers by increasing the difficulty of obtaining data. Generally speaking, you can see the data by logging in and setting a verification code. In order to limit crawlers, the website may require you to log in and enter the verification code to access, regardless of whether you are a real user. For example, in order to limit automatic ticket grabbing, 12306 adopts a strict verification code function, requiring users to correctly select from 8 pictures.
These three situations are common in the crawler world. Crawlers need to formulate different anti-crawling strategies according to different actual situations in order to work smoothly.
Related Recommendations
- Guidelines and recommended resources for obtaining foreign proxy server IP
- Overseas dynamic IP technology frontiers: Integrating innovation and guiding the future
- Lemon8 tops the US download list: revealing the operation of the overseas version of "Little Red Book"
- Amazon multi-site operation: Proxy IP enables flexible switching of IP addresses
- How to utilize proxy IP in the emulator process?
- How to use proxy IP? How to protect information security through IP proxy?
- Shrimp skin multi-account operation: How to use proxy IP to avoid associated titles?
- Do you need native IP to do cross-border e-commerce?
- How to automatically obtain an ip address from an agent?
- How to prevent overseas social media matrix connections through IP proxies?
