Python3 Selenium enables efficient configuration of Chrome proxy IP

When using Selenium for automated web testing or crawler development, you sometimes need to use Proxy IP to hide the real IP address or visit websites in specific regions. This article will introduce in detail how to use Python3 and Selenium library to efficiently configure proxy IP in Chrome browser.

1. Preparation work

1.1 Install necessary libraries

First, make sure you have the Selenium library and ChromeDriver installed. In addition, you also need a service that can obtain a proxy IP (such as a free public proxy or a paid proxy service).

pip install selenium

1.2 Download ChromeDriver

Download the ChromeDriver that matches your Chrome browser version from the ChromeDriver download page and extract it to your system path.

2. Configure Chrome proxy IP

2.1 Create Chrome Options

In Selenium, you can set the Chrome browser startup options through the ChromeOptions category. Here is sample code for how to create a ChromeOptions object and set the proxy IP:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Create ChromeOptions object
chrome_options = Options()

# Set the proxy IP (assuming the proxy IP is 123.123.123.123 and the connection port is 8080)
chrome_options.add_argument('--proxy-server=http://123.123.123.123:8080')

# If you need to bypass the proxy program to access certain addresses, you can add the following parameters (for example, do not proxy localhost)
chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])
chrome_options.add_argument('--disable-blink-features=AutomationControlled')

2.2 Start Chrome browser

# Create Chrome WebDriver instance
driver = webdriver.Chrome(options=chrome_options)

# Visit a website to test whether the proxy is working
driver.get('http://httpbin.org/ip')

# Print the currently accessed IP address (the proxy IP should be displayed)
print(driver.page_source)

# Close browser
driver.quit()

2.3 Dynamic proxy configuration (optional)

If you need to dynamically change the proxy IP (for example, obtain a new proxy IP from the proxy pool), you can achieve this through Selenium's Chrome DevTools Protocol (CDP). The following is a sample code that shows how to dynamically set the proxy IP:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
import json

# Create ChromeOptions object
chrome_options = Options()

# Disable Chrome's automated control prompts
chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])
chrome_options.add_argument('--disable-blink-features=AutomationControlled')

# Start the Chrome WebDriver instance (note: add_argument is not used to set the proxy here)
service = Service(executable_path='/path/to/chromedriver') # Replace with your chromedriver path
driver = webdriver.Chrome(service=service, options=chrome_options)

#Create a new proxy object (here we take SOCKS5 proxy as an example)
proxy = Proxy({
 'proxyType': ProxyType.MANUAL,
 'httpProxy': '127.0.0.1:1080', # SOCKS5 proxy address and port
 'sslProxy': '127.0.0.1:1080', # SOCKS5 proxy address and port (for HTTPS)
 'ftpProxy': '127.0.0.1:1080', # Optional: FTP proxy address and port
 'socksProxy': '127.0.0.1:1080', # SOCKS proxy address and port (usually the same as httpProxy and sslProxy)
 'socksVersion': 5, # SOCKS proxy version (usually 5)
 'noProxy': ['localhost', '127.0.0.1'] # List of addresses that do not pass the proxy
})

# Add the proxy object to Chrome options (note: CDP dynamic configuration is used here)
# You need to start the browser first, and then send the agent setting command through the execute_cdp_cmd method
driver.execute_cdp_cmd('Network.setUserAgentOverride', {
 "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
})

# Set proxy configuration
proxy_config = {
 'mode': 'fixed_servers',
 'server': {
 'scheme': 'socks5', # Proxy protocol (can be http, https, socks4, socks5, etc.)
 'host': '127.0.0.1', # Proxy server address (the address of the local proxy software is used here)
 'port': 1080 # Proxy server port
 }
}

# Convert proxy configuration to JSON string
proxy_config_json = json.dumps(proxy_config)

# Send proxy setting instructions through CDP
driver.execute_cdp_cmd('Network.enable', {})
driver.execute_cdp_cmd('Network.setProxyOverride', {'proxy': proxy_config_json})

# Visit a website to test whether the proxy is working
driver.get('http://httpbin.org/ip')

# Print the currently accessed IP address (the proxy IP should be displayed)
print(driver.page_source)

# Close browser
driver.quit()

Note: In the above code, we use CDP's Network.setProxyOverride command to dynamically change the set proxy IP. This method allows you to configure the proxy after the browser is launched without restarting the browser. In addition, we also use the Network.setUserAgentOverride command to set the user agent string (this is optional, but helps simulate the network behavior of real users).

3. Things to note

1. Validity of proxy IP: Make sure that the proxy IP you use is valid and has not been blocked by the target website.
2. Proxy IP replacement: If you need to change the proxy IP frequently, you can consider using a proxy pool or purchasing a paid proxy service.
3. Error handling: In actual applications, you should add error handling logic to handle abnormal situations such as proxy IP failure and network fluctuations.
4. Security: When using a proxy IP, please pay attention to protecting your sensitive information (such as passwords, API keys, etc.) and avoid transmitting sensitive data through the proxy.
   
   

Through the introduction and sample code of this article, you should be able to learn how to use the Selenium library in Python3 to set a proxy IP for the Chrome browser. Hope this information is helpful to you!