Rotating Proxies with Requests: Handy Snippets

Web scraping is fun—until you get blocked. But don’t worry! That’s where rotating proxies come in. They help you stay online and scrape like a champ.

In this article, we’ll talk about using rotating proxies with the Python requests library. It’s easier than you might think. And we’ll give you some handy snippets to get started fast.

🤔 Why Rotate Proxies?

When you send too many requests from one IP address, websites get suspicious. They might:

• Block your IP
• Show CAPTCHA
• Feed you fake data

Rotating proxies swap your IP with each request. This makes it look like the traffic is coming from many users instead of just one. Smart, right?

🛠️ What You Need

Before you begin, make sure you have:

• Python installed
• The requests library
• A list of proxies (free or paid)

You can install requests like this:

pip install requests

🚀 Rotating Proxies the Easy Way

Here’s a basic way to rotate proxies using a list.

import requests
import random

proxies = [
    'http://12.34.56.78:8080',
    'http://23.45.67.89:3128',
    'http://34.56.78.90:8000'
]

urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
]

headers = {
    'User-Agent': 'Mozilla/5.0'
}

for url in urls:
    proxy = random.choice(proxies)
    print(f'Using proxy: {proxy}')
    try:
        response = requests.get(url, proxies={'http': proxy, 'https': proxy}, headers=headers, timeout=5)
        print(f'Status code: {response.status_code}')
    except Exception as e:
        print(f'Request failed: {e}')

What’s happening here? We loop through URLs and pick a new proxy every time. This keeps your IP looking fresh and less spammy.

👮 Avoid Getting Blocked

Just rotating IPs isn’t always enough. Here are a few extra tricks:

• Use headers: Always include a realistic User-Agent.
• Obey robots.txt: Be nice to the sites you scrape.
• Slow it down: Add delays between requests to look human.

import time

time.sleep(random.uniform(1, 3))  # Sleep between 1 and 3 seconds

Slowing it down keeps you from setting off alarms. It’s like putting on a fake mustache—only slower.

🔁 Using a Proxy Rotation Service

If you want to level up, try a proxy service like:

• ScraperAPI
• Bright Data
• Smartproxy

These services manage the rotating for you. Just send a request, and voilà!

response = requests.get(
    'https://example.com',
    proxies={'http': 'http://user:pass@proxyservice.com:8000',
             'https': 'http://user:pass@proxyservice.com:8000'}
)

🧪 Tips for Testing

Want to see if your proxy is working? Hit this IP-checker:

response = requests.get('https://httpbin.org/ip', proxies={'http': proxy, 'https': proxy})
print(response.json())

This will show the IP for your current session. If it’s different each time—success!

🎯 Final Thoughts

Rotating proxies can be your secret weapon for reliable scraping. With a few lines of code, you avoid bans, get better data, and have more fun.

Remember: Always respect websites and follow their rules.

Happy scraping!