Site icon UnderConstructionPage

Rotating Proxies with Requests: Handy Snippets

Ethan Martinez
Server

Web scraping is fun—until you get blocked. But don’t worry! That’s where rotating proxies come in. They help you stay online and scrape like a champ.

In this article, we’ll talk about using rotating proxies with the Python requests library. It’s easier than you might think. And we’ll give you some handy snippets to get started fast.

🤔 Why Rotate Proxies?

When you send too many requests from one IP address, websites get suspicious. They might:

Rotating proxies swap your IP with each request. This makes it look like the traffic is coming from many users instead of just one. Smart, right?

Image not found in postmeta

🛠️ What You Need

Before you begin, make sure you have:

You can install requests like this:

pip install requests

🚀 Rotating Proxies the Easy Way

Here’s a basic way to rotate proxies using a list.


import requests
import random

proxies = [
    'http://12.34.56.78:8080',
    'http://23.45.67.89:3128',
    'http://34.56.78.90:8000'
]

urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
]

headers = {
    'User-Agent': 'Mozilla/5.0'
}

for url in urls:
    proxy = random.choice(proxies)
    print(f'Using proxy: {proxy}')
    try:
        response = requests.get(url, proxies={'http': proxy, 'https': proxy}, headers=headers, timeout=5)
        print(f'Status code: {response.status_code}')
    except Exception as e:
        print(f'Request failed: {e}')

What’s happening here? We loop through URLs and pick a new proxy every time. This keeps your IP looking fresh and less spammy.

👮 Avoid Getting Blocked

Just rotating IPs isn’t always enough. Here are a few extra tricks:


import time

time.sleep(random.uniform(1, 3))  # Sleep between 1 and 3 seconds

Slowing it down keeps you from setting off alarms. It’s like putting on a fake mustache—only slower.

🔁 Using a Proxy Rotation Service

If you want to level up, try a proxy service like:

These services manage the rotating for you. Just send a request, and voilà!


response = requests.get(
    'https://example.com',
    proxies={'http': 'http://user:pass@proxyservice.com:8000',
             'https': 'http://user:pass@proxyservice.com:8000'}
)
Image not found in postmeta

🧪 Tips for Testing

Want to see if your proxy is working? Hit this IP-checker:


response = requests.get('https://httpbin.org/ip', proxies={'http': proxy, 'https': proxy})
print(response.json())

This will show the IP for your current session. If it’s different each time—success!

🎯 Final Thoughts

Rotating proxies can be your secret weapon for reliable scraping. With a few lines of code, you avoid bans, get better data, and have more fun.

Remember: Always respect websites and follow their rules.

Happy scraping!

Exit mobile version