How to make http requests in python using proxies ?

(Comments)

There are a couple of ways to make HTTP/HTTPS requests in requests. First of all, it deserves keeping in mind that in httplib, you have to spend a great deal of time determining exactly how you would handle TLS or SSL decryption as well as the trust for HTTPS endpoints.

Requests is a Python library (Read here for the basic understanding of Python), It will permit you to send HTTP/1.1 requests using Python. With it, you can include content like headers, form data, multipart files, and parameters via simple Python libraries. It likewise permits you to access the feedback information of Python in the same way.

Install requests:

If you have pip installed on your system, you can install it by running the command-

pip install requests # for python2
pip3 install requests # for python3

Making a GET request:

>>> import requests
>>> response = requests.get('https://www.google.com')
>>> print(response)
<Response [200]>
>>> print(response.status_code)
200
>>> print(response.text)
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-IN"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
...
...
...
</script></div></body></html>

As the variable's name pointing out, response is response object containing some useful information response.status_code contains the status code and response.text contains the source code of request making above.

The output for above code is-

Parameters can also pass using a dictionary. For example-

>>> params = {
... 'q': 'apple',
... 'client': 'ubuntu',
... }
>>> r = requests.get('https://www.google.co.in/search', params=params)
>>> print(r.url)
https://www.google.co.in/search?q=apple&client=ubuntu
>>>

Same like GET using requests module POST, PUT, DELETE, HEAD and OPTIONS requests can also be made. For Detailed documentation visit here. That's all about getting started with requests module. Now coming back to the proxy section we write a script which makes the request using original proxy and proxy through which we want to make the request.

Making a GET request with the proxy-

For proxies without username and password - 

proxies = {
    "http" : 'http://ip:port',
    "https" : 'https://ip:port',
}

For proxies with username and password - 

proxies = {
    "http" : 'http://username:[email protected]:port',
    "https" : 'https://username:[email protected]:port',
}

below is the example-

import requests

url = 'https://api.ipify.org?format=json'

proxies = {
    "http" : 'http://153.149.169.215:3128',
    "https" : 'https://153.149.169.215:3128',
}

req = requests.get(url)
print('original proxy is: ', req.text)
request = requests.get(url, proxies=proxies)
print('using proxy is: ', request.text)

you will see the output-

('original proxy is: ', u'{"ip":"157.37.158.233"}')
('using proxy is: ', u'{"ip":"153.149.169.215"}')

For testing, you can found free proxies here. That’s all we’ve got to say If you have any query post in comments. Happy scraping :)

Comments

Join us for latest updates and blog posts

Recent Posts

Archive

2018

Authors

Categories

Tags

Feeds

RSS / Atom