All the pieces is accessible on the Internet by requests. In case you want info from an internet web page in your Python software, you want an internet request. On this article, we’ll dig into Python requests. We’ll have a look at how an internet request is structured and make a Python request. By the top, you’ll be capable of use the Python requests library, which makes the entire course of simpler.
An Introduction to HTTP Requests
To trade information on the Internet, we firstly want a communication protocol. The protocol used once we browse the Internet is the Hypertext Switch Protocol, or HTTP. HTTP makes use of TCP as a transport protocol, as a result of it wants dependable transport, and solely TCP can assure that.
Let’s say there’s a useful resource we want — such an HTML web page, on an internet server positioned someplace on the planet. We need to entry this useful resource or, in different phrases, we need to have a look at that web page in our net browser. The very first thing we now have to do is make an HTTP request. HTTP is a consumer–server protocol, which signifies that the requests are initiated by the consumer.
After the server receives the requests, it processes them and returns an applicable response.
The server may reply in several methods. It’d ship the useful resource we requested, or reply with standing codes if one thing doesn’t go as anticipated.
In each communication protocol, the data must be in particular fields. That’s as a result of each the consumer and the server ought to know interpret the request or response. Within the subsequent sections, we’ll have a look at how an HTTP request and an HTTP response are constructed. We’ll additionally focus on the position of crucial fields.
The HTTP request
One of the vital essential design options of HTTP is that it’s human readable. Which means that, once we have a look at an HTTP request, we will simply learn all the pieces, even when there’s numerous complexity below the hood. One other function of HTTP is that it’s stateless. Which means that there’s no hyperlink between two requests served one after the opposite. The HTTP protocol doesn’t keep in mind something of the earlier request. This suggests that every request should comprise all the pieces that the server wants to hold out the request.
A sound HTTP request should comprise the next parts:
- an HTTP methodology — equivalent to
GETorPOST - the model of the HTTP protocol
- the trail of the useful resource to fetch
Then, we will additionally add some optionally available headers that specify further details about the sender or the message. One instance of a typical HTTP request header is the Person-Agent or the pure language the consumer prefers. Each of these optionally available headers give details about the consumer that’s making the request.
That is an instance of an HTTP message, and we will clearly perceive all of the fields specified:
~~~http
GET / HTTP/1.1
Host: www.google.com
Settle for-Language: en-GB,en;q=0.5
~~~
The primary line specifies the request kind and the model of the HTTP protocol. Then we specify the Host and the language accepted by the consumer that’s sending the request. Normally, the messages are for much longer, however this provides a touch of what they seem like.
The HTTP response
Now that we now have an thought of what an HTTP request appears like, we will go on and see the HTTP response.
An HTTP response normally accommodates the next parts:
- the model of the HTTP protocol
- a standing code, with a descriptive short-message
- a listing of HTTP headers
- a message physique containing the requested useful resource
Now that we’ve launched the fundamental parts you want, it’s value making a abstract earlier than taking the following step. It ought to be clear by now that, every time a consumer desires to speak with an HTTP server, it should create and ship an HTTP request. Then, when the server receives it, it creates and sends an HTTP response.
We’re lastly able to introduce the Python requests library.
The Python requests Library
The Python requests library lets you ship Python HTTP requests — from primary to sophisticated ones. The Python requests library abstracts the complexities of creating advanced Python requests, offering an easy-to-use interface. Within the subsequent sections, we’ll see create straightforward Python requests and interpret the response. We’ll additionally see a number of the options offered by the Python requests library.
Putting in Python requests
First, we have to set up the Python requests library. Let’s set up it utilizing pip:
$ pip set up requests
As soon as the Python requests library is put in accurately, we will begin utilizing it.
Our first GET request with Python requests
The very first thing we now have to do is to create a Python file. On this instance, we name it net.py. Inside this supply file, insert this code:
import requests
URL = "https://www.google.com"
resp = requests.get(URL)
print(resp)
This program makes a GET request for Google. If we run this program, we’ll in all probability get this output:
$ python net.py
<Response [200]>
So, what does this imply?
We talked concerning the standing code earlier. This output is telling us that our request has been obtained, understood and processed efficiently. There are different codes as nicely, and we will record a couple of of the commonest:
-
301 Moved Completely. This can be a redirection message. The URL of the useful resource we had been on the lookout for has been moved. The brand new URL comes with the response. -
401 Unauthorized. This means a consumer error response. On this case, the server is telling us that we should authenticate earlier than continuing with the request. -
404 Not discovered. This means a consumer error response too. Specifically, because of this the server can’t discover the useful resource we had been on the lookout for.
What if we need to conditionally verify the standing, and supply completely different actions based mostly on the standing code? Nicely, we will simply do that:
import requests
URL = "https://www.google.com/blah"
resp = requests.get(URL)
if resp.status_code == 200:
print("Okay, all good!")
elif resp.status_code == 301:
print("Ops, the useful resource has been moved!")
elif resp.status_code == 404:
print("Oh no, the useful resource wasn't discovered!")
else:
print(resp.status_code)
If we run the script now, we’ll get one thing completely different. Have a try to see what we get. 😉
If we additionally need the descriptive brief message that comes with every standing code, we will use resp.motive. Within the case of a 200 standing code, we’ll merely get OK.
Inspecting the response of the Python request
At this level, we all know make a primary Python request. After the request, we wish the response, proper?
Within the earlier part, we noticed get the standing code of the response. Now, we need to learn the physique of the response, which is the precise useful resource we requested. To do that, we have to use resp.content material. Let’s say that we’re on the lookout for the Google house web page.
That is what we get once we run the script:
b'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content material="textual content/html; [...]
I’ve added [...] above as a result of the useful resource we get — which is a textual content/html doc — is just too lengthy to be printed. By how a lot? We are able to use len(resp.content material) to get this info. Within the case above, it was 13931 bytes — undoubtedly an excessive amount of to be printed right here!
Making use of APIs
One of many explanation why the Python requests library grew to become so common is as a result of it makes interacting with APIs very straightforward. For this instance, we’ll use a easy API for predicting an individual’s age, given their identify. This API is named Agify.
That is the code for the instance:
import requests
import json
URL = "https://api.agify.io/?identify=Marcus"
resp = requests.get(URL)
if resp.status_code == 200:
encoded = resp.json()
print(encoded['age'])
else:
print(resp.status_code)
On this case, we need to know the age of an individual whose identify is Marcus. As soon as we now have the response, if the standing code is 200, we interpret the end in JSON utilizing resp.json(). At this level, we now have a dictionary-like object, and we will print the estimated age.
The estimated age of Marcus is 41 years outdated.
HTTP headers present further info to each events of an HTTP dialog. Within the following instance, we’ll see how we will change the headers of an HTTP GET request. Specifically, we’ll change the Person-Agent and the Settle for-Language headers. The Person-Agent tells the server some details about the applying, the working system and the seller of the requesting agent. The Settle for-Language header communicates which languages the consumer is ready to perceive.
That is our easy code snippet:
import requests
URL = "https://www.google.com"
custom_headers = {'Settle for-Language': 'fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5', 'Person-Agent': 'Mozilla/5.0 (Linux; Android 12; SM-S906N Construct/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Model/4.0 Chrome/80.0.3987.119 Cellular Safari/537.36'}
resp = requests.get(URL, headers=custom_headers)
if resp.status_code == 200:
print(resp.content material[:100])
else:
print(resp.status_code)
If all the pieces goes proper, it’s best to get one thing like this:
$ <!doctype html><html lang="fr"><head><meta charset="UTF-8"><meta content material="width=device-width,mini [...]
On this instance, we’ve modified the Person-Agent, pretending that our request comes from Mozilla Firefox. We’re additionally saying that our working system is Android 12 and that our machine is a Samsung Galaxy S22.
Since we’ve printed the primary 100 characters of the response above, we will see that the HTML web page we’ve obtained is in French.
Conclusion
On this article, we talked concerning the HTTP protocol, with a quick theoretical introduction. Then we regarded on the Python requests library. We noticed write primary Python HTTP requests and customise them based on our wants.
I hope you’ll discover this library and this text helpful on your initiatives.
Associated studying:
