How to Get JSON Data from URL in Python ?

As a Data Scientist and AI programmer, you do most of the work on the JSON data. You usually fetch the JSON data from a particular URL and visualize it. But traversing into JSON data is always a challenging task for beginners. In this intuition, you will know how to get JSON data from URL in python. You will learn the following things.

How to read JSON data from the URL?
Convert JSON string to a dictionary.
Traverse in the JSON response

What is JSON?

JSON stands for JavaScript Object Notation and is widely used as a data interchange format. In fact, If you build a particular JSON restful API, then you can integrate it anywhere like Mobile Apps and Web Applications e.t.c.

How to read JSON data from the URL?

Reading the JSON data from the URL requires urllib request package. The package urllib is a python module with inbuilt methods for opening and retrieving XML, HTML, JSON e.t.c.
Let’s define the method getResponse(url) for retrieving the HTML or JSON from a particular URL. Inside the parameter, we are passing the URL of the JSON response. The following is the full code for the getResponse() function.

def getResponse(url):
    operUrl = urllib.request.urlopen(url)
    if(operUrl.getcode()==200):
       data = operUrl.read()
    else:
       print("Error receiving data", operUrl.getcode())
    return data

First of all, you will open the URL using the urllib.request.urlopen() method. If the address is correct you will get 200 response otherwise 404 response message. The 200 response tells the website is OK to crawl and the 404 error says the webpage does not exist. Check for it using the getcode() method. If it is 200 then read the entire HTML string or JSON as a string else print the error message with the code.

How to convert JSON String to Dictionary?

The above function will return the entire HTML or JSON as a string. To use it as an object in Python you have to first convert JSON into a dictionary. Python has a package JSON that handles this process. Let’s import JSON and add some lines of code in the above method. json.loads() method parse the entire JSON string and returns the JSON object. It completes the function for getting JSON response from the URL.

def getResponse(url):
    operUrl = urllib.request.urlopen(url)
    if(operUrl.getcode()==200):
        data = operUrl.read()
        jsonData = json.loads(data)
    else:
        print("Error receiving data", operUrl.getcode())
    return jsonData

You can see that after reading JSON response I am loading all the JSON data in a jsonData variable and returning it.

How to traverse inside the JSON response?

The first format is the JSON data such that you can easily read and find to traverse inside the JSON. Go to the URL https://jsoneditoronline.org/ and paste the JSON response. You will know the idea of the JSON. Like in the case of our example. The URL gives a response on the Indian state name and its state id.

Steps

First, select the key you want to get values from the key.

Like in our example I select the first states key for traversing.

Traverse to all the JSON responses using for loop.

Select the value you want to get using the key name. For example, I want to get the Indian State name then i will use jsonData[“state][“state_name”]. It means first go to the state key and then the state_name. In the same way for state id jsonData[“state”][“state_id”].

Let’s add some more lines of code in the main function.

def main():

    urlData = "http://vocab.nic.in/rest.php/states/json"
    jsonData = getResponse(urlData)
    # print the state id and state name corresponding
    for i in jsonData["states"]:
        print(f'State Name:  {i["state"]["state_name"]} , State ID : {i["state"]["state_id"]}')

Congrats, you have successfully received the JSON response from the URL in python interpreter. The output will be like this.

Full Code

import urllib.request
import json

def getResponse(url):
    operUrl = urllib.request.urlopen(url)
    if(operUrl.getcode()==200):
        data = operUrl.read()
        jsonData = json.loads(data)
    else:
        print("Error receiving data", operUrl.getcode())
    return jsonData

def main():

    urlData = "http://vocab.nic.in/rest.php/states/json"
    jsonData = getResponse(urlData)
    # print the state id and state name corresponding
    for i in jsonData["states"]:
        print(f'State Name:  {i["state"]["state_name"]} , State ID : {i["state"]["state_id"]}')

if __name__ == '__main__':
    main()

Get more details on the Ajax URL and rest API creation here .

Thanks

Data Science Learner Team