JSON to parquet : How to perform in Python with example ?

JSON to parquet conversion is possible in multiple ways but I prefer via dataframe. Firstly convert JSON to dataframe and then to parquet file. In this article, we will explore the complete same process with an easy example.

JSON to parquet ( Conversion ) –

Let’s break this into steps.

Step 1: Prerequisite JSON object creation –

Here is the code for dummy json creation which we will use for converting into parquet.

record = '''
          {
              "0":{
                  "Identity": "Sam",
                  "Age": "19"
              },
              "1":{
                  "Identity": "Tom",
                  "Age": "14"
              },
              "2":{
                  "Identity": "Mac",
                  "Age":"11"
              }
          }
    '''

Step 2: Converting JSON to dataframe –

We will first import pandas framework and then load the json. After it, We will convert the same into pandas dataframe.

import pandas as pd
record = '''
          {
              "0":{
                  "Identity": "Sam",
                  "Age": "19"
              },
              "1":{
                  "Identity": "Tom",
                  "Age": "14"
              },
              "2":{
                  "Identity": "Mac",
                  "Age":"11"
              }
          }
    '''

df = pd.read_json(record, orient ='index')
print(df)

This read_json() function from Pandas helps convert JSON to pandas dataframe.

Step 3 : Dataframe to parquet file –

This is the last step, Here we will create parquet file from dataframe. We can use to_parquet() function for converting dataframe to parquet file. Here is the code for the same.

df.to_parquet("out.parquet")

When we integrate this piece of code with above master code. We get the parquet file.

Limitations –

All the JSON does not follow the structure which we can convert to dataframe. Hence this approach will only applicable with JSON format where is convertible to data frame.

JSON to Parquet in Pyspark –

Just like pandas, we can first create Pyspark Dataframe using JSON. IN order to do that here is the code-

df = spark.read.json("sample.json")

Once we have pyspark dataframe inplace, we can convert the pyspark dataframe to parquet using below way.

pyspark_df.write.parquet("data.parquet")

Conclusion –

Parquet file is a more popular file format for a table-like data structure. Also, it offers fast data processing performance than CSV file format. In the same way, Parquet file format contains the big volume of data than the CSV file format. I hope this article must help our readers, please feel free to put any concerns related to this topic. Same you can either comment or write back to us via email etc. Please subscribe to us for more related topics.

Thanks

Data Science Learner Team