Convert pkl file

[Innov]

I am looking to convert pkl file to parquet format. I am reading this data from Azure Data Lake and convert and create parquet file outputs to store them in my hive metastore.  

[Kaniz_Fatma]

Hi @Innov, To convert a Pickle (pkl) file to Parquet format and store it in your Azure data lake, you can follow these steps:

1. Azure Data Factory:

   • If you’re using Azure Data Factory, it supports the Parquet format for various connectors, including Azure Data Lake Storage Gen1 and Gen2.
   • You can create a dataset with the necessary properties for the Parquet format¹. Ensure that you have the required 64-bit JRE 8 (Java Runtime Environment) or OpenJDK installed on yo...¹.
   • If you encounter an “Out of Memory” error during copy, adjust the min/max heap size for JVM by setti...¹.

2. Databricks:

   • In Databricks, you can save a DataFrame as a Parquet file in Azure Data Lake. For example:

     save_location = "/mnt/adls/covid/base/Covid19_Cases" + year
     parquet_location = save_location + "temp.folder"
     file_location = save_location + "export.parquet"
     
     df.repartition(1).write.parquet(path=parquet_location, mode="append", header="true")
     
     file = dbutils.fs.ls(parquet_location)[-1].path
     dbutils.fs.cp(file, file_location)
     dbutils.fs.rm(parquet_location, recurse=True)
     

   • Adjust the save_location, parquet_location, and file_location according to your use case².

3. Azure Data Lake Analytics (ADLA):

   • ADLA is a serverless PaaS service in Azure that can process files of any format, including Parquet, ...³.
   • You can use ADLA to transform and prepare large amounts of data stored in Azure Data Lake Store or Azure Blob Storage.

4. Azure Databricks Delta Lake:

   • If you want to migrate from Parquet to Delta Lake, you can use a simple Spark read and write operation. For example:

     spark.read.format("parquet").load(file_path).write.mode("append").saveAsTable(table_name)
     

   • This approach allows you to work with Delta Lake features⁴.

Remember to adjust the code snippets according to your specific requirements and environment. Happy data processing! 🚀🔍