Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-17-2023 09:32 PM
I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0
I've set PySpark commands as follows
path = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd"
schema = "some schema"
df = spark.read.option("compression", "zstd").json(path, schema)
df.createOrReplaceTempView("TestTable")then hit sql
%sql
select * from TestTable limit 100but failed with the following error.
Error in SQL statement: IllegalArgumentException: Codec [zstd] is not available. Known codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.Is there anyway to read `zstd` files there?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-17-2023 10:44 PM
The available compression types are format dependent.
For json, zstd is not (yet) available, whereas for parquet it is.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2023 12:36 AM
Thanks for clarifying!