topic java.io.IOException: Invalid PKCS8 data error when reading data from Google Storage in Data Engineering

java.io.IOException: Invalid PKCS8 data error when reading data from Google Storage

364488 — Mon, 22 Jan 2024 17:56:30 GMT

Databricks workspace is hosted in AWS. Trying to access data in Google Cloud Platform.

I have followed the instructions here: https://docs.databricks.com/en/connect/storage/gcs.html

I get error: "java.io.IOException: Invalid PKCS8 data." when trying to do a dbutils.fs.ls() to view contents.

Looking online, it looks like there is an issue with the private key needing to be encoded as bytes, but I am not sure how I set this in the cluster spark config. Example solutions online are using notebook code to the encoding, but that's not an option in the spark config.

Re: java.io.IOException: Invalid PKCS8 data error when reading data from Google Storage

shan_chandra — Tue, 23 Jan 2024 00:05:08 GMT

@364488 - can you please share the spark configs added to the cluster?

Re: java.io.IOException: Invalid PKCS8 data error when reading data from Google Storage

Debayan — Tue, 23 Jan 2024 03:29:01 GMT

Hi, Could you also please share the whole error stack?