Databricks Community

Witold · 07-23-2024

The Databricks runtime is shipped with two Java Runtimes: JRE 8 and JRE 17. While the first one is used by default, you can use the environment variable JNAME to specify the other JRE: JNAME: zulu17-ca-amd64.FWIW, AFAIK JNAME is available since DBR 1...

Witold

Now I got your point. No, you can't create Append Blob files directly in Volumes, as this is a native Azure functionality. A volume is basically just an abstraction over a native storage.You will still need to use libraries like azure-storage-blob wi...

Witold

Since Databricks uses Spark, you should be able to use e.g. Columnar EncryptionBesides, you can look into this and aes-specific functions.

Witold

It makes sense that you learn how the security model of Unity Catalog works, including long-term storage credentials, external locations, etc: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/#other-securable-objects

Witold

Multiple things.First very obvious thing: "5000000" - It's not surprising that you run OOM when loading such a huge amount of records.The DBR has a built-in MySQL driver, use it insteadUse fetchSize to control the number of records. Start e.g. with 1...

Witold

How do you read and write the records? Which cluster size do you use?

Databricks Community

User Stats

User Activity

Databricks runtime and Java Runtime

Re: Writing append blob files to unity catalog volum

Re: Parquet Encryption/Decryption in Databricks

Re: Writing append blob files to unity catalog volum

Re: Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls