Databricks Community

A bloom filter index is a space-efficient data structure that enables data skipping on chosen columns, particularly for fields containing arbitrary text. The Bloom filter operates by either stating that data is definitively not in the file, or that i...

You'd need to open connections to Databricks web applicationDatabricks secure cluster connectivity (SCC) relayAWS S3 global URLAWS S3 regional URLAWS STS global URLAWS STS regional URLAWS Kinesis regional URLTable metastore RDS regional URL (by data ...

This primarily refers to the fact that notebooks can be shared to the whole org, to groups, to users, and can be limited to read/write/execute. You could argue that MLflow is also a form of collaboration, where multiple users can share an experiment ...

Use Databricks runtime 8.3 ML or later.

Does this help? "No Public IPs": https://docs.microsoft.com/en-us/azure/databricks/security/secure-cluster-connectivity

Standard tiers are allowed to have 1000 saved jobs. Premium tiers have a higher limit at 1500. Some clouds have an enterprise tier which has a saved job limit of 2000. A workspace is limited to 1000 concurrent job runs. A 429 Too Many Requests respon...

By default, the state data (streaming aggregation query) is maintained in the JVM memory of the executors and large number of state objects could put memory pressure on the JVM causing high GC pauses. If you have stateful operations in your streamin...

DBFS is the "Databricks File System", but really it's just a shim / wrapper on top of distributed storage, that makes files in S3 or ADLS look like local files under the path /dbfs/... This can be really useful when working with libraries that do not...

Using the same instance type is a fine default. If you know that you need very large workers, but little happens on the driver, maybe you can save money with a smaller driver. Conversely, you may know that some parts of your notebook involve a lot of...