Databricks Community

Caching is extremely useful than checkpointing when you have lot of available memory to store your RDD or Dataframes if they are massive.Caching will maintain the result of your transformations so that those transformations will not have to be recomp...

1. We want a venue in which we can rapidly iterate and make new releases. The overhead of making a release as a separate project is minuscule (in the order of minutes). A release on Spark takes a lot longer (in the order of days)2. Koalas takes a dif...

Different projects have different focuses. Spark is already deployed in virtually every organization, and often is the primary interface to the massive amount of data stored in data lakes. Koalas was inspired by Dask, and aims to make the transition ...

Short Answer:YesDetailed Answer:User sessions automatically timeout after six hours of idle time. This timeout is not configurable. User sessions are terminated if the user is removed from the workspace. To trigger session end for users who were remo...

for any other non-private previews, they can check out admin console --> advanced tab. there are tons of toggles there to enable/disable features. if it’s not there, there usually isn’t an easy (or direct) way of disabling

you can go to spark UI and Kill it

This will be coming with Unity. There will be a third-level namespace. The default HM will be the default database in unity most likely.

Is it really %%bash or may be you are referring to %sh or %fs or dbutils command

Scala Use JVM to run its code, Scala cannot run different applications at a time with complete isolation of each task inside single jvm , that is the reason Scala doesn't support high concurrency cluster, I don't think it is on road map

We can read from Solr as Dataframe and SolrRDD as well (each RDD is a SolrDocument)more details available here

That’s only available at Premium and Enterprise SKUs in AWS.See the "Enterprise Security" section here:https://databricks.com/product/aws-pricing

On AWS: Enable bucket versioningOn Azure: Enable Soft DeletesOf course this needs to be done before you actually start writing the data.

Please raise a feature request via ideas portal for XML support in autoloader As a workaround, you could look at reading this with wholeTextFiles (which loads the data into a PairRDD with one record per input file) and parsing it with from_xml from ...

Here is the link to the configuration properties https://docs.databricks.com/dev-tools/databricks-connect.html#step-2-configure-connection-properties