Databricks

Databricks pool concept may help reduce cluster start up time. more details are here

If you're looking for python packages, you can use standard conda list commands (see example below):%conda list

There shouldn't be. Generally speaking, models will be serialized according to their 'native' format for well-known libraries like Tensorflow, xgboost, sklearn, etc. Custom model will be saved with pickle. The files exist on distributed storage as ar...

The Model Registry is mostly a workflow tool. It helps 'gate' the process, so that (for example) only authorized users can set a model to be the newest Production version of a model - that's not something just anyone should be able to do!The Registry...

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operationsZORDER BY -> Colocate column information in the sam...

In this scenario, the best option would be to have a single readStream reading a source delta table. Since checkpoint logs are controlled when writing to delta tables you would be able to maintain separate logs for each of your writeStreams. I would...

In this case, you are just writing a file to the working directory of the driver process. It'll be under something like /home/[user] on the local file system. Anything you write to "/dbfs/..." however goes to distributed storage, even though it looks...

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones ...

You could set up dnsmasq to configure  routing between your Databricks workspace and your on-premise network. More details here

Use the Databricks CLI: https://docs.databricks.com/dev-tools/cli/index.htmldatabricks fs cp dbfs:/remote/path /local/path

Databricks allows network customizations / hardening from a security point of view to reduce risks like Data exfiltration. For more detailsData Exfiltration Protection With Databricks on AWSData Exfiltration Protection with Azure Databricks