- 2141 Views
- 1 replies
- 0 kudos
- 2141 Views
- 1 replies
- 0 kudos
- 0 kudos
There shouldn't be. Generally speaking, models will be serialized according to their 'native' format for well-known libraries like Tensorflow, xgboost, sklearn, etc. Custom model will be saved with pickle. The files exist on distributed storage as ar...
- 0 kudos
- 1965 Views
- 1 replies
- 0 kudos
Resolved! What is the point of the model staging and promotion functions in MLflow?
Why not just directly deploy the model where you need it in production?
- 1965 Views
- 1 replies
- 0 kudos
- 0 kudos
The Model Registry is mostly a workflow tool. It helps 'gate' the process, so that (for example) only authorized users can set a model to be the newest Production version of a model - that's not something just anyone should be able to do!The Registry...
- 0 kudos
- 2933 Views
- 1 replies
- 0 kudos
Resolved! Should I use Z Ordering on my Delta table every time I run Optimize?
Wondering if it always makes sense or if there are some situations where you might only want to run optimize
- 2933 Views
- 1 replies
- 0 kudos
- 0 kudos
Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operationsZORDER BY -> Colocate column information in the sam...
- 0 kudos
- 2619 Views
- 1 replies
- 0 kudos
- 2619 Views
- 1 replies
- 0 kudos
- 0 kudos
In this scenario, the best option would be to have a single readStream reading a source delta table. Since checkpoint logs are controlled when writing to delta tables you would be able to maintain separate logs for each of your writeStreams. I would...
- 0 kudos
- 1299 Views
- 1 replies
- 0 kudos
Major changes in spark 3.0
What are the major changes released in spark 3.0
- 1299 Views
- 1 replies
- 0 kudos
- 0 kudos
Check out https://spark.apache.org/docs/latest/sql-migration-guide.html if you're looking for potentially breaking changes you need to be aware of, for any version.For a general overview of the new features, see https://databricks.com/blog/2020/06/18...
- 0 kudos
- 1736 Views
- 1 replies
- 0 kudos
How do I benefit from parallelisation when doing machine learning?
There are in principle four distinct ways of using parallelisation when doing machine learning. Any combination of these can speed up the whole pipeline significantly.1) Using spark distributed processing in feature engineering 2) When the data set...
- 1736 Views
- 1 replies
- 0 kudos
- 0 kudos
Good summary! yes those are the main strategies I can think of.
- 0 kudos
- 2535 Views
- 2 replies
- 0 kudos
- 2535 Views
- 2 replies
- 0 kudos
- 0 kudos
You do not have to cache anything to make it work. You would decide that based on whether you want to spend memory/storage to avoid recomputing the DataFrame, like when you may use it in multiple operations afterwards.
- 0 kudos
- 5417 Views
- 1 replies
- 0 kudos
Resolved! What's the difference between SparkML and Spark MLlib?
I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?
- 5417 Views
- 1 replies
- 0 kudos
- 0 kudos
They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones ...
- 0 kudos
- 5291 Views
- 1 replies
- 1 kudos
Resolved! How can I configure a custom DNS for my databricks workspace to talk to my on-premises systems
- 5291 Views
- 1 replies
- 1 kudos
- 1 kudos
You could set up dnsmasq to configure routing between your Databricks workspace and your on-premise network. More details here
- 1 kudos
- 7054 Views
- 1 replies
- 0 kudos
- 7054 Views
- 1 replies
- 0 kudos
- 0 kudos
Use the Databricks CLI: https://docs.databricks.com/dev-tools/cli/index.htmldatabricks fs cp dbfs:/remote/path /local/path
- 0 kudos
- 2209 Views
- 1 replies
- 0 kudos
- 2209 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks allows network customizations / hardening from a security point of view to reduce risks like Data exfiltration. For more detailsData Exfiltration Protection With Databricks on AWSData Exfiltration Protection with Azure Databricks
- 0 kudos
- 2002 Views
- 1 replies
- 0 kudos
Z ordering best practices
What are the best practices around Z ordering, Should be include as Manu column as Possible in Z order or lesser the better and why?
- 2002 Views
- 1 replies
- 0 kudos
- 0 kudos
With Z-order and Hilbert curves, the effectiveness of clustering decreases with each column added - so you'd want to zorder only the columns that you's actually use so that it's speed up your workloads.
- 0 kudos
- 896 Views
- 0 replies
- 0 kudos
- 896 Views
- 0 replies
- 0 kudos
- 1570 Views
- 1 replies
- 0 kudos
- 1570 Views
- 1 replies
- 0 kudos
- 0 kudos
coalesce avoids a full shuffle and could be used to decrease the number of partitionsrepartition results in a full shuffle and could be used to increase or decrease the number of partitions
- 0 kudos
- 3065 Views
- 1 replies
- 0 kudos
Repos branch control – how can we configure a job to run a specific branch?
For example, how can we ensure our jobs always run off the main/master branch?
- 3065 Views
- 1 replies
- 0 kudos
- 0 kudos
We recommend having a top level folder to run jobs against. Best practice detailed here: https://docs.databricks.com/repos.html#best-practices-for-integrating-repos-with-cicd-workflows
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 480 | |
| 349 | |
| 287 |
