- 459 Views
- 0 replies
- 0 kudos
I'd like to use Deep Learning on Spark within AWS EMR.Is there a best practice or 'recommended' DL framework to run on Spark? It looks like Databricks' spark-deep-learning has been replaced by Horovod—should this the first option to consider?
If th...
- 459 Views
- 0 replies
- 0 kudos
- 294 Views
- 0 replies
- 1 kudos
Advantage of using Photon EngineThe following summarizes the advantages of Photon:Supports SQL and equivalent DataFrame operations against Delta and Parquet tables.Expected to accelerate queries that process a significant amount of data (100GB+) and ...
- 294 Views
- 0 replies
- 1 kudos
- 1656 Views
- 1 replies
- 0 kudos
Wondering about best practices for how to handle collaboration between multiple ML practitioners working on a single experiment. Do we have to share the same notebook between people or is it possible to have individual notebooks going but still work ...
- 1656 Views
- 1 replies
- 0 kudos
Latest Reply
Yes, multiple users could work on individual notebooks and still use the same experiment via mlflow.set_experiment(). You could also assign different permission levels to experiments from a governance point of view
- 1410 Views
- 1 replies
- 0 kudos
The default location or MLflow artifacts is on dbfs, but I would like to save my models to an alternative location. Is this supported, and if it is how can I accomplish it?
- 1410 Views
- 1 replies
- 0 kudos
Latest Reply
You could mount an s3 bucket in the workspace and save your model using the mounts DBFS path For e.gmodelpath = "/dbfs/my-s3-bucket/model-%f-%f" % (alpha, l1_ratio)
mlflow.sklearn.save_model(lr, modelpath)
- 710 Views
- 1 replies
- 0 kudos
Is it possible to rollback changes made to a cluster? The problem I'm trying to solve is to recover from an accidental change made by a user on a cluster that affects interactive and job runs. Cluster policies help, but the policy still provides the ...
- 710 Views
- 1 replies
- 0 kudos
Latest Reply
You could look at automating cluster creation steps and implementing this with an infra-as-code solution like the databricks terraform provider which allows rollback
- 920 Views
- 0 replies
- 1 kudos
Do we have general guidance around how other customers manage Dev and Prod environments in Databricks? Is it recommended to have separate workspaces for them? What are the pros and cons of using the same workspace with folder or repo level isolation?
- 920 Views
- 0 replies
- 1 kudos
- 1451 Views
- 1 replies
- 0 kudos
I'm trying to run Delta Lake MergeMERGE INTO source
USING updates
ON source.d = updates.sessionId
WHEN MATCHED THEN UPDATE *
WHEN NOT MATCHED THEN INSERT *I'm getting an SQL errorParseException: mismatched input 'MERGE' expecting {'(', 'SELECT', 'FR...
- 1451 Views
- 1 replies
- 0 kudos
Latest Reply
The merge SQL support is added in Delta Lake 0.7.0. You also need to upgrade your Apache Spark to 3.0.0 and enable the integration with Apache Spark DataSourceV2 and C
- 428 Views
- 0 replies
- 0 kudos
There's a lot of different ML formats out there and I am confused about how they should be fitting together. How should I be thinking about MLflow and MLeap working together?
- 428 Views
- 0 replies
- 0 kudos
- 878 Views
- 1 replies
- 0 kudos
I am trying to set up a demo with a really simple spark ML model and i see this error repeated over and over in the logs in the serving UI:/databricks/chauffeur/model-runner/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestW...
- 878 Views
- 1 replies
- 0 kudos
Latest Reply
Not sure how the containers for each model version work on the endpoints, but looks like Model serving endpoints use a 7.x runtime. So those would be Spark 3.0, not Spark 3.1
- 1060 Views
- 1 replies
- 0 kudos
I can see an example on how to call the vacuum function for a Delta lake in python here. how to use the same in python %sql
VACUUM delta.`dbfs:/mnt/<myfolder>` DRY RUN
- 1060 Views
- 1 replies
- 0 kudos
Latest Reply
The dry run for non-SQL code is not yet available in Delta version 0.8. I see there is a bug that is opened with delta opensource in git . hope it get resolved soon