- 4160 Views
- 1 replies
- 1 kudos
- 4160 Views
- 1 replies
- 1 kudos
- 1 kudos
If you have configured your Structured Streaming query to use RocksDB as the state store, you can now get better visibility into the performance of RocksDB, with detailed metrics on get/put latencies, compaction latencies, cache hits, and so on. Thes...
- 1 kudos
- 997 Views
- 0 replies
- 1 kudos
docs.databricks.com
Advantage of using Photon EngineThe following summarizes the advantages of Photon:Supports SQL and equivalent DataFrame operations against Delta and Parquet tables.Expected to accelerate queries that process a significant amount of data (100GB+) and ...
- 997 Views
- 0 replies
- 1 kudos
- 3330 Views
- 1 replies
- 2 kudos
- 3330 Views
- 1 replies
- 2 kudos
- 2 kudos
check if your workspace has the IP access list feature enabled, call the get feature status API (GET /workspace-conf). Pass keys=enableIpAccessLists as arguments to the request.In the response, the enableIpAccessListsthe field specifies either true o...
- 2 kudos
- 3714 Views
- 1 replies
- 0 kudos
Can multiple users collaborate together on MLflow experiments?
Wondering about best practices for how to handle collaboration between multiple ML practitioners working on a single experiment. Do we have to share the same notebook between people or is it possible to have individual notebooks going but still work ...
- 3714 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, multiple users could work on individual notebooks and still use the same experiment via mlflow.set_experiment(). You could also assign different permission levels to experiments from a governance point of view
- 0 kudos
- 3449 Views
- 1 replies
- 0 kudos
Resolved! Can I save MLflow artifacts to locations other than the dbfs?
The default location or MLflow artifacts is on dbfs, but I would like to save my models to an alternative location. Is this supported, and if it is how can I accomplish it?
- 3449 Views
- 1 replies
- 0 kudos
- 0 kudos
You could mount an s3 bucket in the workspace and save your model using the mounts DBFS path For e.gmodelpath = "/dbfs/my-s3-bucket/model-%f-%f" % (alpha, l1_ratio) mlflow.sklearn.save_model(lr, modelpath)
- 0 kudos
- 2079 Views
- 1 replies
- 2 kudos
- 2079 Views
- 1 replies
- 2 kudos
- 2 kudos
Not yet, but stay-tuned it's being cooked in the kitchen
- 2 kudos
- 1556 Views
- 1 replies
- 0 kudos
- 1556 Views
- 1 replies
- 0 kudos
- 0 kudos
Data is stored in the control plane. Metadata (eg feature table descriptions, column types, etc) is stored in the control plane. The location where the Delta table is stored is determined by the database location. The customer could call CREATE DATA...
- 0 kudos
- 1888 Views
- 0 replies
- 1 kudos
Dev and Prod environments
Do we have general guidance around how other customers manage Dev and Prod environments in Databricks? Is it recommended to have separate workspaces for them? What are the pros and cons of using the same workspace with folder or repo level isolation?
- 1888 Views
- 0 replies
- 1 kudos
- 2576 Views
- 1 replies
- 0 kudos
Delta Lake MERGE INTO statement error
I'm trying to run Delta Lake MergeMERGE INTO source USING updates ON source.d = updates.sessionId WHEN MATCHED THEN UPDATE * WHEN NOT MATCHED THEN INSERT *I'm getting an SQL errorParseException: mismatched input 'MERGE' expecting {'(', 'SELECT', 'FR...
- 2576 Views
- 1 replies
- 0 kudos
- 0 kudos
The merge SQL support is added in Delta Lake 0.7.0. You also need to upgrade your Apache Spark to 3.0.0 and enable the integration with Apache Spark DataSourceV2 and C
- 0 kudos
- 1122 Views
- 0 replies
- 0 kudos
Should I be saving my SparkML models in MLflow using MLeap?
There's a lot of different ML formats out there and I am confused about how they should be fitting together. How should I be thinking about MLflow and MLeap working together?
- 1122 Views
- 0 replies
- 0 kudos
- 3445 Views
- 1 replies
- 0 kudos
Resolved! Setup a model serving REST endpoint?
I am trying to set up a demo with a really simple spark ML model and i see this error repeated over and over in the logs in the serving UI:/databricks/chauffeur/model-runner/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestW...
- 3445 Views
- 1 replies
- 0 kudos
- 0 kudos
Not sure how the containers for each model version work on the endpoints, but looks like Model serving endpoints use a 7.x runtime. So those would be Spark 3.0, not Spark 3.1
- 0 kudos
- 2057 Views
- 1 replies
- 0 kudos
Using l vacuum with a dry run in Python for a Delta Lake
I can see an example on how to call the vacuum function for a Delta lake in python here. how to use the same in python %sql VACUUM delta.`dbfs:/mnt/<myfolder>` DRY RUN
- 2057 Views
- 1 replies
- 0 kudos
- 0 kudos
The dry run for non-SQL code is not yet available in Delta version 0.8. I see there is a bug that is opened with delta opensource in git . hope it get resolved soon
- 0 kudos
- 1972 Views
- 0 replies
- 0 kudos
MLflow not logging metrics
I have run a few MLflow experiments and I can see them in the experiment history, but none of the metrics have been logged along with them. I thought this was supposed to be automatically included. Any idea why they wouldn't be showing up?
- 1972 Views
- 0 replies
- 0 kudos
- 2146 Views
- 1 replies
- 0 kudos
Resolved! where Can I find the the logs of spark job runs in Azure storage
Hi Want to find the storage bucket where all my runs' logs are stored , I want to do analytics on logs , can you please help me knowing which bucket or path I should look for
- 2146 Views
- 1 replies
- 0 kudos
- 0 kudos
The root bucket where are not directly accessible outside databricks so you need to read the logs from databricks notebook only
- 0 kudos
- 2889 Views
- 1 replies
- 0 kudos
Resolved! Exception: Run with UUID l567845ae5a7cf04a40902ae789076093c is already active.
I'm trying to create a new experiment on mlflow but I have this problem:Exception: Run with UUID l142ae5a7cf04a40902ae9ed7326093c is already active. snippet mlflow.set_experiment("New experiment 2") mlflow.set_tracking_uri('http://mlflow:5000') ...
- 2889 Views
- 1 replies
- 0 kudos
- 0 kudos
You have to run mlflow.end_run() to finish the first experiment. Then you can create another
- 0 kudos
