cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

User16826992666
by Valued Contributor
  • 2446 Views
  • 1 replies
  • 0 kudos

Resolved! Can I copy my MLflow experiments from one workspace to another?

I would like to move my saved experiments and artifacts to a different Databricks workspace from where I originally created them. Is this possible?

  • 2446 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

It might be possible with a bit of code via mlflow client api ( there seems to be a way to run list_registered_models and extract info ) - but haven't tried it out. If the requirement is to share models between workspaces, one approach could be to h...

  • 0 kudos
MoJaMa
by Databricks Employee
  • 977 Views
  • 1 replies
  • 0 kudos
  • 977 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 0 kudos

If it's disabled / lost, then it's broken. If customer cannot recover the key, then it's gone forever. Databricks has no knowledge of the key (us knowing it would render it insecure).

  • 0 kudos
User16790091296
by Contributor II
  • 1106 Views
  • 1 replies
  • 0 kudos
  • 1106 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Depends on what you're looking for from a management perspective, but one option is the Account API which allows deploying/updating/configuring multiple workspaces in a given E2 accountUse this API to programmatically deploy, update, and delete works...

  • 0 kudos
brickster_2018
by Databricks Employee
  • 1645 Views
  • 1 replies
  • 0 kudos
  • 1645 Views
  • 1 replies
  • 0 kudos
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2020-09-01" | jq '.compute.tagsList[] | select(.name=="Creator") | .value'

  • 0 kudos
aladda
by Databricks Employee
  • 3951 Views
  • 1 replies
  • 1 kudos

Why do Databricks deployments require 2 subnets for each workspace

Databricks must have access to at least two subnets for each workspace, with each subnet in a different availability zone per docs here

  • 3951 Views
  • 1 replies
  • 1 kudos
Latest Reply
aladda
Databricks Employee
  • 1 kudos

This is designed for optimal user experience and as a capacity planning strategy where if instances are not available in one AZ, the other subnet in a different AZ can be used to deploy instance from instead

  • 1 kudos
brickster_2018
by Databricks Employee
  • 2830 Views
  • 1 replies
  • 1 kudos
  • 2830 Views
  • 1 replies
  • 1 kudos
Latest Reply
brickster_2018
Databricks Employee
  • 1 kudos

Find the DriverDaemon%sh jpsTake the heap dump%sh jmap -dump:live,format=b,file=pbs_worker_DriverDaemon.hprof 2413Copy out to download%sh cp pbs_worker_DriverDaemon.hprof /dbfs/FileStore/pbs_worker_04-30-2021T15-50-00.hprof

  • 1 kudos
User16826992666
by Valued Contributor
  • 6004 Views
  • 1 replies
  • 0 kudos

Resolved! When using MLflow should I use log_model or save_model?

They seem to have similar functions. What is the recommended pattern here?

  • 6004 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

mlflow.<model-type>.log_model(model, ...) saves the model to the MLflow tracking server. mlflow.<model-type>.save_model(model, modelpath) saved the model locally to  a DBFS path.More details at https://docs.databricks.com/applications/mlflow/models...

  • 0 kudos
Anonymous
by Not applicable
  • 2539 Views
  • 2 replies
  • 2 kudos

Resolved! Spot instances - Best practice

We are having difficulties running our jobs with spot instances that get re-claimed by AWS during shuffles. Do we have any documentation / best-practices around this? We went through this article but is there anything else to keep in mind?

  • 2539 Views
  • 2 replies
  • 2 kudos
Latest Reply
User16783853906
Contributor III
  • 2 kudos

Due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price (>100%) are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail.To mitigate this, w...

  • 2 kudos
1 More Replies
Ryan_Chynoweth
by Esteemed Contributor
  • 1027 Views
  • 1 replies
  • 0 kudos
  • 1027 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

No, each table must be defined once. You can Use UNION If you need to combine multiple inputs to create a table. Adding or removing UNION from an incremental table is a breaking operation that requires a full-refresh.

  • 0 kudos
User16826992666
by Valued Contributor
  • 1101 Views
  • 1 replies
  • 0 kudos

Where can I find the tables I created in my Delta Live Tables pipeline?

I created several tables in my DLT pipeline but didn't specify a location to save them on creation. The pipleline seems to have ran, but I don't know where the tables actually are. How can I find them?

  • 1101 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

Checkout the configuration storage under settings . If you didn't specify the  storage  setting, the system will default to a location in  dbfs:/pipelines/

  • 0 kudos
User16826987838
by Contributor
  • 1324 Views
  • 1 replies
  • 0 kudos
  • 1324 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

Yes, in your write stream you can save it as a table in the delta format without a problem. In DBR 8, the default table format is delta. See this code, please note that the "..." is supplied to show that additional options may be required: df.writeSt...

  • 0 kudos
User16826992666
by Valued Contributor
  • 2045 Views
  • 1 replies
  • 0 kudos

When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?

When using SQL, I can use the Create Live Table command and the Create Incremental Live Table command to set the run type I want the table to use. But I don't seem to have that same syntax for python. How can I set this table type while using Python?

  • 2045 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

The documentation at https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#mixing-complete-tables-and-incremental-tables has an example the first two functions load data incrementally and the last one loads...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels