cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Satyadeepak
by Databricks Employee
  • 1459 Views
  • 1 replies
  • 1 kudos
  • 1459 Views
  • 1 replies
  • 1 kudos
Latest Reply
aladda
Databricks Employee
  • 1 kudos

Can you provide an example of what exactly do you mean? If the reference is to how "Repos" shows up in the UI, that's more for a Ux convenience. Repos as such are designed to be a container for version controlled notebooks that live in the Git reposi...

  • 1 kudos
User16790091296
by Contributor II
  • 2503 Views
  • 2 replies
  • 0 kudos
  • 2503 Views
  • 2 replies
  • 0 kudos
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Delta Lake on Databricks has added runtime optimizations of the Delta Engine that further enhance the performance and scale of the open source Delta Format. In additional you also get access to a whole host of capabilities available on the Databricks...

  • 0 kudos
1 More Replies
MoJaMa
by Databricks Employee
  • 4192 Views
  • 1 replies
  • 1 kudos
  • 4192 Views
  • 1 replies
  • 1 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 1 kudos

Yes, you can pass parameters from ADF —> Azure Databricks.https://docs.microsoft.com/en-us/azure/data-factory/solution-template-databricks-notebook#how-to-use-this-templateYou can also pass values back from the Notebook --> ADF via the dbutils.notebo...

  • 1 kudos
brickster_2018
by Databricks Employee
  • 1803 Views
  • 1 replies
  • 0 kudos

Maven library conflicts

The import statements work fine with maven libraries freshly installed on a running cluster and on restart of the same cluster, it fails to import the classes.

  • 1803 Views
  • 1 replies
  • 0 kudos
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

When you install maven libraries one by one in running clusters, it is resolved and gets downloaded individually. The behavior would be different when you restart the clusters since the libraries are resolved together and can run into issues due to c...

  • 0 kudos
MoJaMa
by Databricks Employee
  • 1806 Views
  • 1 replies
  • 0 kudos
  • 1806 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 0 kudos

The cluster will be terminated. But anyone with CAN_MANAGE can continue to start it, or clone it, and then delete it.

  • 0 kudos
aladda
by Databricks Employee
  • 2328 Views
  • 1 replies
  • 0 kudos
  • 2328 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Databricks Employee
  • 0 kudos

For an optimal processing experience Databricks segregates the Spark application traffic from the management traffic to avoid Network contention. Spark application traffic communications between the Driver-Executor and the Executors themselves where ...

  • 0 kudos
brickster_2018
by Databricks Employee
  • 1380 Views
  • 1 replies
  • 0 kudos

Resolved! Does Delta perform listing of data directories?

I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over timeHow about the log directories. I have more than 100k JSON files in the log directory

  • 1380 Views
  • 1 replies
  • 0 kudos
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. Similarly with in the Delta log directory listing is performed. However, it's performed using a continu...

  • 0 kudos
User16826992666
by Valued Contributor
  • 3135 Views
  • 1 replies
  • 0 kudos

Resolved! Can I copy my MLflow experiments from one workspace to another?

I would like to move my saved experiments and artifacts to a different Databricks workspace from where I originally created them. Is this possible?

  • 3135 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

It might be possible with a bit of code via mlflow client api ( there seems to be a way to run list_registered_models and extract info ) - but haven't tried it out. If the requirement is to share models between workspaces, one approach could be to h...

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels