Databricks Community

-werners- · 10-08-2024

Did anyone succeed in using already existing compute policies (created using the UI) in asset bundles for creating a job?I defined the policy_id in the resources/job yml for the job_cluster, but when deploying I get errors saying spark version is not...

-werners- · 02-21-2024

I created on our dev environment a cluster using the shared access mode, for our devs to use (instead of separate single user clusters).What I notice is that the performance of this cluster is terrible. And I mean really terrible: notebook cells wit...

-werners- · 12-05-2023

Hi all,we are in the process of rolling out a new unity-enabled databricks env with 2 tiers: dev and prod.Initially we had the plan to completely decouple dev and prod, each with their own data lake as storage.While this is the safest option, it does...

-werners- · 10-20-2023

We use typesafe (scala) to read configuration values from hocon files.When not using Unity, we read the configuration files from /dbfs/... works fine.However, with Unity, usage of dbfs is frowned upon.So I started looking into alternatives.And unfor...

-werners- · 09-19-2023

Hi all,I try to create an external volume in a catalog/schema on a subpath of the external location.This works fine, however for a certain path I cannot create an external volume. I always getInput path url overlaps with managed storage within 'Crea...

-werners-

Since you replied on a rather old topic: TorchDistributor enables pytorch on spark in distributed mode.But a cluster with only 1 worked and 1 driver will not run in distributed mode.The driver does not execute spark tasks, it handles spark overhead a...

-werners-

There is no databricks documentation on this as it is only involved for a very tiny bit:"CREATE TABLE catalog.schema.table USING PARQUET LOCATION 'url_to_the_parquet_files'.All the rest is done in Azure Data Factory, or you can even use the built-in ...

-werners-

I suppose this is a Dedicated Sql pool you are talking about?If so, I'd use Data Factory to extract the data from synapse and land it in ADLS in parquet format f.e. Then create a UC/Hive table on this.

-werners-

Variables are useful but it depends on how you set up the bundles.I define a policy per target. Since 'policy' is not known to the target, I create a variable and assign it a different value depending on the environment.This variable is then used i...

-werners-

the overwrite goes after the write.What you can do is this:First write to a new table like testTable1 (delta lake).The run the write again using write.mode("overwrite")... on the same table, should work.

Databricks Community

User Stats

User Activity

asset bundles and compute policies

performance issues using shared compute access mode in scala

multiple storage credentials/external locations to same physical location

best way to store config files in a Unity workspace (Scala/typesafe)

external volume: overlap error with empty catalog?

Re: Does Databricks supports the Pytorch Distributed Training for multiple devices?

Re: Migrating huge table from synapse to databricks

Re: Migrating huge table from synapse to databricks

Re: policy_id in databricks asset bundle workflow

Re: saveAsTable sometimes works sometimes dont