Databricks Community

PrebenOlsen · 04-22-2024

Hi!I want to migrate all my databricks related code from one github repo to another. I knew this wouldn't be straight forward. When I copy my code for one DLT, I get the errororg.apache.spark.sql.catalyst.ExtendedAnalysisException: Table 'vessel_batt...

PrebenOlsen · 04-17-2024

Hi!Started a job yesterday. It was iterating over data, 2-months at a time, and writing to a table. It was successfully doing this for 4 out of 6 time periods. The 5th time period however, got stuck, 5 hours in.I can find one Failed Stage that reads ...

PrebenOlsen · 07-27-2023

Hi!Getting error message:DatabricksError: User XXX is not part of org: YYY. Config: host=https://adb-ZZZ.azuredatabricks.net, auth_type=runtimeI am in the admin's group, but I cannot alter this in any way. I've tried using the databricks-SDK using:fr...

PrebenOlsen · 08-29-2022

I have some data in silver that I read in as a view using the __apply_changes function on. I create a table based on this, and I then want to create my gold-table, after doing a .groupBy() and .pivot(). The transformations I do in the gold-table aren...

PrebenOlsen · 08-26-2022

I have a delta live table that I'm trying to run GroupBy on, but getting an error: "RuntimeError: Query function must return either a Spark or Koalas DataFrame". Here is my code:@dlt.table def groups_hierarchy(): df = dlt.read_stream("groups_h...

PrebenOlsen · 04-29-2024

Does cloning take considerably less time then recreating the tables?Can I resume append operations to a cloned table?

PrebenOlsen · 04-17-2024

Hi Werners, I agree about your explanation for read and write - but that's what the GUI looks like. For each iteration (spark.read.table.where(col("month") == "January") (and then February in the next iteration), it spends about 30 minutes on only 3 ...

PrebenOlsen · 08-01-2023

I am both Contributor and Data Contributor in Azure. In Databricks, I belong to the Admin group.Noone in my team can do anything with the cluster as per now.

PrebenOlsen · 08-30-2022

I have found a temporary solution to solve this. The .pivot("columnName") should automatically grab all the values it can find, but for some reason it does not. I need to specify the values, using.pivot("group_name", "group0", "group1", "group2"...) ...

PrebenOlsen · 08-28-2022

No, there is no join in my code above. The problem occurs at the very last line, when trying to use a .groupBy. This works fine when reading a non-streaming view, but fails as soon as it is a stream. Is groupBy not supported yet for streams?

Databricks Community

User Stats

User Activity

How to migrate Git repos with DLT configurations

Job stuck while utilizing all workers

Can't start, delete, unpin or edit cluster: User is not part of org

Why does @dlt.table from a table give different results than from a view?

GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

Re: How to migrate Git repos with DLT configurations

Re: Job stuck while utilizing all workers

Re: Can't start, delete, unpin or edit cluster: User is not part of org

Re: Why does @dlt.table from a table give different results than from a view?

Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"