cancel
Showing results for 
Search instead for 
Did you mean: 
Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

Activity in Discussions

ksamborn
by New Contributor
  • 80 Views
  • 1 replies
  • 0 kudos

withColumnRenamed error on Unity Catalog 14.3 LTS

Hi -  We are migrating to Unity Catalog 14.3 LTS and have seen a change in behavior using withColumnRenamed.There is an error COLUMN_ALREADY_EXISTS on the join key, even though the column being renamed is a different column.   The joined DataFrame do...

Data Engineering
Data Lineage
Unity Catalog
  • 80 Views
  • 1 replies
  • 0 kudos
Latest Reply
Palash01
New Contributor III
  • 0 kudos

Hey @ksamborn I can think of 2 solutions:Rename the column in df_2 before joining: df_1_alias = df_1.alias("t1") df_2_alias = df_2.alias("t2") join_df = df_1_alias.join(df_2_alias, df_1_alias.key == df_2_alias.key) rename_df = join_df.withColumnRenam...

  • 0 kudos
Brad
by New Contributor
  • 71 Views
  • 2 replies
  • 0 kudos

Dash in Databricks notebook directly

Hi team,Is there a way to embed plotly dash directly inside Databricks notebook?Thanks

  • 71 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brad
New Contributor
  • 0 kudos

Thanks Dave.Is it possible to embed Dash to Databricks notebook cell result directly? Visualization in Databricks notebook can be published to dashboards. We expect to use Dash inside the Databricks notebook and publish it so inside dashboard people ...

  • 0 kudos
1 More Replies
Peter_Jones
by New Contributor
  • 26 Views
  • 0 replies
  • 0 kudos

Syntax of UPDATE Command in DataBricks

Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...

  • 26 Views
  • 0 replies
  • 0 kudos
isaac_gritz
by Valued Contributor II
  • 5113 Views
  • 3 replies
  • 4 kudos

Using Plotly Dash with Databricks

How to use Plotly Dash with DatabricksWe recommend checking out this article for the latest on building Dash Applications on top of the Databricks Lakehouse.Let us know in the comments if you use Plotly and if you're planning on adopting the latest i...

  • 5113 Views
  • 3 replies
  • 4 kudos
Latest Reply
dave-at-plotly
  • 4 kudos

Hey all.  Just wanted to make sure everyone had some up-to-date intel regarding leveraging Plotly Dash with Databricks.Most Dash app integrations w Databricks today leverage the Databricks Python SQL Connector.  More technical details are available v...

  • 4 kudos
2 More Replies
277745
by New Contributor
  • 28 Views
  • 0 replies
  • 0 kudos

Pandas_Udod max batch size not working in notebook

Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...

  • 28 Views
  • 0 replies
  • 0 kudos
johnboscow
by New Contributor
  • 43 Views
  • 0 replies
  • 0 kudos

Passed the Databricks Data Engineer Associate exam

Passed the Databricks Data Engineer Associate exam V3 on Sat 24 feb 2024. With a score of 96%. These questions really help. Please note settle differences between the questions here and the ones in the practice tests. They are small differences that ...

  • 43 Views
  • 0 replies
  • 0 kudos
SirCrayon
by New Contributor
  • 58 Views
  • 0 replies
  • 0 kudos

Do shared clusters have multiple drivers?

Hi,I know that with single clusters, theres a single driver node and one driver per cluster. With shared clusters, multiple jobs can run concurrently. Does this still run on a single driver container or multiple driver containers run per application?...

  • 58 Views
  • 0 replies
  • 0 kudos
ss6
by New Contributor
  • 56 Views
  • 0 replies
  • 0 kudos

Liquid Cluster - SHOW CREATE TABLE error

We've got this table with liquid clustering turned on at first, but then we switch off with below command.ALTER TABLE table_name CLUSTER BY NONE;Now, our downstream process that usually runs "SHOW CREATE TABLE" is hitting a snag. It's throwing this e...

  • 56 Views
  • 0 replies
  • 0 kudos
Mathias
by New Contributor II
  • 1272 Views
  • 3 replies
  • 0 kudos

Different settings per target with Asset bundles

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.I have dlt pipelines th...

  • 1272 Views
  • 3 replies
  • 0 kudos
Latest Reply
137292
New Contributor II
  • 0 kudos

-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?

  • 0 kudos
2 More Replies
Brad
by New Contributor
  • 95 Views
  • 2 replies
  • 0 kudos

Is there a way to control the cluster runtime version for DLT

Hi team, When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.Thanks

  • 95 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brad
New Contributor
  • 0 kudos

Thanks. I mean running DLT, not run a cell from notebook sourced for DLT.

  • 0 kudos
1 More Replies
Heisenberg
by New Contributor
  • 187 Views
  • 0 replies
  • 0 kudos

Migrate a workspace from one AWS account to another AWS account

Hi everyone,We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach t...

Data Engineering
AWS
Databricks Migration
migration
queries
Workflows
  • 187 Views
  • 0 replies
  • 0 kudos
Mumrel
by New Contributor III
  • 114 Views
  • 1 replies
  • 0 kudos

Service Principal can be deleted but permissions not managed

On Azure I added a service principal X to my databricks workspace. I therefore had the Service Prinicpal Manager role on that service principal X. I accidentally downgraded my rights to Service Principal User and now can's get my Managers role back. ...

Mumrel_0-1708705677371.png
  • 114 Views
  • 1 replies
  • 0 kudos
Latest Reply
jamessmith3
New Contributor II
  • 0 kudos

Do you have federated identity enabled on your workspace?

  • 0 kudos
Dhruv_Sinha
by New Contributor
  • 124 Views
  • 2 replies
  • 1 kudos

Parallelizing processing of multiple spark dataframes

Hi all, I am trying to create a collection rd that contains a list of spark dataframes. I want to parallelize the cleaning process for each of these dataframes. Later on, I am sending each of these dataframes to another method. However, when I parall...

  • 124 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Dhruv_Sinha, The issue you’re encountering—where the Spark context (sc) cannot be accessed from worker nodes—is a common challenge when working with Spark. Let’s explore why this happens and discuss potential workarounds. Spark Context and W...

  • 1 kudos
1 More Replies
dbph
by New Contributor
  • 57 Views
  • 0 replies
  • 0 kudos

Databricks asset bundles error "failed to instantiate provider"

Hi all,I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for dat...

  • 57 Views
  • 0 replies
  • 0 kudos
exilon
by New Contributor
  • 60 Views
  • 0 replies
  • 0 kudos

DLT streaming with sliding window missing last windows interval

Hello, I have a DLT pipeline where I want to calculate the rolling average of a column for the last 24 hours which is updated every hour.I'm using the below code to achieve this:       @Dlt.table() def gold(): df = dlt.read_stream("silver_table")...

Data Engineering
dlt
spark
streaming
window
  • 60 Views
  • 0 replies
  • 0 kudos
Top Kudoed Authors