cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Chandraw
by • New Contributor III
  • 2858 Views
  • 2 replies
  • 0 kudos

Resolved! Malformed Input Exception while saving or retreiving Table

Hi everyone,I am using DBR version 13 and Managed tables in a custom catalog location of table is AWS S3.running notebook on single user clusterI am facing MalformedInputException while saving data to Tables or reading it.When I am running my noteboo...

Data Engineering
Reply
  • 2858 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Chandraw
New Contributor III
  • 0 kudos

@Retired_mod  The issue is resolved as soon as I deployed it to mutlinode dev cluster.Issue is only occurring in single user clusters. Looks like limitation of running all updates in one node as distributed system.

  • 0 kudos
1 More Replies
BerkerKozan
by • New Contributor III
  • 1893 Views
  • 2 replies
  • 1 kudos

Creating All Purpose Cluster in Data Asset Bundles

There is no resource to create All Purpose Cluster, but I need it, so does it mean I should create it via Terraform or DBX and reference to it, which I dont prefer?

Data Engineering
Reply
  • 1893 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
BerkerKozan
New Contributor III
  • 1 kudos

Hello @Ayushi_Suthar, Thanks for the quick reply! Where can I see these requests?https://ideas.databricks.com/ideas/DB-I-9451 ? 

  • 1 kudos
1 More Replies
Andriy
by • New Contributor II
  • 7477 Views
  • 2 replies
  • 1 kudos

Get Job Run Status

Is there a way to get a child Job Run status and show the result within the parent notebook execution?Here is the case: I have a master notebook and several child notebooks. As a result, I want to see which notebook failed: For example Notebook job s...

Screenshot 2024-02-06 at 17.41.51.png
Data Engineering
Reply
  • 7477 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
BR_DatabricksAI
Contributor
  • 1 kudos

Hello, Are you also managing any return status while calling the notebook. Have a look the following reference URL : Run a Databricks notebook from another notebook | Databricks on AWS 

  • 1 kudos
1 More Replies
anupam676
by • New Contributor II
  • 2609 Views
  • 2 replies
  • 1 kudos

Resolved! How can I enable disk cache in this scenario/

I have a notebook where I read multiple tables from delta lake (let say schema is db) and after that I did some sort of transformation (image enclosed) using all these tables lwith transformations like join,filter etc. After transformation and writin...

Data Engineering
Reply
  • 2609 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
anupam676
New Contributor II
  • 1 kudos

Thank you @shan_chandra 

  • 1 kudos
1 More Replies
vroste
by • New Contributor III
  • 1958 Views
  • 1 replies
  • 1 kudos

Delta live tables running count output mode?

I have a DLT with a table that I want to contain the running aggregation (for the sake of simplicitly let's assume it's a count) for each value of some key column, using a session window. The input table goes back several years and to clean up aggreg...

Data Engineering
Reply
  • 1958 Views
  • 1 replies
  • 1 kudos
View Replies
luisvasv
by • New Contributor II
  • 18960 Views
  • 5 replies
  • 2 kudos

Init script problems | workspace location

At this moment, I'm working on removing Legacy global and cluster-named init scripts due, it will be disabled for all workspaces on 01 Sept.At this moment, I'm facing a strange problem regarding moving init scripts from dbfs to the Workspace location...

image.png image
Data Engineering
Reply
  • 18960 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
DE-cat
New Contributor III
  • 2 kudos

Using the new CLI v0.214, uploading ".sh" file works fine.`databricks workspace import --overwrite --format AUTO --file init_setup /init/user/job/init_setup`

  • 2 kudos
4 More Replies
Gauthy1825
by • New Contributor II
  • 6582 Views
  • 9 replies
  • 3 kudos

How to write to Salesforce from Databricks using the spark salesforce library

Hi, Im facing an issue while writing to Salesforce sandbox from Databricks. I have installed the "spark-salesforce_2.12-1.1.4" library and my code is as follows:-df_newLeads.write\      .format("com.springml.spark.salesforce")\      .option("username...

Data Engineering
Reply
  • 6582 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
addy
New Contributor III
  • 3 kudos

I made a function that used the code below and returned url, connectionProperties, sfwriteurl ="https://login.salesforce.com/"dom = url.split('//')[1].split('.')[0]session_id, instance = SalesforceLogin(username=connectionProperties['name'], password...

  • 3 kudos
8 More Replies
Heisenberg
by • New Contributor II
  • 2135 Views
  • 2 replies
  • 1 kudos

Migrate a workspace from one AWS account to another AWS account

Hi everyone,We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach t...

Data Engineering
AWS
Databricks Migration
migration
queries
Workflows
Reply
  • 2135 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
katherine561
New Contributor II
  • 1 kudos

For a streamlined migration of your Databricks workspace from one AWS account to another, start by exporting notebook, workflow, and saved query configurations using Databricks REST API or CLI. Employ Deep Clone or Delta Sharing for managed table dat...

  • 1 kudos
1 More Replies
Luke_H
by • New Contributor II
  • 3028 Views
  • 2 replies
  • 2 kudos

Resolved! Variable referencing in EXECUTE IMMEDIATE

Hi all,As part of an on-going exercise to refactor existing T-SQL code into Databricks, we've stumbled into an issue that we can't seem to overcome through Spark SQL.Currently we use dynamic SQL to loop through a number of tables, where we use parame...

Data Engineering
sql
Variables
Reply
  • 3028 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
SergeRielau
Databricks Employee
  • 2 kudos

DECLARE OR REPLACE varfield_names1 STRING; SET VAR varfield_names1 = 'field1 STRING'; DECLARE OR REPLACE varsqlstring1 STRING; SET VAR varsqlstring1 = 'CREATE TABLE table1 (PrimaryKey STRING, Table STRING, ' || varfield_names1 || ')'; EXECUTE IMMEDI...

  • 2 kudos
1 More Replies
ksamborn
by • New Contributor II
  • 4083 Views
  • 2 replies
  • 0 kudos

withColumnRenamed error on Unity Catalog 14.3 LTS

Hi -  We are migrating to Unity Catalog 14.3 LTS and have seen a change in behavior using withColumnRenamed.There is an error COLUMN_ALREADY_EXISTS on the join key, even though the column being renamed is a different column.   The joined DataFrame do...

Data Engineering
Data Lineage
Unity Catalog
Reply
  • 4083 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Palash01
Valued Contributor
  • 0 kudos

Hey @ksamborn I can think of 2 solutions:Rename the column in df_2 before joining: df_1_alias = df_1.alias("t1") df_2_alias = df_2.alias("t2") join_df = df_1_alias.join(df_2_alias, df_1_alias.key == df_2_alias.key) rename_df = join_df.withColumnRenam...

  • 0 kudos
1 More Replies
RabahO
by • New Contributor III
  • 1878 Views
  • 2 replies
  • 0 kudos

Resolved! Unit tests in notebook not working

Hello, I'm trying to setup a notebook for tests or data quality checks. The name is not important.I basically read a table (the ETL output process - actual data).Then I read another table and do the calculation in the notebook (expected data)I'm stuc...

Data Engineering
Reply
  • 1878 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
RabahO
New Contributor III
  • 0 kudos

thank you for the nutter. Tried it and it seems to answer my problematic.

  • 0 kudos
1 More Replies
Avinash_Narala
by • Valued Contributor II
  • 2199 Views
  • 0 replies
  • 0 kudos

Unable to use SQL UDF

Hello, I want to create an sql udf as follows:%sqlCREATE or replace FUNCTION get_type(s STRING)  RETURNS STRING  LANGUAGE PYTHON  AS $$    def get_type(table_name):      from pyspark.sql.functions import col      from pyspark.sql import SparkSession ...

Data Engineering
sqludf
Reply
  • 2199 Views
  • 0 replies
  • 0 kudos
kiko_roy
by • Contributor
  • 2099 Views
  • 2 replies
  • 1 kudos

Resolved! DLT cluster : can be manipulated?

Hi All I am using DLT pipeline to pull data from a ADLS gen2 which is mounted. The cluster that is getting fired has access mode is set to shared. I want to change it to single user. But the cluster being attached to DLT , I am not ablr to update and...

Data Engineering
Reply
  • 2099 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
AlliaKhosla
Databricks Employee
  • 1 kudos

Hi @kiko_roy  Greetings! You can't use a single-user cluster to query tables from a Unity Catalog-enabled Delta Live Tables pipeline, including streaming tables and materialized views in Databricks SQL. To access these tables, you need to use a share...

  • 1 kudos
1 More Replies
ss6
by • New Contributor II
  • 1056 Views
  • 1 replies
  • 0 kudos

Resolved! Liquid Cluster - SHOW CREATE TABLE error

We've got this table with liquid clustering turned on at first, but then we switch off with below command.ALTER TABLE table_name CLUSTER BY NONE;Now, our downstream process that usually runs "SHOW CREATE TABLE" is hitting a snag. It's throwing this e...

Data Engineering
Reply
  • 1056 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @ss6 , Hope you are doing well!  We would like to inform you that currently, SHOW CREATE TABLE is not supported after running ALTER TABLE CLUSTER BY NONE. This is a known issue and our Engineering team is prioritizing a fix to retain the clusterin...

  • 0 kudos
isaac_gritz
by • Databricks Employee
  • 19648 Views
  • 2 replies
  • 2 kudos

Using Plotly Dash with Databricks

How to use Plotly Dash with DatabricksWe recommend checking out this article for the latest on building Dash Applications on top of the Databricks Lakehouse.Let us know in the comments if you use Plotly and if you're planning on adopting the latest i...

Data Engineering
Reply
  • 19648 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
dave-at-plotly
New Contributor III
  • 2 kudos

Hey all.  Just wanted to make sure everyone had some up-to-date intel regarding leveraging Plotly Dash with Databricks.Most Dash app integrations w Databricks today leverage the Databricks Python SQL Connector.  More technical details are available v...

  • 2 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels