cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
RKNutalapati
by • Valued Contributor
  • 1602 Views
  • 3 replies
  • 0 kudos

Jobs API "run now" - How to set task wise parameters

I have a job with multiple tasks like Task1 -> Task2 -> Task3. I am trying to call the job using api "run now". Task details are belowTask1 - It executes a Note Book with some input parametersTask2 - It runs using "ABC.jar", so its a jar based task ...

Data Engineering
Reply
  • 1602 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Harsha777
New Contributor III
  • 0 kudos

Hi,It would be a good feature to pass parameters at task level. We have scenarios where we would like to create a job with multiple tasks (notebook/dbt) and pass parameters at task level.

  • 0 kudos
2 More Replies
hadoan
by • New Contributor II
  • 642 Views
  • 2 replies
  • 1 kudos

How to define DLT table with cyclic reference

 @Dlt.table def table_A(): return ( dlt.read_stream(...) ) @dlt.table def table_join_A_and_C(): df_A = dlt.read_stream(table_A) df_C = dlt.read_stream(table_C) return ( ....df_A.join(df_C) ) @dlt.table def table_C(): return ( ...

Data Engineering
Reply
  • 642 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Community Manager
  • 1 kudos

Hi @hadoan , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback ...

  • 1 kudos
1 More Replies
safoineext
by • New Contributor
  • 581 Views
  • 1 replies
  • 0 kudos

Uploading wheel using `dbutils.fs.cp` to workspace and install it in Runtime>15

I have been trying to find an alternative to copying a wheel file from my local file system to Databricks and then installing it into the cluster. Doing this databricks_client.dbutils.fs.cp("file:/local..../..whl", "dbfs:/Workspace/users/..../..whl")...

safoineext_0-1720009993682.png
Data Engineering
Reply
  • 581 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Community Manager
  • 0 kudos

Hi @safoineext , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...

  • 0 kudos
Mahesh_Yadav
by • New Contributor
  • 425 Views
  • 1 replies
  • 0 kudos

System Access Column lineage showing inaccurate results

Hi All,I have been trying to leverage the system column lineage table to check the overall journey of a column. But i am getting inaccurate results wherever unpivot transformations are used.Instead of showing the results in a way that 20 columns are ...

Mahesh_Yadav_1-1719985303244.png
Data Engineering
Reply
  • 425 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Community Manager
  • 0 kudos

Hi @Mahesh_Yadav , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your fee...

  • 0 kudos
beautrincia
by • New Contributor
  • 450 Views
  • 1 replies
  • 0 kudos

How to get data permissions from Sharepoint and Confluence to Unity Catalog for RAG LLM chatbot

We're implementing a chatbot where documents in SharePoint and pages in Confluence augment the results. We want to adhere to existing RBAC policies in these data sources so that the chatbot doesn't produce results that someone should not see. Are you...

Data Engineering
Reply
  • 450 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Community Manager
  • 0 kudos

Hi @beautrincia , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feed...

  • 0 kudos
Tiwarisk
by • New Contributor II
  • 1191 Views
  • 5 replies
  • 3 kudos

How can I preserve the data type of the delta tables while writing to Azure blob storage ?

I am writing a file using this but the data type of columns get changed while reading. df.write.format("com.crealytics.spark.excel").option("header", "true").mode("overwrite").save(path) Due to this I have to manual change every time as I can't chang...

Data Engineering
Reply
  • 1191 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Community Manager
  • 3 kudos

Hi @Tiwarisk , Thank you for reaching out to our community! We're here to help you.To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...

  • 3 kudos
4 More Replies
938452
by • New Contributor III
  • 11610 Views
  • 3 replies
  • 2 kudos

Resolved! Executor memory increase limitation based on node type

Hi Databricks community,I'm using Databricks Jobs Cluster to run some jobs. I'm setting the worker and driver type to AWS m6gd.large, which has 2 cores and 8G of memory each.After seeing it's defaulting executor memory to 2G, I wanted to increase it,...

Data Engineering
Reply
  • 11610 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
938452
New Contributor III
  • 2 kudos

I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memoryIt seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databric...

  • 2 kudos
2 More Replies
drag7ter
by • New Contributor III
  • 461 Views
  • 0 replies
  • 0 kudos

SQL AI functions in EU region

I know that currently foundation model with pay-per-token are not available in EU only in US. In EU I should create serving point and use provisioned foundation model. But even creating a serving point with llm from catalog (share models). I used the...

drag7ter_0-1720601756043.png
Data Engineering
Reply
  • 461 Views
  • 0 replies
  • 0 kudos
sathya08
by • New Contributor
  • 284 Views
  • 0 replies
  • 0 kudos

Databricks Asset Bundle Error

Hello,I am trying the Databricks Asset bundle for the first time. I am using Databricks CLI and can able to validate the bundle but when I am trying to run it it errors out  error="expected a KEY of the resource to run" .In the resource yml file I ha...

Data Engineering
Reply
  • 284 Views
  • 0 replies
  • 0 kudos
2vinodhkumar
by • New Contributor II
  • 375 Views
  • 1 replies
  • 0 kudos

Autoloader - Ingestion of CSV files when there is not operation column

Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...

Data Engineering
Reply
  • 375 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
crankerkor
by • New Contributor III
  • 2245 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks JDBC SQL Warehouse Encoding Issue

Hi Everyone.I am trying to connect and read data from the Databricks table using SQL Warehouse and return it using Azure API.However, the non-English characters, for example, 'Ä', are present in the response as following: ��.I am using the databricks...

Data Engineering
Reply
  • 2245 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
151640
New Contributor III
  • 1 kudos

If Databricks support/Product Management following the forum, note that PDF from SIMBA in 2.6.28 does not discuss the name-value pairs in the above solution.Other errata includes PreparedMetadataLimitZero.

  • 1 kudos
2 More Replies
yvuignie
by • Contributor
  • 6447 Views
  • 12 replies
  • 3 kudos

Resolved! Unity catalog - How do you modify groups properly ?

Hello,What is the best practice to modify/delete/recreate groups properly ?In order to rename a group, the only mean was to delete/recreate. But after deletion in the account console, the permissions granted to the deleted groups in the tables were i...

Data Engineering
Reply
  • 6447 Views
  • 12 replies
  • 3 kudos
View Replies
Latest Reply
RobinK
Contributor
  • 3 kudos

Hello,I have exactly the same issue - I am also using terraform.I deleted a group and the catalog permissions are in bad state.  I am not able to revoke access to this group using the Databricks UI nor REST API. I also tried to recreate the group wit...

  • 3 kudos
11 More Replies
Fabich
by • New Contributor II
  • 2194 Views
  • 3 replies
  • 1 kudos

What's the ETA for supporting Java 21 in the JDBC Driver ?

Hello,I have seen this other post about the Java JDBC driver not working in Java 21.The post is now 3 months old and Java 21 has been available for even longer, is there any update on the topic ?Can you communicate any ETA of when we can expect the d...

Data Engineering
driver
java
java21
JDBC
Reply
  • 2194 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Walter_C
Honored Contributor
  • 1 kudos

Hello, unfortunately as of now there is still no ETA of support of JAVA 32 with the Arrow functionality, the team is working on this but still no information of release has been provided

  • 1 kudos
2 More Replies
data-engineer-d
by • Contributor
  • 1374 Views
  • 1 replies
  • 2 kudos

Liquid Clustering - Number of files are increasing

We enabled liquid clustering on one of the large tables (380GBs). This table goes many operations daily, which improved many folds after liquid clustering. However, after enabling liquid clustering and optimizing it number of files are increased.Prev...

Data Engineering
Databricks
delta
Liquid clustering
Reply
  • 1374 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
data-engineer-d
Contributor
  • 2 kudos

Thank you for detailed explanation @Retired_mod .

  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels