cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
yvuignie
by • Contributor
  • 6973 Views
  • 12 replies
  • 3 kudos

Resolved! Unity catalog - How do you modify groups properly ?

Hello,What is the best practice to modify/delete/recreate groups properly ?In order to rename a group, the only mean was to delete/recreate. But after deletion in the account console, the permissions granted to the deleted groups in the tables were i...

Data Engineering
Reply
  • 6973 Views
  • 12 replies
  • 3 kudos
View Replies
Latest Reply
RobinK
Contributor
  • 3 kudos

Hello,I have exactly the same issue - I am also using terraform.I deleted a group and the catalog permissions are in bad state.  I am not able to revoke access to this group using the Databricks UI nor REST API. I also tried to recreate the group wit...

  • 3 kudos
11 More Replies
Fabich
by • New Contributor II
  • 2425 Views
  • 3 replies
  • 1 kudos

What's the ETA for supporting Java 21 in the JDBC Driver ?

Hello,I have seen this other post about the Java JDBC driver not working in Java 21.The post is now 3 months old and Java 21 has been available for even longer, is there any update on the topic ?Can you communicate any ETA of when we can expect the d...

Data Engineering
driver
java
java21
JDBC
Reply
  • 2425 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Hello, unfortunately as of now there is still no ETA of support of JAVA 32 with the Arrow functionality, the team is working on this but still no information of release has been provided

  • 1 kudos
2 More Replies
data-engineer-d
by • Contributor
  • 1569 Views
  • 1 replies
  • 2 kudos

Liquid Clustering - Number of files are increasing

We enabled liquid clustering on one of the large tables (380GBs). This table goes many operations daily, which improved many folds after liquid clustering. However, after enabling liquid clustering and optimizing it number of files are increased.Prev...

Data Engineering
Databricks
delta
Liquid clustering
Reply
  • 1569 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
data-engineer-d
Contributor
  • 2 kudos

Thank you for detailed explanation @Retired_mod .

  • 2 kudos
GeKo
by • New Contributor III
  • 1330 Views
  • 1 replies
  • 1 kudos

Resolved! Asset Bundles : how to conditionally set content of a template file

Hello,since Asset Bundles is based on GO templating mechanism, I am wondering how it is possible to use IF-ELSE construct within a template file, to define which file content will be set in the generated file ( I want to have that in my custom templa...

Data Engineering
asset bundle
bundles
template
Reply
  • 1330 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
GeKo
New Contributor III
  • 1 kudos

Never mind....I figured out the solution I just have to prefix my template file with ".tmpl", then it gets rendered correctly

  • 1 kudos
ajithgaade
by • New Contributor III
  • 862 Views
  • 4 replies
  • 0 kudos

Databricks Job Params

Hi,Job params override the task params(same name params). Is there a way task params override the job params.Use case:job params: a = "param-1".job has 12 tasks. 10 of them should use job param(a = "param-1").2 of them should override the job param(a...

Data Engineering
Reply
  • 862 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @ajithgaade,Unfortunately, I don't think it is currently possible. It's clearly stated in documentation that:" Job parameters take precedence over task parameters. If a job parameter and a task parameter have the same key, the job parameter overri...

  • 0 kudos
3 More Replies
DataSax
by • New Contributor III
  • 1660 Views
  • 4 replies
  • 3 kudos

Resolved! Just a beginner in Data Engineer

Hi Everyone,I am happy to be part of this great community.I just determined to be a Data Engineer by profession and I will need a lot of advice on how I can quickly grab it and become  a professional.I have Python Programming knowledge and Web develo...

Data Engineering
Reply
  • 1660 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 3 kudos

Most import thing, at least at the beginning of your data journey is to grasp a good understanding of SQL. It's cornerstone in data world.Definitely you should familiarize yourself with a concept of data modeling, especially dimensional modeling that...

  • 3 kudos
3 More Replies
taimoor123
by • New Contributor
  • 912 Views
  • 1 replies
  • 0 kudos

Failure to Create Materialize View - Error Starting DLT Service on Databricks Cluster xxxx-xxxxxx-xx

 Hi,I am trying to create a materialized view; however, it failed. I could not find any reason for this issue.Cluster Summary:Cluster Type: SQL warehouse serverless with Unity CatalogWorkers: 1-4 Workers (16-64 GB Memory, 4-16 Cores)Driver: 1 Driver ...

taimoor123_0-1718271224512.png taimoor123_2-1718272469077.png
Data Engineering
Reply
  • 912 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
gabsylvain
Databricks Employee
  • 0 kudos

Hey @taimoor123 , could you share the stack trace for the error please ?  Also, I find it a bit confusing. You mention you are using a Serverless SQL warehouse to run the query, but you are also providing the details of the configuration of what seem...

  • 0 kudos
negrinij
by • New Contributor
  • 25433 Views
  • 3 replies
  • 0 kudos

Understanding Used Memory in Databricks Cluster

Hello, I wonder if anyone could give me any insights regarding used memory and how could I change my code to "release" some memory as the code runs. I am using a Databricks Notebook.Basically, what we need to do is perform a query, create a spark sql...

image.png image
Data Engineering
Reply
  • 25433 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
JKR
Contributor
  • 0 kudos

Did anyone find the solution for mentioned issue?

  • 0 kudos
2 More Replies
Gareema
by • New Contributor III
  • 873 Views
  • 3 replies
  • 2 kudos

Data Lineage with Apply Changes

Hello TeamI am using DLT. I am able to see the lineage when doing normal process. However as soon as I use 'APPLY_Changes' feature after the lineag ebreaks and I am no more able to see the Data lineage from the catalog after going to table. Is there ...

Gareema_0-1720379993368.png
Data Engineering
Reply
  • 873 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Gareema
New Contributor III
  • 2 kudos

@Retired_mod : Is there any way this can be achieved or can we expect this problem to be resolved in next releases?

  • 2 kudos
2 More Replies
leungi
by • Contributor
  • 687 Views
  • 2 replies
  • 0 kudos

Resolved! Init Script Fails Intermittently on Workflow Job

An init script is used to install system libraries, per below.Adding the script to a Personal Compute consistently works. The same script is added to a Workflows job via cluster config, which intermittently fails, as shown in error message below.Both...

leungi_0-1718291897408.png
Data Engineering
Reply
  • 687 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
amr
Databricks Employee
  • 0 kudos

Check the cluster event log to see if there is a clue why the script is failing. if the script failed and returned none zero status the cluster wont start

  • 0 kudos
1 More Replies
Enrique1987
by • New Contributor III
  • 568 Views
  • 1 replies
  • 0 kudos

Resolved! Overwriting mode do not overwrite

I have the following codePreviously I have a delta table with 180 columns in my_path´, I select a column and try to overwrite columns_to_select = ["one_column"] df_one_column = df.select(*columns_to_select) df_one_column.write.format("de...

Data Engineering
Reply
  • 568 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Enrique1987
New Contributor III
  • 0 kudos

ok I get the Issue  .option("mergeSchema", "true")  Is usefull to add more columns, but if you want to reduce columns in your target delta.Then you need.option("overwriteSchema", "true") 

  • 0 kudos
Phani1
by • Valued Contributor II
  • 299 Views
  • 1 replies
  • 0 kudos

Metastore Access

Hi team,We are operating on a single-tenant basis with just one metastore. How can other teams use Databricks Unity Catalog without being granted access to our metadata within single tenantRegards,Janga 

Data Engineering
Reply
  • 299 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jacovangelder
Honored Contributor
  • 0 kudos

You can enable/set workspace catalog isolation for each catalog used by your teams. That way, they'll only be able to see their own catalog and corresponding data assets.

  • 0 kudos
giladba
by • New Contributor III
  • 666 Views
  • 4 replies
  • 0 kudos

Databricks API - Create Connection

Hi,Is it possible to use the Databricks API to create a connection to a different Azure Databricks workspace?Thanks

Data Engineering
Reply
  • 666 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
giladba
New Contributor III
  • 0 kudos

My apologies for not being cleared in the first post.Terraform can handle it. The API is probably behind and can't handle the connection to Databricks yet. The best solution would probably be to just move to Unity Catalog...Thanks for your quick repl...

  • 0 kudos
3 More Replies
Chvyaken
by • New Contributor III
  • 1167 Views
  • 5 replies
  • 7 kudos

Resolved! Change Runtime version from 10.4 to 11.3 in ADF

Hello! At the moment, I am considering replacing runtime 10.4 with runtime 11.3 in my ADF. I would like to know how big the differences are between them, and if I can just change the version without worrying about breaking anything? 

Chvyaken_0-1720363212242.png
Data Engineering
Reply
  • 1167 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
Witold
Contributor III
  • 7 kudos

On thing worth mentioning is the DBR Migration Guide. There you'll find all potential changes between the runtimes.

  • 7 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels