cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RobsonNLPT
by Contributor III
  • 3712 Views
  • 2 replies
  • 0 kudos

Delta Identity latest value after insert

Hi all.I would like to know if databricks has some feature to retrieve the latest identity column value (always generated) after insert or upserts operations? (dataframe apis and sql)Database engines as Azure SQL  and Oracle have feature that enable ...

  • 3712 Views
  • 2 replies
  • 0 kudos
Latest Reply
tapash-db
Databricks Employee
  • 0 kudos

Hi, You can always query "SELECT MAX(identity_column) FROM your_table_name" and see the latest value of the identity column. However, there are no direct functions available to give the latest identity column value.

  • 0 kudos
1 More Replies
eballinger
by Contributor
  • 1746 Views
  • 2 replies
  • 0 kudos

Looking for ways to speed up DLT testing

Hi Guys,I am new to this community. I am guessing we have a typical setup (DLT tables, 3 layers - bronze, silver and gold) and while it works fine in our development environment I have always looked for ways to speed things up for testers. For exampl...

  • 1746 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

There isn't a direct way to achieve this within the current DLT framework. When a DLT table is undeclared, it is designed to be removed from the pipeline, which includes the underlying data. However, there are a few strategies you can consider to spe...

  • 0 kudos
1 More Replies
darkolexis
by New Contributor
  • 3390 Views
  • 2 replies
  • 1 kudos

Service Principal types in Azure Databricks

In Azure Databricks, we can create two types of Service Principals, namely:1. Databricks Managed SP2. Microsoft Entra ID Managed SP What is the difference between two, other than one being specific to single workspace, and another being usable from m...

  • 3390 Views
  • 2 replies
  • 1 kudos
Latest Reply
arunprakash1986
New Contributor II
  • 1 kudos

So, what use would it be in a situation where I have a Docker image that runs as a job using Databricks Compute. Here the Job has "Run As" which is set to a service principal, say "svc1" which is a databricks managed service principal. I believe that...

  • 1 kudos
1 More Replies
Cosimo_F_
by Contributor
  • 2301 Views
  • 4 replies
  • 0 kudos

Autoloader schema inference

Hello,is it possible to turn off schema inference with AutoLoader? Thank you,Cosimo

  • 2301 Views
  • 4 replies
  • 0 kudos
Latest Reply
shivagarg
New Contributor II
  • 0 kudos

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#language-pythonyou can enforce the schema or use the "cloudFiles.schemaHints"  to override the Inference. df = spark.readStream.format("cloudFiles") \ .option("...

  • 0 kudos
3 More Replies
aranjan99
by Contributor
  • 9899 Views
  • 4 replies
  • 1 kudos

system.access.table_lineage table missing data

I am using the system.access.table_lineage table  to figure out the tables accessed by sql queries and the corresponding SQL queries. However I am noticing this table missing data or values very often.For eg for sql queries executed by our DBT jobs, ...

  • 9899 Views
  • 4 replies
  • 1 kudos
Latest Reply
goldenmountain
New Contributor II
  • 1 kudos

@aranjan99 did you ever get an answer or conclusion to the limitations of Unity Catalog in regards to tracking access via SQL?

  • 1 kudos
3 More Replies
Direo
by Contributor II
  • 2991 Views
  • 2 replies
  • 0 kudos

Migrating to Unity Catalog: Read-Only Connections to SQL Server and Snowflake

We are in the process of migrating to Unity Catalog, establishing connections to SQL Server and Snowflake, and creating foreign catalogs that mirror your SQL Server and Snowflake databases. This allows us to leverage Unity Catalog’s query syntax and ...

Data Engineering
UnityCatalog SQLServer Snowflake Governance Permissions
  • 2991 Views
  • 2 replies
  • 0 kudos
Latest Reply
goldenmountain
New Contributor II
  • 0 kudos

I’m also trying to figure out if this is a limitation in Unity Catalog. I recently used a JDBC URL to write data to an Amazon Aurora PostgreSQL database, but noticed that no entries appeared in the `system.access.table_lineage` table. Has anyone else...

  • 0 kudos
1 More Replies
Tamizh035
by New Contributor II
  • 3005 Views
  • 3 replies
  • 1 kudos

[INSUFFICIENT_PERMISSIONS] Insufficient privileges:

While reading csv file using spark and listing the files under a folder using data bricks utils, I am getting below error:[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501File <comma...

  • 3005 Views
  • 3 replies
  • 1 kudos
Latest Reply
mpalacio
New Contributor II
  • 1 kudos

I have the same issue, did you manage to solve it?I have the Databricks extension well configured and my role has enough permissions. Everything used to work propertly but now when I run my notebooks is giving me this issue and 'no module named dbrun...

  • 1 kudos
2 More Replies
brendanc19
by New Contributor III
  • 5926 Views
  • 6 replies
  • 2 kudos

Resolved! Does cancelling a job run rollback any actions performed by query plan?

If I were to stop a rather large job run, say half way thru execution, will any actions performed on our Delta tables persist or will they be rolled back?Are there any other risks that I need to be aware of in terms of cancelling a job run half way t...

  • 5926 Views
  • 6 replies
  • 2 kudos
Latest Reply
fabian_r
New Contributor II
  • 2 kudos

Hi, is there any way to ensure transaction control in delta protocol in 2024 across tables for failing jobs?

  • 2 kudos
5 More Replies
techg
by New Contributor II
  • 1410 Views
  • 4 replies
  • 1 kudos

Missing selection for Parameter error

Hi All,I have created three parameters in an SQL query in Databricks. If no value is entered for a parameter, I would like the query to retrieve all values for that particular column. Currently, I'm getting an error message: "Missing selection for Pa...

  • 1410 Views
  • 4 replies
  • 1 kudos
Latest Reply
techg
New Contributor II
  • 1 kudos

I'm creating this query with parameters in SQL Editor in Databricks and added it to the SQL Dashboard.Do we need to create Widget while creating parameters in SQL Editor? When i tried creating widget in SQL editor, Im getting syntax error near Widget...

  • 1 kudos
3 More Replies
Gusman
by New Contributor II
  • 992 Views
  • 2 replies
  • 1 kudos

Resolved! Natural language queries through REST API?

Natural language queries provided by Genie are really powerful and a compeling tool.Is there any way to execute these natural language queries through the REST API to integrate them into in-house applications?

  • 992 Views
  • 2 replies
  • 1 kudos
Latest Reply
stacey45
New Contributor II
  • 1 kudos

@Gusman wrote:Natural language queries provided by Genie are really powerful and a compeling tool.Is there any way to execute these natural language queries through the REST API to integrate them into in-house applications?While there's no direct RES...

  • 1 kudos
1 More Replies
Clara
by New Contributor
  • 512 Views
  • 1 replies
  • 1 kudos

Retrieve data older than the one year window : system.access.table_lineage

Hello,I am currently using table_lineage from system.access.table_lineage. It is a great feature but I am experiencing missing data. After some search I have seen that "Because lineage is computed on a one-year rolling window, lineage collected more ...

  • 512 Views
  • 1 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @Clara ,I don't think so. But you can build history such history tables by yourself. Desing ETL process that will extract data from system tables and store them in your own data tables. 

  • 1 kudos
sboxi
by New Contributor II
  • 669 Views
  • 2 replies
  • 1 kudos

Can we create Materialized view n exsting view and table?

Dear All,Is it possible to create Materialized view on view and table (Joining view and table)?I suspect it is not possible. Please suggest.Also please provide best way to schedule the refresh of Materialized view. Regards,Surya 

  • 669 Views
  • 2 replies
  • 1 kudos
Latest Reply
sboxi
New Contributor II
  • 1 kudos

Thanks @Alberto_Umana . I will try that.

  • 1 kudos
1 More Replies
TimW
by New Contributor
  • 5755 Views
  • 4 replies
  • 1 kudos

Resolved! Help - Can't create table from tutorial. Is my setup wrong?

Trying out databricks for the first time and followed the Get Started steps. I managed to successfully create a cluster and ran the simple sql tutorial to query data from a notebook. However, got the following error:Query:DROP TABLE IF EXISTS diamond...

  • 5755 Views
  • 4 replies
  • 1 kudos
Latest Reply
patwilliams
New Contributor III
  • 1 kudos

It seems as though you're doing great with your Databricks arrangement, however this sort of mistake could be connected with a couple of expected issues. In light of the subtleties you've shared, here are a few things you should check:Group Setup: Gu...

  • 1 kudos
3 More Replies
ashraf1395
by Honored Contributor
  • 2041 Views
  • 2 replies
  • 2 kudos

Resolved! applying column tags

Can anyone tell me the correct syntax for applying a column tag to a specific tableThese are what I tried   ALTER TABLE accounts_and_customer.bronze.BB1123_loans ALTER/CHANGE COLUMN loan_number SET TAGS ('classification' = 'confidential')<p>I got thi...

  • 2041 Views
  • 2 replies
  • 2 kudos
Latest Reply
ashraf1395
Honored Contributor
  • 2 kudos

Hi there @Takuya-Omi ,I agree. The syntax was correct. I was facing some completely different problems with schemas and I solved it. Thanks though. Or I would have spent hours banging my head to find the reason for error.

  • 2 kudos
1 More Replies
JothyGanesan
by New Contributor III
  • 682 Views
  • 1 replies
  • 1 kudos

CDF table partition - Real time Data

Hi team,We are currently working in loading CDF table using data events from Kafka. The table is going to hold data across geographies. When we tried partitioning it is slowing down the ingestion time. But without partition the downstream application...

  • 682 Views
  • 1 replies
  • 1 kudos
Latest Reply
ozaaditya
Contributor
  • 1 kudos

 1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels