- 3712 Views
- 2 replies
- 0 kudos
Delta Identity latest value after insert
Hi all.I would like to know if databricks has some feature to retrieve the latest identity column value (always generated) after insert or upserts operations? (dataframe apis and sql)Database engines as Azure SQL and Oracle have feature that enable ...
- 3712 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, You can always query "SELECT MAX(identity_column) FROM your_table_name" and see the latest value of the identity column. However, there are no direct functions available to give the latest identity column value.
- 0 kudos
- 1746 Views
- 2 replies
- 0 kudos
Looking for ways to speed up DLT testing
Hi Guys,I am new to this community. I am guessing we have a typical setup (DLT tables, 3 layers - bronze, silver and gold) and while it works fine in our development environment I have always looked for ways to speed things up for testers. For exampl...
- 1746 Views
- 2 replies
- 0 kudos
- 0 kudos
There isn't a direct way to achieve this within the current DLT framework. When a DLT table is undeclared, it is designed to be removed from the pipeline, which includes the underlying data. However, there are a few strategies you can consider to spe...
- 0 kudos
- 3390 Views
- 2 replies
- 1 kudos
Service Principal types in Azure Databricks
In Azure Databricks, we can create two types of Service Principals, namely:1. Databricks Managed SP2. Microsoft Entra ID Managed SP What is the difference between two, other than one being specific to single workspace, and another being usable from m...
- 3390 Views
- 2 replies
- 1 kudos
- 1 kudos
So, what use would it be in a situation where I have a Docker image that runs as a job using Databricks Compute. Here the Job has "Run As" which is set to a service principal, say "svc1" which is a databricks managed service principal. I believe that...
- 1 kudos
- 2301 Views
- 4 replies
- 0 kudos
Autoloader schema inference
Hello,is it possible to turn off schema inference with AutoLoader? Thank you,Cosimo
- 2301 Views
- 4 replies
- 0 kudos
- 0 kudos
https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#language-pythonyou can enforce the schema or use the "cloudFiles.schemaHints" to override the Inference. df = spark.readStream.format("cloudFiles") \ .option("...
- 0 kudos
- 9899 Views
- 4 replies
- 1 kudos
system.access.table_lineage table missing data
I am using the system.access.table_lineage table to figure out the tables accessed by sql queries and the corresponding SQL queries. However I am noticing this table missing data or values very often.For eg for sql queries executed by our DBT jobs, ...
- 9899 Views
- 4 replies
- 1 kudos
- 1 kudos
@aranjan99 did you ever get an answer or conclusion to the limitations of Unity Catalog in regards to tracking access via SQL?
- 1 kudos
- 2991 Views
- 2 replies
- 0 kudos
Migrating to Unity Catalog: Read-Only Connections to SQL Server and Snowflake
We are in the process of migrating to Unity Catalog, establishing connections to SQL Server and Snowflake, and creating foreign catalogs that mirror your SQL Server and Snowflake databases. This allows us to leverage Unity Catalog’s query syntax and ...
- 2991 Views
- 2 replies
- 0 kudos
- 0 kudos
I’m also trying to figure out if this is a limitation in Unity Catalog. I recently used a JDBC URL to write data to an Amazon Aurora PostgreSQL database, but noticed that no entries appeared in the `system.access.table_lineage` table. Has anyone else...
- 0 kudos
- 3005 Views
- 3 replies
- 1 kudos
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
While reading csv file using spark and listing the files under a folder using data bricks utils, I am getting below error:[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501File <comma...
- 3005 Views
- 3 replies
- 1 kudos
- 1 kudos
I have the same issue, did you manage to solve it?I have the Databricks extension well configured and my role has enough permissions. Everything used to work propertly but now when I run my notebooks is giving me this issue and 'no module named dbrun...
- 1 kudos
- 5926 Views
- 6 replies
- 2 kudos
Resolved! Does cancelling a job run rollback any actions performed by query plan?
If I were to stop a rather large job run, say half way thru execution, will any actions performed on our Delta tables persist or will they be rolled back?Are there any other risks that I need to be aware of in terms of cancelling a job run half way t...
- 5926 Views
- 6 replies
- 2 kudos
- 2 kudos
Hi, is there any way to ensure transaction control in delta protocol in 2024 across tables for failing jobs?
- 2 kudos
- 1410 Views
- 4 replies
- 1 kudos
Missing selection for Parameter error
Hi All,I have created three parameters in an SQL query in Databricks. If no value is entered for a parameter, I would like the query to retrieve all values for that particular column. Currently, I'm getting an error message: "Missing selection for Pa...
- 1410 Views
- 4 replies
- 1 kudos
- 1 kudos
I'm creating this query with parameters in SQL Editor in Databricks and added it to the SQL Dashboard.Do we need to create Widget while creating parameters in SQL Editor? When i tried creating widget in SQL editor, Im getting syntax error near Widget...
- 1 kudos
- 992 Views
- 2 replies
- 1 kudos
Resolved! Natural language queries through REST API?
Natural language queries provided by Genie are really powerful and a compeling tool.Is there any way to execute these natural language queries through the REST API to integrate them into in-house applications?
- 992 Views
- 2 replies
- 1 kudos
- 1 kudos
@Gusman wrote:Natural language queries provided by Genie are really powerful and a compeling tool.Is there any way to execute these natural language queries through the REST API to integrate them into in-house applications?While there's no direct RES...
- 1 kudos
- 512 Views
- 1 replies
- 1 kudos
Retrieve data older than the one year window : system.access.table_lineage
Hello,I am currently using table_lineage from system.access.table_lineage. It is a great feature but I am experiencing missing data. After some search I have seen that "Because lineage is computed on a one-year rolling window, lineage collected more ...
- 512 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Clara ,I don't think so. But you can build history such history tables by yourself. Desing ETL process that will extract data from system tables and store them in your own data tables.
- 1 kudos
- 669 Views
- 2 replies
- 1 kudos
Can we create Materialized view n exsting view and table?
Dear All,Is it possible to create Materialized view on view and table (Joining view and table)?I suspect it is not possible. Please suggest.Also please provide best way to schedule the refresh of Materialized view. Regards,Surya
- 669 Views
- 2 replies
- 1 kudos
- 5755 Views
- 4 replies
- 1 kudos
Resolved! Help - Can't create table from tutorial. Is my setup wrong?
Trying out databricks for the first time and followed the Get Started steps. I managed to successfully create a cluster and ran the simple sql tutorial to query data from a notebook. However, got the following error:Query:DROP TABLE IF EXISTS diamond...
- 5755 Views
- 4 replies
- 1 kudos
- 1 kudos
It seems as though you're doing great with your Databricks arrangement, however this sort of mistake could be connected with a couple of expected issues. In light of the subtleties you've shared, here are a few things you should check:Group Setup: Gu...
- 1 kudos
- 2041 Views
- 2 replies
- 2 kudos
Resolved! applying column tags
Can anyone tell me the correct syntax for applying a column tag to a specific tableThese are what I tried ALTER TABLE accounts_and_customer.bronze.BB1123_loans ALTER/CHANGE COLUMN loan_number SET TAGS ('classification' = 'confidential')<p>I got thi...
- 2041 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Takuya-Omi ,I agree. The syntax was correct. I was facing some completely different problems with schemas and I solved it. Thanks though. Or I would have spent hours banging my head to find the reason for error.
- 2 kudos
- 682 Views
- 1 replies
- 1 kudos
CDF table partition - Real time Data
Hi team,We are currently working in loading CDF table using data events from Kafka. The table is going to hold data across geographies. When we tried partitioning it is slowing down the ingestion time. But without partition the downstream application...
- 682 Views
- 1 replies
- 1 kudos
- 1 kudos
1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
