- 5224 Views
- 1 replies
- 0 kudos
Resolved! Diffrence b/w community edition and Free edition Databrick platform
Recently there is a news from DATA +AI summit mentioning Free edition of Databricks platform, how its different from community edition?Follow up question.Is there any limitation on compute resources.Will it support other cloud services provider apart...
- 5224 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Pavankumar,Regarding your questions:The difference between free edition and CE: "Free Edition has been designed and extended to include full access to the Data Intelligence Platform. It provides an easy-to-use environment where you can build AI...
- 0 kudos
- 1865 Views
- 6 replies
- 0 kudos
Experiencing sorting problems with bigint columns
Experiencing sorting problems with bigint columns across tables tested. Example:In the table the projectid (bigint as per schema - which is correct) yet when sorted in SQL is sorting as per a string value 1000903 is returned as the minimum projectid ...
- 1865 Views
- 6 replies
- 0 kudos
- 0 kudos
Try running explain command what's happening in the background, also if table is small try to write in another table and check
- 0 kudos
- 3069 Views
- 2 replies
- 1 kudos
Resolved! Batch Python UDFs in Unity Catalog and Spark SQL
Hello datanauts 六,I'm encountering a conceptual challenge regarding Batch Python UDFs within Spark SQL in Databricks. My primary question is: can Batch Python UDFs be used directly via Spark SQL? As a Databricks beginner, I'm seeking to understand ...
- 3069 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @stefan-vulpe Looking at your code and the behavior you're describing, I can identify the core issue and provide some insights about Batch Python UDFs in Databricks.The Core ProblemThe issue you're encountering is related to session isolation and ...
- 1 kudos
- 1348 Views
- 1 replies
- 1 kudos
Is it possible to reprocess only a portion of a streaming table data using DLT?
Hi all,Currently I have a standard notebook that it takes 2 dates as parameters, start date and end date it goes to the source then it pull only that portion of data then on target table deletes if necessary (if data within those ranges exists) the u...
- 1348 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @MauricioS Yes, you can achieve similar reprocessing functionality with DLT streaming tables,but it requires a different approach than your current batch process. Here are the main strategies:1. CDC Pattern with Tombstone RecordsThe most common ap...
- 1 kudos
- 3063 Views
- 1 replies
- 0 kudos
Resolved! Connection with Azure service principal
I have configured a data bricks connection with Azure using a service principal who is configured in both Azure system and data bricks. I am getting an issue calling this API since early June. https://$URI$/api/2.0/secrets/scopes/list Headers = @{ ...
- 3063 Views
- 1 replies
- 0 kudos
- 0 kudos
For anyone interested, this particular case was due migrating Azure Powershell to a new version. This is the link to follow just in case https://learn.microsoft.com/en-us/powershell/azure/migrate-az-14.0.0?view=azps-14.1.0
- 0 kudos
- 1907 Views
- 2 replies
- 1 kudos
Resolved! Specifing Output mode and Path when using For Each Batch
Since .foreachBatch() is "hijacking" the stream and executing arbitrary code in it, do I need to specify Output mode and Path:(df.writeStream .format("delta") .trigger(availableNow = True) .option("checkpointLocation", "check_point_location") .forea...
- 1907 Views
- 2 replies
- 1 kudos
- 4907 Views
- 1 replies
- 0 kudos
Parquet file for delta streaming live table with pipeline
I am having an issue with parquet files: I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). I am using a Delta streaming live table with a pipeline. If I r...
- 4907 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ClarkElliott Good day!! Cause Databricks Runtime versions 11.3 LTS and above do not support the TIMESTAMP_NANOS type in open source Apache Spark and Databricks Runtime. If a Parquet file contains fields with the TIMESTAMP_NANOS type, attempts to...
- 0 kudos
- 6337 Views
- 1 replies
- 0 kudos
Where are default temporary checkpoint locations created for streaming queries with display command?
Hello!I created a streaming query using Auto Loader to read data from S3 and used display command to see if the query was working. Initially, cloudFiles.includeExistingFiles was set to True, but since we have data in Glacier that needs to be retrieve...
- 6337 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shavya Good day!! When you do not specify a checkpointLocation in a streaming query in Databricks. It uses a temporary system directory such as: dbfs:/local_disk0/tmp/temporary-<random_uuid> To remove the temporary checkpoint, please ...
- 0 kudos
- 1689 Views
- 1 replies
- 0 kudos
Streaming query error - [STREAMING_STATEFUL_OPERATOR_NOT_MATCH_IN_STATE_METADATA]
[STREAM_FAILED] Query [id = 6a821fbc-490b-4ad8-891d-e4cacc2af1d6, runId = e055fede-8012-4369-861b-47183999e91d] terminated with exception: [STREAMING_STATEFUL_OPERATOR_NOT_MATCH_IN_STATE_METADATA] Streaming stateful operator name does not match with ...
- 1689 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @lprevost Good day!! Please find below my analysis for your issue. Error: [STREAM_FAILED] Query [id = 6a821fbc-490b-4ad8-891d-e4cacc2af1d6, runId = e055fede-8012-4369-861b-47183999e91d] terminated with exception: [STREAMING_STATEFUL_OPERATOR_NOT...
- 0 kudos
- 2847 Views
- 1 replies
- 4 kudos
Resolved! Handling partition overwrite in Liquid Clustering
Hello,Currently we have delta tables in TBs partitioned by year, month, day. We perform dynamic partition overwrite using partitionOverwriteMode as dynamic to handle rerun/corrections.With liquid clustering, since explicit partitions are not require...
- 2847 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Klusener Good day!!Dynamic partition overwrites only supports selective overwrites for partitioned columns, not for liquid clustering or regular columns.If you know the exact predicates, use replaceWhere. Note: This is not possible without knowin...
- 4 kudos
- 2315 Views
- 1 replies
- 1 kudos
Resolved! Unable to add primary key constraint to nullable identity column
While we can in fact define a primary key during table creation for an identity column that's nullable (i.e., not constrained using NOT NULL), it's not possible to add such a primary key constraint after the table has been created.We get an error mes...
- 2315 Views
- 1 replies
- 1 kudos
- 1 kudos
Creating a table with a nullable IDENTITY column and defining the primary key at creation time works.The database quietly interprets the column as NOT NULL for the purposes of the primary key, even though it's technically defined as nullable (i.e., n...
- 1 kudos
- 741 Views
- 1 replies
- 0 kudos
Databricks on Azure
We are setting up data hubs that interacts with each other for Gen AI use cases. I want to prove that catalog sharing works across azure subscriptions if all UCs are mapped to the same metastore. Can you point me to the right documentation?
- 741 Views
- 1 replies
- 0 kudos
- 0 kudos
I believe you need to follow below steps.1. Deploying a metastore in one region.2. Linking each workspace (from different Azure subscriptions but same tenant and region) to it.3. Then validating that metadata objects like catalogs, schemas, and table...
- 0 kudos
- 2168 Views
- 2 replies
- 0 kudos
Getting concurrent issue on delta table using liquid clustering
In our project, we are testing liquid clustering using a test table called status_update, where we need to update the status for different market IDs. We are attempting to update the status_update table in parallel using the UPDATE command.ALTER TABL...
- 2168 Views
- 2 replies
- 0 kudos
- 0 kudos
@Walter_C We are using Liquid Clustering as our first strategy. Our Databricks Runtime is 13.3, and we have a table named status_update containing approximately 30 market IDs, each with a single record. In our pipeline, if any market fails, we need t...
- 0 kudos
- 3283 Views
- 6 replies
- 0 kudos
Default catalog created wrong on my workspace
We have provisioned a new databricks account and workspace on premium plan. When built out workspace using terraform, we expected to see a default catalog matching workspace name as per this documentation. However I dont see it. All I see are the 3 c...
- 3283 Views
- 6 replies
- 0 kudos
- 0 kudos
Hello,Meanwhile I try to get help on Databricks default catalog behavior, I found this topic.If I can give my advice here, one reason I see for the behavior of @amarnadh-gadde is that you deployed your new workspace in a region where there is already...
- 0 kudos
- 5039 Views
- 6 replies
- 7 kudos
Resolved! autoloader strategy write ( APPEND, MERGE, UPDATE, COMPLETE, OVERWRITE)
Hello Guys, I want to know if operations like overwrite, merge, update in static write its the same when we using autoloader. I'm so confusing about the behavior of mode like ( complete, update and append). After that, i want to know what its the co...
- 5039 Views
- 6 replies
- 7 kudos
- 7 kudos
Thanks for discussion. I have a tiny suggestion. Based on my experience working with streaming loads, I often find the checkpoint location hard enough to actually check the offset information or delete that directory for fresh load of data. Hence I h...
- 7 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
