- 757 Views
- 1 replies
- 1 kudos
Custom stream-stream join using transformWithState - expanded example
Hi all,Just wondering if anyone has more information / an expanded example of the "Custom stream-stream join using transformWithState" example on Stateful applications page: Example stateful applications | Databricks on AWSI'm looking to implement so...
- 757 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @alexbush-mas Good day!! Unioning the streams is the standard method for feeding multiple input streams into a single transformWithStateInPandas operation for custom stream-stream joins, so your intuition is correct. After grouping the input by...
- 1 kudos
- 2947 Views
- 3 replies
- 4 kudos
Resolved! Permission denied on schema evolution view
Hey Databricks community,We are registering views in Databricks with schema evolution, however these views fail when a user (other than the owner) is the first to query the view after the schema change.PERMISSION_DENIED: User is not an owner of Table...
- 2947 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Sinkrad, There is an internal case about this: ES-1260035, and a fix is going to be out for this quarter. The behavior where only the view owner can update the view definition after a schema change is not intended. It is expected that the view sh...
- 4 kudos
- 1363 Views
- 7 replies
- 5 kudos
Resolved! Issue in renaming the table
We have tried to rename a tableEx: alter rename tablea to tablebwhile we checked the table b properties we are seeing the tableb location is being pointed to tablea location.
- 1363 Views
- 7 replies
- 5 kudos
- 5 kudos
Hello @Rajasaiharish , The above is expected by design.ALTER TABLE RENAME TO updates only the table name in the catalog.The underlying Delta log + data files remain in the original storage location Try setting the location to a new location ALTER ...
- 5 kudos
- 1663 Views
- 3 replies
- 4 kudos
Job Owner Changes in Asset Bundle
When I deploy a fresh job, I'm set as owner, which makes sense because I'm creating it.Then at some point, the owner switches to a service principal. This is strange. I believe it happens when a scheduled job triggers.Can job owners change?
- 1663 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @T0M, how are you doing?You’re right about the statuses at the level above the job; however, my reference to IS_OWNER comes from the job-level statuses, as shown below:IS_OWNER exists and is valid for resource-level permissions (e.g., inside resou...
- 4 kudos
- 4554 Views
- 3 replies
- 5 kudos
[UNRESOLVED_ROUTINE] Cannot resolve function `date_format`
Hi all,We are getting the following error log in a Workflow:AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `date_format` on search path [`system`.`builtin`, `system`.`session`]. SQLSTATE: 42883The Workflow consists in different noteb...
- 4554 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello @Jaclaglez13 I am really interested in your ticket because we have the same kind of error.We loose the SQL built-in function when we run the notebook a second time without restarting our "all purpose compute".There is a point I would like to cl...
- 5 kudos
- 1177 Views
- 3 replies
- 1 kudos
Resolved! Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns
Hi,I'm facing a weird error when trying to incrementally merge into a table with a masked column.Table is external, but catalog is managed. Using Unity Catalog.Error is:[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: Path-based access t...
- 1177 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @thedatanerd , Thanks for the reply. Yes, you can access that by DeltaTable.forName if you know the table name or use MERGE into using SQL. If your question is answered,could you please accept this as a solution, as it would be useful to others...
- 1 kudos
- 1076 Views
- 5 replies
- 3 kudos
Resolved! unable to access databricks pages
Hello databricks team I am unable to access this URL, but I can see some discription available. I tried to search for this to solve a case but unable to open it. https://docs.databricks.com/_extras/notebooks/source/kb/clusters/event-log-replay.html h...
- 1076 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @Khaja_Zaffer! The notebook is no longer available at that URL. You can use this valid link: https://docs.databricks.com/aws/en/notebooks/source/kb/clusters/event-log-replay.html
- 3 kudos
- 1441 Views
- 1 replies
- 1 kudos
Resolved! mongodb spark
Hello Guys,Someone know a technique to perform write for a delta table to Mongo using connection mongodb org.mongodb.spark:mongo-spark-connector_2.12:10.5.0I have 1 bilion records to write Thanks
- 1441 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @seefoods ,Yes, it's well described at mongo db connector documentation page. To write data to MongoDB you need call the write function on your DataFrame object. This function returns a DataFrameWriter object, which you can use to specify the form...
- 1 kudos
- 416 Views
- 2 replies
- 2 kudos
Dropdown parameter in new SQL editor
New SQL editor UI doesn't allow adding a list of parameter values. Instead values has to be added one by one and that's just stupid. Also it doesn't allow using query results as values. In notebooks thats of course possible.
- 416 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @JuhaLeppanen ,If you'd like to see such features in SQL Editor please let know product team using Databricks Ideas portal.Feedback | Databricks
- 2 kudos
- 1681 Views
- 14 replies
- 11 kudos
Resolved! Spark UI Replay Notebook
I need to load the Spark UI events of a long-running job.I found this Databricks knowledge base article on the topic:https://kb.databricks.com/clusters/replay-cluster-spark-eventsHowever, the referenced notebook is no longer accessible.Does anyone st...
- 1681 Views
- 14 replies
- 11 kudos
- 11 kudos
Hi @der ,Here's the content of Event Log Replay notebook. You can use it to achieve your goals %scala dbutils.widgets.removeAll() dbutils.widgets.text("event_log_path", "", "event_log_path") val eventLogPath = dbutils.widgets.get("event_log_path"...
- 11 kudos
- 7500 Views
- 13 replies
- 1 kudos
Resolved! Limitations for Unity Catalog on single user access mode clusters
Hello! According to Databricks documentation on azure :"On Databricks Runtime 15.3 and below, fine-grained access control on single user compute is not supported. Specifically:You cannot access a table that has a row filter or column mask.You cannot ...
- 7500 Views
- 13 replies
- 1 kudos
- 1 kudos
@arzb @Nes_Hdr Command support for FGAC on dedicated compute was added in DBR 17.1: https://docs.databricks.com/aws/en/compute/single-user-fgac#support-for-ddl-show-describe-and-other-commands
- 1 kudos
- 20262 Views
- 3 replies
- 8 kudos
Resolved! How do I resolve problems when deploying a workspace with AWS Quickstart cloud formation template?
I am unable to deploy a workspace on AWS using Quickstart from my account console.Short description-You might receive one of the following common errors users face:Wrong credentialsElastic IP and VPC limit reachedRegion unavailableResolution-Wrong cr...
- 20262 Views
- 3 replies
- 8 kudos
- 8 kudos
CreateStorageConfiguraitonI encounter this issue, but the stack template is different from this example, and I dont have a password for my account.Are the solutions here still relevant?
- 8 kudos
- 882 Views
- 2 replies
- 3 kudos
Resolved! Serverless Workspace Observability
I’m setting up observability for a Databricks serverless workspace on AWS and need some guidance.I know we can configure audit logs for S3 delivery, but I’m unsure if that alone is sufficient.For a complete observability setup especially when integra...
- 882 Views
- 2 replies
- 3 kudos
- 567 Views
- 1 replies
- 3 kudos
Job Run Failed - "Cluster became unreachable during run" with Cause: "requirement failed: Execution
I'm encountering a failure while running a job in Databricks. The run fails with the following error message:Cluster became unreachable during run Cause: requirement failed: Execution is doneDetails:Runtime version: 15.4 LTS (includes Apache Spark 3....
- 567 Views
- 1 replies
- 3 kudos
- 3 kudos
Hello @apurvasawant I'm sorry you are seeing this behavior while using Jobs. Definitely, these messages don't help much. When this happens, I suggest taking a step back and reviewing the configuration of your Job and some troubleshooting: What is th...
- 3 kudos
- 602 Views
- 3 replies
- 1 kudos
GIT automate DEV Databricks instance to PROD instance
Hello,Would like to use GIT to automate the process of syncing between a DEV Databricks instance and a PROD Databricks instance. Something like :On GIT console pull changes/sync with DEV Databricks.Have some kind of approval process in GIT (like some...
- 602 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @adrianhernandez ,This can be easily achieved using Databricks Assets Bundles in combination with Azure Devops Pipeline (or github actions).So, typical CI/CD workflows looks something like this:Store: Store your Databricks code and notebooks in a ...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 1050 | |
| 791 | |
| 553 | |
| 349 |
