- 3947 Views
- 1 replies
- 3 kudos
Resolved! Using DeltaTable.merge() and generating surrogate keys on insert?
I'm using merge to upsert data into a table:DeltaTable.forName(DESTINATION_TABLE).as("target").merge(merge_df.as("source") ,"source.topic = target.topic and source.key = target.key").whenMatched().updateAll().whenNotMatched().insertAll().execute()Id ...
- 3947 Views
- 1 replies
- 3 kudos
- 3 kudos
@Dekova 1) uuid() is non-deterministic meaning that it will give you different result each time you run this function2) Per the documentation "For Databricks Runtime 9.1 and above, MERGE operations support generated columns when you set spark.databri...
- 3 kudos
- 3391 Views
- 3 replies
- 2 kudos
Databricks SQL - Conditional Catalog query
Hi is there a way we can do%sqlselect * from {{ catalog }}.schema.tableWhere `{{ catalog }}` is a template variable extracted/evaluated from either an environment variable, a databricks secret, or somewhere else? (note: not a widget)
- 3391 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @102842 You can use query parameters to perform this - https://docs.databricks.com/sql/user/queries/query-parameters.htmlYou can define the catalog name as a query parameter. You should declare the catalog name parameter as a drop down list, becau...
- 2 kudos
- 1567 Views
- 1 replies
- 0 kudos
Activation Email is not coming up in the email
Activation Email is not showing up in the email. I am not able to start my trial. @Sujitha or @Kaniz - Please help!
- 1567 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sujitha or @Kaniz - Need your help regarding the validate email not reaching inbox / spam folder to activate trial.
- 0 kudos
- 846 Views
- 0 replies
- 0 kudos
Validate email is not showing up in inbox / spam
No email is coming up to validate and start the trial, need help.
- 846 Views
- 0 replies
- 0 kudos
- 5113 Views
- 4 replies
- 0 kudos
Import dbfs file into workspace using Python SDK
Hello,I am looking to replicate the functionality provided by the databricks_cli Python package using the Python SDK. Previously, using the databricks_cli WorkspaceApi object, I could use the import_workspace or import_workspace_dir methods to move a...
- 5113 Views
- 4 replies
- 0 kudos
- 0 kudos
Even, I am looking for a way to bring files present in S3 to Workspace programmatically.
- 0 kudos
- 1352 Views
- 0 replies
- 0 kudos
Big time differences in reading tables
When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know ...
- 1352 Views
- 0 replies
- 0 kudos
- 3915 Views
- 1 replies
- 1 kudos
ENDPOINT_NOT_FOUND error coming for /2.0/clusters/list-zones API on Databricks running over GCP
Hi,I am trying to build the ETL data pipelines on databricks workspace that is running over GCP.For automated cluster creation, when I´m trying to access list availability zones REST API of cluster creation, then it is failing with end point not foun...
- 3915 Views
- 1 replies
- 1 kudos
- 4494 Views
- 1 replies
- 1 kudos
To enroll for featured member interview
What is the procedure to enroll ourselves into feature member interview.
- 4494 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @KVNARK We appreciate your interest in becoming a part of our featured member recognition system. Regrettably, at the moment, enrollment is not possible through an application process. Instead, we identify the top contributors based on their ac...
- 1 kudos
- 3156 Views
- 2 replies
- 4 kudos
Resolved! Is there a plan to support workflow jobs to be stored in a subfolder?
I have many workflow jobs created and they all in a flat list. Is there a way to create (kind of) sub folders that I can category my databricks workflow jobs into it (kind of organizer)...
- 3156 Views
- 2 replies
- 4 kudos
- 4 kudos
@Anonymous thanks for the suggestion. And thanks @Vinay_M_R a lot for answering the question. The solution mentioned is doable but less optimized way to do. Everyone in the team has to follow the same rules especially for shared jobs, and sometimes n...
- 4 kudos
- 3459 Views
- 0 replies
- 0 kudos
terraform jobs depends_on
I am attempting to automate Jobs creation using Databrick Terraform provider. I have a number of task that will "depends_on" each other and am trying to use dynamic content to do this. Each task name is stored in a string array so looping over th...
- 3459 Views
- 0 replies
- 0 kudos
- 2938 Views
- 1 replies
- 0 kudos
When is LakehouseIQ available?
Hi all, saw the lakehouse IQ stuff at the keynote - really excited to learn more and hopefully get access!
- 2938 Views
- 1 replies
- 0 kudos
- 0 kudos
I also want to know.
- 0 kudos
- 809 Views
- 0 replies
- 0 kudos
Init scripts in legacy workspace (pre-E2)
Hello,I've got a legacy workspace (not E2) and I am trying to move my cluster scoped init script to the workspace area (from DBFS). It doesn't seem to be possible to store a shell script in the workspace area (Accepted formats: .dbc, .scala, .py, .sq...
- 809 Views
- 0 replies
- 0 kudos
- 2946 Views
- 2 replies
- 1 kudos
Resolved! Databricks SQL warehouse best practices
How best we can design Databricks SQL warehouse for multiple environments, and multiple data marts, is there any best practices or guidelines?
- 2946 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Phani1 We haven't heard from you since the last response from @Vinay_M_R , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. Also...
- 1 kudos
- 2352 Views
- 2 replies
- 3 kudos
Amazing Women in Data AI session at Data AI Summit 2023
Great first time experience attending the keynotes, sessions, trainings and certifications. It was a great opportunity to connect with like minded individuals and women in Data AI panel and learn about the community.
- 2352 Views
- 2 replies
- 3 kudos
- 956 Views
- 0 replies
- 0 kudos
AWS S3 Bucket Access from Unity Catalog?
Asking for a new logo customer…Let's say my unity catalog is in account A of AWS. The buckets that I need to access are in account B of AWS. The unity catalog is unable to create an external location based of this bucket even though all the necessary...
- 956 Views
- 0 replies
- 0 kudos
