- 2009 Views
- 1 replies
- 2 kudos
Is Delta Live Tables planned to be open source in the future?
Is Delta Live Tables planned to be open source in the future?
- 2009 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello there @G Z​ I would say "we have a history of open sourcing our biggest innovations but there's no concrete timeline for dlt. It's built on the open APIs of spark and delta, so the most important parts (your transformation logic and you data) ...
- 2 kudos
- 4889 Views
- 4 replies
- 4 kudos
Resolved! Databricks - Workflow- Jobs- Script to automate
Hi - I have created a Databricks job - under Workflow - its running fine without any issues . I would like to promote this job to other workspaces using a script.Is there a way to script the job definition and deploy it across multiple workspaces .I ...
- 4889 Views
- 4 replies
- 4 kudos
- 3186 Views
- 2 replies
- 0 kudos
Cross region Databricks to SQL Connection
We are trying to connect Azure Databricks Cluster to Azure SQL database but the firewalls at SQL level is causing an issue.Whitelisting dbks subnet is not an option here as both the resources are in two different azure regions. Is there a secure way ...
- 3186 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Timir Ranjan​,Have you tried looking into private endpoints? This allows you to expose your Azure SQL database from the Azure backbone and is cross-regional supported.https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overviewP...
- 0 kudos
- 12758 Views
- 10 replies
- 0 kudos
Resolved! Large MERGE Statements - 500+ lines of code!
I'm new to databricks. (Not new to DB's - 10+ year DB Developer).How do you generate a MERGE statement in DataBricks? Trying to manually maintain a 500+ or 1000+ lines in a MERGE statement doesn't make much sense? Working with Large Tables of between...
- 12758 Views
- 10 replies
- 0 kudos
- 0 kudos
In my opinion, when possible MERGE statement should be on the primary key. If not possible you can create your own unique key (by concatenate some fields and eventually hashing them) and then use it in merge logic.
- 0 kudos
- 5643 Views
- 5 replies
- 7 kudos
- 5643 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @KVNARK .​ Could you please send the query that you are executing, that will help me to debug the error.
- 7 kudos
- 6852 Views
- 3 replies
- 6 kudos
Why is my lineage extraction not showing up in the Unity Catalog
Im trying to get the lineage graph to work in Unity catalog, however nothing seems to appear even though I followed the docs. I did the following steps1. Created a Unity metastore and attached the workspace to that metastore.2. Created a Single user ...
- 6852 Views
- 3 replies
- 6 kudos
- 6 kudos
@Stephan Smit​ We finally got a solution from level 3 support (Databricks support).You may check your firewall logs.On our side, we had to open communication to "Event Hub endpoint".The destination depends on your workspace region: Azure Databricks r...
- 6 kudos
- 2434 Views
- 1 replies
- 0 kudos
Monitoring
Are there any event streams that are or could be exposed in AWS (such as Cloudwatch Eventbridge events or SNS messages? In particular I'm interested in events that detail jobs being run. The use case here would be for monitoring jobs from our web app...
- 2434 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, there are several event streams in AWS that can be used to monitor jobs being run. Your Texas BenefitsCloudWatch Events: This service allows you to set up rules to automatically trigger actions in response to specific events in other AWS service...
- 0 kudos
- 27014 Views
- 19 replies
- 10 kudos
Resolved! Correlated column exception in SQL UDF when using UDF parameters.
EnvironmentAzure Databricks 10.1, including Spark 3.2.0ScenarioI want to retrieve the average of a series of values between two timestamps, using a SQL UDF.The average is obviously just an example. In a real scenario, I would like to hide some additi...
- 27014 Views
- 19 replies
- 10 kudos
- 10 kudos
Thanks for your suggestion. The fact that I want to do this in SparkSQL is because there is no underlying SQLServer.
- 10 kudos
- 7170 Views
- 5 replies
- 0 kudos
Resolved! Can Unity catalog grant the access to a file inside azure datalake storage?
Hi databricks community,I have searched quite a while through the internet but did not find an answer. If I have configured the azure datalake connection in Unity data catalog, is it possible to grant the access to users for a specific file or a fold...
- 7170 Views
- 5 replies
- 0 kudos
- 0 kudos
As @werners said service principal needs to have access to the file level.In the unity catalog, you can use "READ FILES"/"WRITE FILES" permission to give someone the possibility of reading files from the storage level (but through databricks).
- 0 kudos
- 14335 Views
- 8 replies
- 10 kudos
Resolved! Dataframe rows missing after write_to_delta and read_from_delta
Hi, i am trying to load mongo into s3 using pyspark 3.1.1 by reading them into a parquet. My code snippets are like:df = spark \ .read \ .format("mongo") \ .options(**read_options) \ .load(schema=schema)df = df.coalesce(64)write_df_to_del...
- 14335 Views
- 8 replies
- 10 kudos
- 10 kudos
So i think i have solved the mystery here it was to do with the retention config. By setting the retentionEnabled to True and rention hours being 0, we somewhat loses a few rows in the first file as they were mistaken as files from last session and ...
- 10 kudos
- 1672 Views
- 1 replies
- 0 kudos
I am not able to login with my credentials. This is happening with me again and again.i have created different account then also,i am facing the same ...
I am not able to login with my credentials. This is happening with me again and again.i have created different account then also,i am facing the same proble..please help me to resolve this issue...i am a new learner here
- 1672 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @PREM RANJAN​ It might be a temporary issue, for any issue with Academy learning/certifications, you can raise a ticket in the below link, sharing it with you for your future reference as well.https://help.databricks.com/s/contact-us?ReqType=train...
- 0 kudos
- 2563 Views
- 2 replies
- 0 kudos
Resolved! When Databricks on AWS will support c6i/m6i/r6i EC2 instance types?
The instances are almost 1.5 years old now and provide better efficiency that the 5 gen.
- 2563 Views
- 2 replies
- 0 kudos
- 0 kudos
@Sergey Ivanychev​ those instance types are under development and should be GA very soon. No official date AFAIK
- 0 kudos
- 4773 Views
- 4 replies
- 0 kudos
Running notebook from another notebook does not work when running notebook from github actions
I have a setup-notebook that users %run to run a series of notebooks. The notebook is in the root folder of my repo. In a subfolder I have several notebooks I want to run. If I run the notebook in databricks, the relative paths work (%run "./subfolde...
- 4773 Views
- 4 replies
- 0 kudos
- 0 kudos
It seems that the directory is different under GitHub actions. Not sure what exactly mentioned GitHub actions do and which one it is. Maybe you can share GitHub action used.
- 0 kudos
- 3584 Views
- 4 replies
- 1 kudos
Passing the secret scope to the url
We are trying to call an URL by using the credentials, we are able to get the data when we hard code the credentials.Not returning any data when we pass the secret scope credentials.below is the code.import requestssource_db_scope = "dev-hnd-secret-s...
- 3584 Views
- 4 replies
- 1 kudos
- 1 kudos
please try to debug what secret scope is returning. The ugly way to do it is:for letter in username: print(letter, ' ')
- 1 kudos
- 1716 Views
- 1 replies
- 8 kudos
Now you can read the Delta Sharing table as a stream.Â
Now you can read the Delta Sharing table as a stream.
- 1716 Views
- 1 replies
- 8 kudos
- 8 kudos
Very informative, Thanks for sharing
- 8 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
