- 3196 Views
- 2 replies
- 1 kudos
Given user id, what API can we use to find out which cluster the user is connected to?
I want to know the cluster that user is connected to in databricks. It would be great if we can also get the duration that the user is connected.
- 3196 Views
- 2 replies
- 1 kudos
- 1 kudos
You can track activity logs by activating audit logs.I'm not sure which cloud provider you're using, but ex. for Azure you can find a manual here:https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/audit-logs
- 1 kudos
- 1738 Views
- 1 replies
- 1 kudos
What are the requirements in order for the event log to collect backlog metrics?
I am trying to use the event log to collect metrics on the 'flow_progess' under the 'event_type' field. In the the docs it suggests that this information may not be collected based on the data source and runtime used (see screenshot). Can anyone let ...
- 1738 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for contacting Databricks Support! I understand that you're looking for information on unsupported data source types and runtimes for the backlog metrics. Unfortunately, we currently have not documented that information. It's possible that som...
- 1 kudos
- 7006 Views
- 5 replies
- 7 kudos
Databricks ADLS vs Azure Sql ? which is better for datawarehousing ? and why
Databricks ADLS vs Azure Sql ? which is better for datawarehousing ? and why
- 7006 Views
- 5 replies
- 7 kudos
- 7 kudos
Databricks is the data lake / lakehouse and Azure SQL is the database.
- 7 kudos
- 5151 Views
- 4 replies
- 1 kudos
Databricks workspace with custom VPC using terraform in Google Cloud
I am working on Google Cloud and want to create Databricks workspace with custom VPC using terraform. Is that supported ? If yes is it similar to AWS way ?Thank youHoratiu
- 5151 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @horatiu guja​ GCP Workspace provisioning using Terraform is public preview now. Please refer to the below doc for the steps.https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/gcp-workspace
- 1 kudos
- 10488 Views
- 4 replies
- 0 kudos
SELECT from table saved under path
Hi!I saved a dataframe as a delta table with the following syntax:(test_df .write .format("delta") .mode("overwrite") .save(output_path) )How can I issue a SELECT statement on the table?What do I need to insert into [table_name] below?SELECT ...
- 10488 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @John B​ there is two way to access your delta table-SELECT * FROM delta.`your_delta_table_path`df.write.format("delta").mode("overwrite").option("path", "your_path").saveAsTable("table_name")Now you can use your select query-SELECT * FROM [table_...
- 0 kudos
- 2100 Views
- 1 replies
- 2 kudos
Is Delta Live Tables planned to be open source in the future?
Is Delta Live Tables planned to be open source in the future?
- 2100 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello there @G Z​ I would say "we have a history of open sourcing our biggest innovations but there's no concrete timeline for dlt. It's built on the open APIs of spark and delta, so the most important parts (your transformation logic and you data) ...
- 2 kudos
- 5009 Views
- 4 replies
- 4 kudos
Resolved! Databricks - Workflow- Jobs- Script to automate
Hi - I have created a Databricks job - under Workflow - its running fine without any issues . I would like to promote this job to other workspaces using a script.Is there a way to script the job definition and deploy it across multiple workspaces .I ...
- 5009 Views
- 4 replies
- 4 kudos
- 3285 Views
- 2 replies
- 0 kudos
Cross region Databricks to SQL Connection
We are trying to connect Azure Databricks Cluster to Azure SQL database but the firewalls at SQL level is causing an issue.Whitelisting dbks subnet is not an option here as both the resources are in two different azure regions. Is there a secure way ...
- 3285 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Timir Ranjan​,Have you tried looking into private endpoints? This allows you to expose your Azure SQL database from the Azure backbone and is cross-regional supported.https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overviewP...
- 0 kudos
- 13143 Views
- 10 replies
- 0 kudos
Resolved! Large MERGE Statements - 500+ lines of code!
I'm new to databricks. (Not new to DB's - 10+ year DB Developer).How do you generate a MERGE statement in DataBricks? Trying to manually maintain a 500+ or 1000+ lines in a MERGE statement doesn't make much sense? Working with Large Tables of between...
- 13143 Views
- 10 replies
- 0 kudos
- 0 kudos
In my opinion, when possible MERGE statement should be on the primary key. If not possible you can create your own unique key (by concatenate some fields and eventually hashing them) and then use it in merge logic.
- 0 kudos
- 5864 Views
- 5 replies
- 7 kudos
- 5864 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @KVNARK .​ Could you please send the query that you are executing, that will help me to debug the error.
- 7 kudos
- 7047 Views
- 3 replies
- 6 kudos
Why is my lineage extraction not showing up in the Unity Catalog
Im trying to get the lineage graph to work in Unity catalog, however nothing seems to appear even though I followed the docs. I did the following steps1. Created a Unity metastore and attached the workspace to that metastore.2. Created a Single user ...
- 7047 Views
- 3 replies
- 6 kudos
- 6 kudos
@Stephan Smit​ We finally got a solution from level 3 support (Databricks support).You may check your firewall logs.On our side, we had to open communication to "Event Hub endpoint".The destination depends on your workspace region: Azure Databricks r...
- 6 kudos
- 2525 Views
- 1 replies
- 0 kudos
Monitoring
Are there any event streams that are or could be exposed in AWS (such as Cloudwatch Eventbridge events or SNS messages? In particular I'm interested in events that detail jobs being run. The use case here would be for monitoring jobs from our web app...
- 2525 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, there are several event streams in AWS that can be used to monitor jobs being run. Your Texas BenefitsCloudWatch Events: This service allows you to set up rules to automatically trigger actions in response to specific events in other AWS service...
- 0 kudos
- 27689 Views
- 19 replies
- 10 kudos
Resolved! Correlated column exception in SQL UDF when using UDF parameters.
EnvironmentAzure Databricks 10.1, including Spark 3.2.0ScenarioI want to retrieve the average of a series of values between two timestamps, using a SQL UDF.The average is obviously just an example. In a real scenario, I would like to hide some additi...
- 27689 Views
- 19 replies
- 10 kudos
- 10 kudos
Thanks for your suggestion. The fact that I want to do this in SparkSQL is because there is no underlying SQLServer.
- 10 kudos
- 7394 Views
- 5 replies
- 0 kudos
Resolved! Can Unity catalog grant the access to a file inside azure datalake storage?
Hi databricks community,I have searched quite a while through the internet but did not find an answer. If I have configured the azure datalake connection in Unity data catalog, is it possible to grant the access to users for a specific file or a fold...
- 7394 Views
- 5 replies
- 0 kudos
- 0 kudos
As @werners said service principal needs to have access to the file level.In the unity catalog, you can use "READ FILES"/"WRITE FILES" permission to give someone the possibility of reading files from the storage level (but through databricks).
- 0 kudos
- 14664 Views
- 8 replies
- 10 kudos
Resolved! Dataframe rows missing after write_to_delta and read_from_delta
Hi, i am trying to load mongo into s3 using pyspark 3.1.1 by reading them into a parquet. My code snippets are like:df = spark \ .read \ .format("mongo") \ .options(**read_options) \ .load(schema=schema)df = df.coalesce(64)write_df_to_del...
- 14664 Views
- 8 replies
- 10 kudos
- 10 kudos
So i think i have solved the mystery here it was to do with the retention config. By setting the retentionEnabled to True and rention hours being 0, we somewhat loses a few rows in the first file as they were mistaken as files from last session and ...
- 10 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
