- 44 Views
- 2 replies
- 0 kudos
POC on spark 4.x
I need to do some POC with spark 3.5.7 and 4.x and need some local setup with some sample Kafka source. The POC would read data from Kafka via streaming job and write to delta table and I would like to do this on spark-4.x ..Do you know of any quick ...
- 44 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @RevanthV , I did some digging and here are some helpful tips. Got it — here are fast, reproducible ways to stand up a local Kafka source and run a Spark Structured Streaming job that writes to a Delta table, plus the common fixes for the conne...
- 0 kudos
- 603 Views
- 2 replies
- 1 kudos
Dashboard Filters (Showing Description) but selecting the id value for use query parameters
HiIs there a way to have a Single or multiple filter in a Dashboard that shows titles/names but on selection passes in the identifier as query parameter filters. The cost management dashboard shipped by Databricks seems to do this for the workspace s...
- 603 Views
- 2 replies
- 1 kudos
- 1 kudos
I wanted to share my solution to this, hope this helps. Create a SQL function that accepts an ARRAY<STRING>, extracts a numeric ID enclosed in parentheses from the end of each string, and returns an ARRAY<BIGINT>.Example UsageInput: ["Item (1)", "Ite...
- 1 kudos
- 20 Views
- 1 replies
- 0 kudos
Serverless compute cannot access internet
I'm experiencing "Network is unreachable" errors when trying to access external APIs from serverless compute jobs, despite having a network policy configured to allow access to all destinations. I have the "Serverless egress control" in the account s...
- 20 Views
- 1 replies
- 0 kudos
- 0 kudos
I've run diagnostic tests and found: **DNS Resolution**: Partially working- `github.com` resolves successfully and HTTPS works- `pokeapi.co` fails DNS resolution- `google.com` fails DNS resolution- `databricks.com` fails DNS resolution **Outbound HT...
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
How do I choose between a standard cluster and a shared cluster in Databricks?
When should I use a single-user cluster, and when should I use a multi-user/shared cluster? What’s the difference and how do I pick the right one?
- 20 Views
- 0 replies
- 0 kudos
- 106 Views
- 3 replies
- 2 kudos
Resolved! Cannot view nested MLflow experiment runs without changing URL
Hello,I've recently been testing out Databricks experiments for a project of mine. I wanted to nest runs, and then see these runs grouped by their parent in the experiments UI. For the longest time, I couldn't figure out how to do this. I was seeing ...
- 106 Views
- 3 replies
- 2 kudos
- 2 kudos
@africke , If you’re happy with the results, please go ahead and accept this as the solution so others know it worked.
- 2 kudos
- 73 Views
- 4 replies
- 1 kudos
Resolved! Got an empty query file when cloning Query file.
Hello Community,In our AWS Databricks environment, we’ve encountered some behavior we don’t understand while performing the following operation.When we clone a query file that already has content, a new file is created with the same name and “(clone)...
- 73 Views
- 4 replies
- 1 kudos
- 1 kudos
@kyeongmin_baek - There is no aut-save or CMD+S for Query as it get saved only once it is attached to a cluster AND 'SAVE' icon is used. However, it still stays in cache as unsaved in that 'query' window but cloning or other file operations may lose ...
- 1 kudos
- 58 Views
- 2 replies
- 0 kudos
Resolved! Error while creating databricks custom app
I am trying to create a simple databricks custom app but I am getting Error: Could not import 'app'. error.app.yaml fileenv: - name: FLASK_APP value: '/Workspace/Users/sam@xxx.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry/app'comm...
- 58 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks, I have modified the yaml file but still getting Error: Could not import 'app' errorenv: - name: FLASK_APP value: '/Workspace/Users/xxx@zzz.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry'command: [ "flask", "--app", "...
- 0 kudos
- 39 Views
- 1 replies
- 1 kudos
Usage Dashboard from Cody Austin Davis displays only DBUs or overall cost including the VM uptime?
Hello,I have been looking at the new Dashboard created by @CodyA ( great job! ) and I was wondering if the cost displaying only provides visibility to only the databricks mark-up on each job ( i.e. $ DBUs ) or to the overall cost including cloud prov...
- 39 Views
- 1 replies
- 1 kudos
- 1 kudos
As the dashboard uses System Tables (system.billing.usage) to show spend across Jobs, SQL, and Notebooks, I dont beleive it includes cloud provider VM costs.
- 1 kudos
- 49 Views
- 4 replies
- 1 kudos
Want to see logs for lineage view run events
Hi All,I need your help, as I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. I see below steps.Job Started: The job is triggered.Waiting for Cluster: The job wait...
- 49 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Team/Member,As I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. We find below steps and also added screenshot of it. I want screenshot stages logs, where i wil...
- 1 kudos
- 93 Views
- 4 replies
- 4 kudos
Resolved! Autoloader Managed File events
Hi all,We are in the process of migrating from directory listing to managed file events in Azure Databricks. Our data is stored in an Azure Data Lake container with the following folder structure:To enable file events in Unity Catalog (UC), I created...
- 93 Views
- 4 replies
- 4 kudos
- 4 kudos
Recommended approach to continue your existing pattern:Keep the External Location enabled for file events at the high-level path (/Landing).Run a separate Structured Streaming job for each table, specifying the full sub-path in the .load() function (...
- 4 kudos
- 74 Views
- 3 replies
- 3 kudos
how to know which join type was used (broadcast, shuffle hash or sort merge join) for a query?
What is the best way to know what kind of join was used for a SQL query between broadcast, shuffle hash and sort merge? How can the spark UI or the query plan be interpreted?
- 74 Views
- 3 replies
- 3 kudos
- 3 kudos
@smoortema , Spark performance tuning is one of the hardest topics to teach or learn, and it’s even tougher to do justice to in a forum thread. That said, I’m really glad to see you asking the question. Tuning is challenging precisely because there a...
- 3 kudos
- 1330 Views
- 7 replies
- 4 kudos
create an external connection to oracle
Hi! I've been trying to create an external connection to oracle but getting the following error message "Detailed error message: ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found" I searched online and found some...
- 1330 Views
- 7 replies
- 4 kudos
- 4 kudos
hey @austinoyoung ,I don't have an Oracle database to be able to test this for you, but I believe you can get around this error by following the steps laid out in here:https://stackoverflow.com/questions/9156379/ora-01882-timezone-region-not-foundIn ...
- 4 kudos
- 30 Views
- 2 replies
- 1 kudos
Resolved! Unable to navigate/login to Databricks Account Console
Hi,I have deployed Azure Databricks using email id (say xx@gmail.com) and able to lauch a workspace.When I try to access account console, it throws below errorSelected user account does not exist in tenant 'Microsoft Services' and cannot access the ...
- 30 Views
- 2 replies
- 1 kudos
- 1 kudos
And old link but still relevant - https://github.com/cloudboxacademy/azure_databricks_course/blob/main/known-issues/unable-to-login-to-azure-databricks-account-console.md
- 1 kudos
- 3785 Views
- 1 replies
- 1 kudos
Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks
Hello Databricks CommunityI am currently using Azure Databricks with PCI-DSS compliance enabled in our workspace, as maintaining stringent security standards is crucial for our organization. However, I've discovered that once PCI-DSS compliance is tu...
- 3785 Views
- 1 replies
- 1 kudos
- 1 kudos
Once PCI-DSS compliance is enabled in Azure Databricks, the workspace is locked into a set of restrictions to maintain those standards and safeguard sensitive data. These restrictions include disabling access to features like serverless compute, whic...
- 1 kudos
- 56 Views
- 2 replies
- 1 kudos
Create a Hudi table with Databrick 17
Hi I'm trying to run my existing code which has worked on the older DB version.CREATE TABLE IF NOT EXISTS catalog.demo.ABTHudi USING org.apache.hudi.Spark3DefaultSource OPTIONS ('primaryKey' = 'ID','hoodie.table.name' = 'ABTHudi') AS SELECT * FROM pa...
- 56 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank You for your response, I will wait for more updates on that.RegardsZiggy
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 492 | |
| 349 | |
| 287 |
