- 1777 Views
- 1 replies
- 0 kudos
Urgent: Need Authentication Reset for Databricks Workspace Access
I am unable to access my Databricks workspace because it is still redirecting to Microsoft Entra ID (Azure AD) authentication, even after I have removed the Azure AD enterprise application and changed the AWS IAM Identity Center settings.Issue Detail...
- 1777 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Pooviond! Please submit a ticket with the Databricks Support team for assistance in resolving this issue.
- 0 kudos
- 6068 Views
- 4 replies
- 1 kudos
Resolved! How best to measure the time-spent-waiting-for-an-instance?
I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...
- 6068 Views
- 4 replies
- 1 kudos
- 11367 Views
- 7 replies
- 2 kudos
Resolved! Move multiple notebooks at the same time (programmatically)
If I want to move multiple (hundreds of) notebooks at the same time from one folder to another, what is the best way to do that? Other than going to each individual notebook and clicking "Move".Is there a way to programmatically move notebooks? Like ...
- 11367 Views
- 7 replies
- 2 kudos
- 2 kudos
You can use the export and import API calls in order to export this notebook to your local machine and then import it to the new workspace.Export: https://docs.databricks.com/api/workspace/workspace/exportImport: https://docs.databricks.com/api/works...
- 2 kudos
- 4830 Views
- 1 replies
- 0 kudos
Resolved! Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?
Hi, I have following streaming setup:I want to remove duplicates in streaming.1) deduplication strategy is defined by two fields: extraction_timestamp and hash (row wise hash)2) watermark strategy: extraction_timestamp with "10 seconds" interval--> R...
- 4830 Views
- 1 replies
- 0 kudos
- 0 kudos
Found solution. https://kb.databricks.com/streaming/how-to-efficiently-manage-state-store-files-in-apache-spark-streaming-applications <-- these two parameters.
- 0 kudos
- 3242 Views
- 6 replies
- 2 kudos
Disable exiting current cell when moving around with keyboard arrows
Is there any way do disable exiting current cell when I move cursor around with arrows. When I press up arrow or down arrow it will exit the current cell and go to another cell. Can that functionally be disabled so when I hold up or down arrow key, c...
- 3242 Views
- 6 replies
- 2 kudos
- 7530 Views
- 4 replies
- 0 kudos
Use Python notebook to read data from Databricks
I'm very new to Databricks. I hope this is the right place to ask this question.I want to use PySpark in a notebook to read data from a Databricks database with the below codes. databricks_host = "adb-xxxx.azuredatabricks.net" http_path = "/sql/1.0/w...
- 7530 Views
- 4 replies
- 0 kudos
- 0 kudos
I would try changing the query to something like the following, it should return the column names in the table so you can see if the jdbc call is actually returning the data correctlySELECT * FROM wphub_poc.gold.v_d_building limit 10
- 0 kudos
- 8597 Views
- 8 replies
- 22 kudos
Turn Off Auto-reveal of Navigation Sidebar
I work with the navigation sidebar closed and use the stacked hamburgers symbol in the upper left to reveal it when I want. Now, if you mouse over the left edge of the browser window too slowly it will auto-reveal the navigation sidebar. I do not wan...
- 8597 Views
- 8 replies
- 22 kudos
- 22 kudos
I've checked with the team, and there's no way to turn this off. However, they are making adjustments to improve the experience, and a fix to refine the sidebar functionality is on the way.
- 22 kudos
- 1395 Views
- 1 replies
- 0 kudos
New Group in Denver
Can we create a new Group in Denver?
- 1395 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes Please.
- 0 kudos
- 4566 Views
- 3 replies
- 0 kudos
Virtual Learning Festival Enrollment
Hi everyone,I tried to enroll to Virtual Learning Festival: 9 April - 30 April but upon clicking the Customers & Prospects link for LEARNING PATHWAY 1: ASSOCIATE DATA ENGINEERING I got the error (refer attached image).Thank you in advance for the hel...
- 4566 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Advika,Refer attached image. I thought it was attached in my question. I really new to here.
- 0 kudos
- 1077 Views
- 1 replies
- 0 kudos
Expectation in DLT using multiple columns
Is it possible to define an expectation in DLT pipeline using multiple columns?For example, my source has two fields - Division, Material_Number. For division 20, material number starts with 5; for 30 material number starts with 9.Can we have this ...
- 1077 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Master_DataBric , Yes its possibleHere is the doc link : - https://docs.databricks.com/aws/en/dlt/expectations?language=Python- https://docs.databricks.com/aws/en/dlt/expectations?language=SQL
- 0 kudos
- 14944 Views
- 2 replies
- 1 kudos
POC Comparison: Databricks vs AWS EMR
Hello,I need some assistance with a comparison between Databricks and AWS EMR. We've been evaluating the Databricks Data Intelligence platform for a client and found it to be significantly more expensive than AWS EMR. I understand the challenge in ma...
- 14944 Views
- 2 replies
- 1 kudos
- 1 kudos
Databricks is highly optimized for Delta, which leverages columnar storage, indexing, and caching for better performance.Instead of directly processing CSV files, convert them to Delta first, then perform aggregations and joins, see if this helps
- 1 kudos
- 1307 Views
- 1 replies
- 1 kudos
Is it possible to concatenate two notebooks?
I don't think it's possible but I thought I would check. I need to combine notebooks. While developing I might have code in various notebooks. I read them in with "%run".Then when all looks good I combine many cells into fewer notebooks. Is there any...
- 1307 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @397973, Combining multiple notebooks into a single notebook isn't an out-of-the-box feature, but will try to combine %run commands ando output them to see if it works, sort of like: %run "/path/to/notebook1"%run "/path/to/notebook2"
- 1 kudos
- 2463 Views
- 2 replies
- 1 kudos
Databricks Lakehouse Monitoring
Hi,I am trying to implement lakehouse monitoring using Inference profile for my inference data that I have, I see that when I create the monitor, two tables get generated profile and drift, I wanted to understand how are these two tables generating a...
- 2463 Views
- 2 replies
- 1 kudos
- 1 kudos
When you create a Databricks Lakehouse Monitoring monitor with an Inference profile, the system automatically generates two metric tables: a profile metrics table and a drift metrics table. Here's how this process works: Background Processing When yo...
- 1 kudos
- 3952 Views
- 2 replies
- 0 kudos
Liquid Clustering Key Change Question
If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...
- 3952 Views
- 2 replies
- 0 kudos
- 0 kudos
@ShivangiB You're correct in your understanding. When you change a clustering key using ALTER TABLE followed by OPTIMIZE, it doesn't automatically recluster existing data. Let me explain why this happens and what options you have.In Delta Lake (which...
- 0 kudos
- 1145 Views
- 1 replies
- 0 kudos
Unable to Access S3 from Serverless but Works on Cluster
Hi everyone,I am trying to access data from S3 using an access key and secret. When I run the code through Databricks clusters, it works fine. However, when I try to do the same from a serverless cluster , I am unable to access the data.I have alread...
- 1145 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @HarryRichard08! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original post. I recommend continuing the discussion in that thread to keep the conversation focused and organized.
- 0 kudos
