- 13075 Views
- 4 replies
- 0 kudos
Resolved! What version of Python is used for the 16.1 runtime
I'm trying to create a spark udf for a registered model and getting:Exception: Python versions in the Spark Connect client and server are different. To execute user-defined functions, client and server should have the same minor Python version. Pleas...
- 13075 Views
- 4 replies
- 0 kudos
- 0 kudos
Does this mean that:1. A new dbx runtime comes out2. Serverless compute automatically switches to the new runtime + new python version3. Any external environments that use serverless ie, local VScode / CICD environments also need to upgrade their pyt...
- 0 kudos
- 1513 Views
- 1 replies
- 0 kudos
Lakehouse monitoring metrices tables not created automatically.
Hello,I have an external table created in databricks unity catalog workspace and trying to "Create a monitor" for the same from quality tab.While creating the same the dashboard is getting created however the two metrices tables "profile" & "drift" a...
- 1513 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @nikhil_2212! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original post. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
- 0 kudos
- 1326 Views
- 1 replies
- 0 kudos
Stream processing large number of JSON files and handling exception
application writes several JSON (small) files and the expected volumes of these files are high ( Estimate: 1 million during the peak season in a hourly window) . As per current design, these files are streamed through Spark Stream and we use autolo...
- 1326 Views
- 1 replies
- 0 kudos
- 0 kudos
We have customers that read millions of files per hour+ using Databricks Auto Loader. For high-volume use cases, we recommend enabling file notification mode, which, instead of continuously performing list operations on the filesystem, uses cloud nat...
- 0 kudos
- 1750 Views
- 1 replies
- 0 kudos
Urgent: Need Authentication Reset for Databricks Workspace Access
I am unable to access my Databricks workspace because it is still redirecting to Microsoft Entra ID (Azure AD) authentication, even after I have removed the Azure AD enterprise application and changed the AWS IAM Identity Center settings.Issue Detail...
- 1750 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Pooviond! Please submit a ticket with the Databricks Support team for assistance in resolving this issue.
- 0 kudos
- 6021 Views
- 4 replies
- 1 kudos
Resolved! How best to measure the time-spent-waiting-for-an-instance?
I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...
- 6021 Views
- 4 replies
- 1 kudos
- 3369 Views
- 2 replies
- 1 kudos
Resolved! When is it time to change from ETL in notebooks to whl/py?
Hi!I would like some input/tips from the community regarding when is it time to go from a working solution in notebooks to something more "stable", like whl/py-files?What are the pros/cons with notebooks compared to whl/py?The way i structured things...
- 3369 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @Forssen ,My advice:Using .py files and .whl packages is generally more secure and scalable, especially when working in a team. One of the key advantages is that code reviews and version control are much more efficient with .py files, as changes ...
- 1 kudos
- 11267 Views
- 7 replies
- 2 kudos
Resolved! Move multiple notebooks at the same time (programmatically)
If I want to move multiple (hundreds of) notebooks at the same time from one folder to another, what is the best way to do that? Other than going to each individual notebook and clicking "Move".Is there a way to programmatically move notebooks? Like ...
- 11267 Views
- 7 replies
- 2 kudos
- 2 kudos
You can use the export and import API calls in order to export this notebook to your local machine and then import it to the new workspace.Export: https://docs.databricks.com/api/workspace/workspace/exportImport: https://docs.databricks.com/api/works...
- 2 kudos
- 4810 Views
- 1 replies
- 0 kudos
Resolved! Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?
Hi, I have following streaming setup:I want to remove duplicates in streaming.1) deduplication strategy is defined by two fields: extraction_timestamp and hash (row wise hash)2) watermark strategy: extraction_timestamp with "10 seconds" interval--> R...
- 4810 Views
- 1 replies
- 0 kudos
- 0 kudos
Found solution. https://kb.databricks.com/streaming/how-to-efficiently-manage-state-store-files-in-apache-spark-streaming-applications <-- these two parameters.
- 0 kudos
- 3197 Views
- 6 replies
- 2 kudos
Disable exiting current cell when moving around with keyboard arrows
Is there any way do disable exiting current cell when I move cursor around with arrows. When I press up arrow or down arrow it will exit the current cell and go to another cell. Can that functionally be disabled so when I hold up or down arrow key, c...
- 3197 Views
- 6 replies
- 2 kudos
- 7474 Views
- 4 replies
- 0 kudos
Use Python notebook to read data from Databricks
I'm very new to Databricks. I hope this is the right place to ask this question.I want to use PySpark in a notebook to read data from a Databricks database with the below codes. databricks_host = "adb-xxxx.azuredatabricks.net" http_path = "/sql/1.0/w...
- 7474 Views
- 4 replies
- 0 kudos
- 0 kudos
I would try changing the query to something like the following, it should return the column names in the table so you can see if the jdbc call is actually returning the data correctlySELECT * FROM wphub_poc.gold.v_d_building limit 10
- 0 kudos
- 8541 Views
- 8 replies
- 22 kudos
Turn Off Auto-reveal of Navigation Sidebar
I work with the navigation sidebar closed and use the stacked hamburgers symbol in the upper left to reveal it when I want. Now, if you mouse over the left edge of the browser window too slowly it will auto-reveal the navigation sidebar. I do not wan...
- 8541 Views
- 8 replies
- 22 kudos
- 22 kudos
I've checked with the team, and there's no way to turn this off. However, they are making adjustments to improve the experience, and a fix to refine the sidebar functionality is on the way.
- 22 kudos
- 1367 Views
- 1 replies
- 0 kudos
New Group in Denver
Can we create a new Group in Denver?
- 1367 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes Please.
- 0 kudos
- 4547 Views
- 3 replies
- 0 kudos
Virtual Learning Festival Enrollment
Hi everyone,I tried to enroll to Virtual Learning Festival: 9 April - 30 April but upon clicking the Customers & Prospects link for LEARNING PATHWAY 1: ASSOCIATE DATA ENGINEERING I got the error (refer attached image).Thank you in advance for the hel...
- 4547 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Advika,Refer attached image. I thought it was attached in my question. I really new to here.
- 0 kudos
- 1054 Views
- 1 replies
- 0 kudos
Expectation in DLT using multiple columns
Is it possible to define an expectation in DLT pipeline using multiple columns?For example, my source has two fields - Division, Material_Number. For division 20, material number starts with 5; for 30 material number starts with 9.Can we have this ...
- 1054 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Master_DataBric , Yes its possibleHere is the doc link : - https://docs.databricks.com/aws/en/dlt/expectations?language=Python- https://docs.databricks.com/aws/en/dlt/expectations?language=SQL
- 0 kudos
- 14522 Views
- 2 replies
- 1 kudos
POC Comparison: Databricks vs AWS EMR
Hello,I need some assistance with a comparison between Databricks and AWS EMR. We've been evaluating the Databricks Data Intelligence platform for a client and found it to be significantly more expensive than AWS EMR. I understand the challenge in ma...
- 14522 Views
- 2 replies
- 1 kudos
- 1 kudos
Databricks is highly optimized for Delta, which leverages columnar storage, indexing, and caching for better performance.Instead of directly processing CSV files, convert them to Delta first, then perform aggregations and joins, see if this helps
- 1 kudos
