In today’s fast-paced world, finding genuine connections and meaningful relationships can be challenging. Whether you’re seeking love, friendship, or just some fun, having the right tool at your fingertips can make all the difference. Enter Taimi, th...
Hi Team, My Databricks Certified exam got suspended.I was continuously in front of the camera and an alert appeared and then my exam resumed. Then later a support person asked me to show the entire table and entire room, I have showed around the room...
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam without giving any reaso...
@Kaniz @Cert-Team @Sujitha Yesterday I received a response from the support team, and they asked for a date and time for rescheduling the exam, I replied today's date(May-25-2024) for a rescheduled exam but I didn't receive any confirmation mail from...
I have create a neural network and I am training the model with the code as below. The code fails to write to the databricks file storage. is there any other way to write the checkpoint to databricks storage or to an s3 bucket directly?custom_early_...
I'm having trouble saving to local storage in Databricks DBFS due to an "invalid operation" message. Although I can save models to S3 with my AWS credentials, I need to save the best model checkpoint during training locally. For more details please c...
Clusters configuration doesn't work, it's displaying a blank page. I need to add some libraries for the cluster
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam, saying I have exceeded ...
Hi @ariharansiva00 I'm sorry to hear your exam was suspended. Thank you for filing a ticket with our support team. Please allow 24-48 hours for a resolution.
Hello fellow community members,In our organization, we have developed, deployed and utilized an API-based MLOps pipeline using Azure DevOps.The CI/CD pipeline has been developed and refined for about 18 months or so, and I have to say that it is pret...
Hello,I have a quick question. If my source code call pysark collect() or any method related to rdd methods, then pytest on my local PC will report the following error. My local machine doesn't have any specific setting for pyspark and I used findspa...
Hi,The error message and stack trace doesn't seem to suggest that this is a failed pytest issue. If that's true, can you please try to replicate what's being invoked from `src\tests\test_calculate_psi_for_each_column.py` in a Python REPL?And separate...
The feature store UI is lagging (Azure Databricks). It is extremely slow to update - takes several days from creating a feature table before it appears in the UI. Anyone share the same experience?
Hi @Kjetil, I’m sorry to hear that you’re experiencing lag with the Azure Databricks feature store UI. The Databricks feature store is a centralized repository that enables data scientists to find and share features. It ensures that the same code ...
Hi,I want to run a python code on databricks notebook and return the value to my local machine. Here is the summary:I upload files to volumes on databricks. I generate a md5 for local file. Once the upload is finished, I create a python script with t...
Hello @pshuk, You could check the below CLI commands: get-run-output Get the output for a single run. This is the REST API reference, which relates to the CLI command: https://docs.databricks.com/api/workspace/jobs/getrunoutput export-run There's al...
Hello, I'm attempting to use the databricks API to list the catalogs in the metastore. When I send the GET request to `/api/2.1/unity-catalog/catalogs` , I get this error I have checked multiple times and yes, we do have a metastore associated with t...
Turns out I was using the wrong databricks host url when querying from postman. I was using my Azure instance instead of my AWS instance.
Hi,I have a spark streaming job which reads from kafka and process data and write to delta lake.Number of kafka partition: 100number of executor: 2 (4 core each)So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to un...
Hi @subham0611, In Spark Streaming, the number of threads is not explicitly controlled by the user. Instead, the parallelism is determined by the number of partitions in the Kafka topic. Each partition is consumed by a single Spark task. When you ...
I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...
Hi @MohsenJ, You can indeed develop and run Databricks notebooks with .py extension on your local computer using the Databricks extension for Visual Studio Code and Databricks Connect1. Here are the steps you need to follow: Enable Databricks Con...
Assessment(Assessment job need to be deployed using Terraform)1.Install latest version of UCX 2.UCX will add the assessment job and queries to the workspace3.Run the assessment using ClusterHow to write code for this by using Terraform. Can anyone he...
Hi @unity_Catalog, To deploy an assessment job using Terraform, you’ll need to write a Terraform configuration that includes the following steps: Install the latest version of UCX: This will likely involve using a provider to manage software inst...
The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...
What you can do is define a SQL endpoint as a linked server. Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...
