- 2520 Views
- 4 replies
- 0 kudos
How to Create a DataBricks Notebook using API
import requestsimport json# Databricks workspace API URLdatabricks_url = "https://dbc-ab846cbe-f48b.cloud.databricks.com/api/2.0/workspace/import"# Databricks API token (generate one from your Databricks account)databricks_token = "xxxxxxxxxxxxxxxxxx...
- 2520 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @pranav2, The error message Failed to create notebook: 401 Unauthorized Indicates that the server is not recognizing your credentials. This could be due to various reasons, including an incorrect or expired token or a lack of necessary permissions...
- 0 kudos
- 994 Views
- 1 replies
- 1 kudos
Connect to delta table from mlflow pyfunc serving endpoint
Hi, I'm creating an mlflow pyfunc serving endpoint and I would like to connect to a delta table to retrieve some information within the pyfunc. Is this possible?I ask because I don't think that serving endpoint environment has access to spark, and we...
- 994 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Chibberto, Yes, connecting to a delta table within the MLflow sync serving endpoint is possible. However, as you correctly pointed out, the serving endpoint environment cannot access Spark. Therefore, you would need to instantiate a Spark sessio...
- 1 kudos
- 2378 Views
- 5 replies
- 2 kudos
Resolved! URGENT - Data Bricks certification Exam Suspended
Hi Team, I scheduled my exam today and I showed the room in proctor. They said dull light. But I turned on with a better place. They again wanted to share the room and suspended the exam. Please help me asapWebassessor Id: npt.senthil@gmail.com
- 2378 Views
- 5 replies
- 2 kudos
- 1702 Views
- 1 replies
- 0 kudos
Databricks cluster restarted inconsistantly causing job failure ( all purpose cluster)
"run failed with error message Driver of the cluster (0307-***-gpbwt) was restarted during the run.", "effectiveIntegrationRuntime": "vnet-ir-*-**** (East US) while performing merge operation.This error is not consistant. DB runtime: 11.3 LTS (inclu...
- 1702 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SDas1 , Based on the provided information, the error message "Driver of the cluster (0307-***-gpbwt) was restarted during the run" indicates that the driver node of your cluster has restarted during the job run. This could be due to a variety of...
- 0 kudos
- 2174 Views
- 1 replies
- 1 kudos
Resolved! Delta Live Tables and GIT
Notebooks that runs in a delta live table are GIT enabled, but what about the Delta Live Table pipeline?I'm looking for a good way to deploy pipelines from DEV to TEST and TEST to PROD that not just deploy the notebooks but also the pipeline.What pos...
- 2174 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Henrik , - Delta Live Tables (DLT) pipelines can be managed and deployed using Databricks API.- The API allows you to create, edit, delete, start, and view pipeline details.- You can develop and test DLT pipelines in a DEV environment.- Use the ...
- 1 kudos
- 855 Views
- 1 replies
- 0 kudos
Bootstrap Timeout during cluster start on AWS cloud
Hi!We had bunch of strange failures for our jobs during 28-29 of September.Some jobs` runs could not start for some time (30-50 mins) and then were failed with an error:Unexpected failure while waiting for the cluster (0929-002141-2zkekhdj) to be rea...
- 855 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rpaschenko, The failures you experienced on September 28-29 could be due to various reasons. For the jobs that could not start for some time and then failed with an error, there appears to be a timeout issue while initializing the instance. This...
- 0 kudos
- 2046 Views
- 1 replies
- 0 kudos
Databricks on Virtualization
Hi Team,Can you please direct me to any content on Databricks on Virtualization?Regards,Phanindra
- 2046 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Phani1, Databricks provides the ability to deploy a Databricks workspace to your virtual network, also known as VNet injection. This option allows for network customization and offers a variety of features: * Connect Databricks to other Azure se...
- 0 kudos
- 4180 Views
- 0 replies
- 3 kudos
What's new in Databricks for September 2023
Platform You can now use Structured Streaming to Stream Data from Apache Pulsar on Databricks. For more information : https://docs.databricks.com/en/structured-streaming/pulsar.html (DBR 14.1 required)Databricks Runtime 14.1 and 14.1 ML are now avail...
- 4180 Views
- 0 replies
- 3 kudos
- 1950 Views
- 3 replies
- 0 kudos
om.microsoft.azure.storage.StorageException: The specifed resource name contains invalid characters.
Hi guys I'm relatively new to Databricks and struggling to implement an autoloader ( with trigger once = true ) in file notifications mode. I have CSV files in one container (landing zone). I would like the autoloader to pick up new and existing file...
- 1950 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Kaniz, thank you for your reply. I initially made the mistake of using a capital letter in the queue as part of config files. I can now write, there is no error as a batch process. However, when I try to run the write stream, it says"Running Comma...
- 0 kudos
- 7812 Views
- 9 replies
- 1 kudos
My exam Datbricks Data Engineer Associate got suspended_need immediate help please (10/09/2023)
Hello Team,I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file ...
- 7812 Views
- 9 replies
- 1 kudos
- 1643 Views
- 2 replies
- 2 kudos
Is Datbricks-Salesforce already available?
Reference: Salesforce and Databricks Announce Strategic Partnership to Bring Lakehouse Data Sharing and Shared AI Models to Businesses - Salesforce NewsI was going through this article and wanted to know if anyone in community is planning to use this...
- 1643 Views
- 2 replies
- 2 kudos
- 2564 Views
- 3 replies
- 3 kudos
Resolved! Terraform provider, problemi in creating dependant task!
Hola all.I have a serious problem, perhaps I missed something, but can't find the solution. I need to push a job description to Databricks using TERRAFORM. I wrote the code, but there is no way to get a task dependant from two different tasks.Conside...
- 2564 Views
- 3 replies
- 3 kudos
- 3 kudos
@6502 You need to make multiple depends_on blocks for each dependency, ex.depends_on { task_key = "ichi" } depends_on { task_key = "ni" }
- 3 kudos
- 1513 Views
- 2 replies
- 0 kudos
Connection VPC AWS in Databricks for extract data Oracle Onpremise
Hi, help meHow can I consume a VPC when it is already anchored in the "network" of Databricks to extract information from a server that really made ping and works
- 1513 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Kaniz_Fatma I have already configured in "Cloud resources"/ "Network" the Private endpoint links and the linked VPC in ibabricks. How can I connect to Oracle using the EC2 virtual machine?Thank u
- 0 kudos
- 754 Views
- 1 replies
- 0 kudos
Error while attempt to give Lakehouse fundamentals exam
I would like to know why I am getting this error when I tried to earn badges for lake house fundamentals. I can't access the quiz page. Can you please help on this?I am getting 403: Forbidden error.
- 754 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @simran_27 could you please try using the link
- 0 kudos
- 7704 Views
- 3 replies
- 1 kudos
Structured Streaming of S3 source
I am trying to setup s3 as a structured streaming source. The bucket receives ~17K files/day and the original load to the bucket was ~54K files. The bucket was first loaded 3 months ago and we haven't started reading from it since. So let's say there...
- 7704 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks,We were able to make things work by increasing the driver instance size so it has more memory for the initial load. After initial load we scaled the instance down for subsequent runs. We're still testing, if we aren't able to make it work we'l...
- 1 kudos
