HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...
Hi @Marinagomes,
Try Using try_to_timestamp: Instead of to_timestamp, consider using try_to_timestamp. It returns null for malformed expressions, which can help identify problematic rows.
When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?
Hi I am trying to get query history data from my SQL warehouse. Following previous examples is not working. databricks_workspace_url = "xxx"token = "xxx"start_time = 1707091200end_time = 1707174000api_endpoint = f"{databricks_workspace_url}/api/2.0/s...
@Cheryl - you can use query_start_time=2023-01-01T00:00:00Z as a parameter to filter for the time frame. available filter criteria are given below - https://docs.databricks.com/api/workspace/queryhistory/list#filter_by-query_start_time_range
Hello everybody.Whenever I am trying to run a simple cell I receive the following error message now:Notebook detached. Exception when creating expectation context: java.net.SocketTimeoutException: Connect Timeout.After that error message the cluster ...
Sample Input data setClusterIdEventEventTime1212-18-r9u1kzn1RUNNING2024-02-02T11:38:30.168+00:001212-18-r9u1kzn1TERMINATING2024-02-02T13:43:33.933+00:001212-18-r9u1kzn1STARTING2024-02-02T15:50:05.174+00:001212-18-r9u1kzn1RUNNING2024-02-02T15:54:21.51...
Hi @RajNath , Handling event times and aggregations in large datasets can be challenging, but Structured Streaming in Databricks provides powerful tools to address this.
Let’s break down your requirements and explore how you can achieve them:
Ru...
I am new to databricks delta sharing. In case of delta sharing, i don't see any cluster running. Tried looking for documentation but only hint i got is, it usage delta sharing server but what is the cost of it and how to configure and optimize for la...
Hi @RajNath, Let’s dive into the world of Delta Sharing and explore how it works, its cost implications, and optimization strategies.
What is Delta Sharing?
Delta Sharing is a secure data-sharing platform developed by Databricks. It allows you to ...
"Notebook detached Exception when creating execution context: java.net.SocketTimeout Exception: Connect Timeout" when trying to connect my cluster to a notebook. Then "Error trying to handle that request We failed to handle that request, please try a...
Trying to use pdf2image on databricks, but its failing with "PDFInfoNotInstalledError: Unable to get page count. Is poppler installed and in PATH?"I've installed pdf2image & poppler-utils by running the following in a cell:%pip install pdf2image%pip ...
Seems like this thread has died, but for posterity, databricks provides the following code for installing poppler on a cluster. The code is sourced from the dbdemos accelerators, specifically the "LLM Chatbot With Retrieval Augmented Generation (RAG)...
We have been trying to authenticate azure-event-hub with azure AD(service principal) instead of shared access key(connection string) and read events from azure-event-hub and it is failing to initialise azure-event-hubs. And throwing no such method ex...
@swathi-dataops I have added ServicePrincipalCredentialsAuth and ServicePrincipalAuthBase as a normal classes instead of creating a separate jar for these 2 classes and packaged them as a part of my project jar.And used the below code for configuring...
I have setup authentication using this page https://docs.databricks.com/sql/api/authentication.html and run curl -n -X GET https://<databricks-instance>.cloud.databricks.com/api/2.0/sql/history/queriesTo get history of all sql endpoint queries, but I...
Hello Team,I encountered Pathetic experience while attempting my 1st Data Bricks certification. I was continuously in front of the camera and an alert appeared and then my exam resumed.Later a support person asked me to show the full room and I showe...
I'm looking at this page (Databricks Asset Bundles development work tasks) in the Databricks documentation.When repo assets are deployed to a databricks workspace, it is not clear if the "databricks bundle deploy" will remove files from the target wo...
One further question:The purpose of “databricks bundle destroy” is to remove all previously-deployed jobs, pipelines, and artifacts that are defined in the bundle configuration files.Which bundle configuration files? The ones in the repo? Or are ther...
Hi @mexp2, AWS Government Cloud now supports the Unity Catalog. Starting from November 8, 2023, new workspaces are being automatically enabled for Unity Catalog, with a gradual rollout across accounts1. If you’re using Databricks on AWS, you can set ...
We use spark mssql connector to connect sql server, it works well on dbr runtime 10.*, 11.* and 12.*. But when we use dbr 13.*, we got the error below. It happens when we are trying to use df.write to save the data to the sql database.We have encount...
I was also facing the same issue while writing to a sql server. Was able to resolve it by updating the format to "jdbc" instead of "com.micorsoft.sqlserver.jdbc.spark".df.write.format("jdbc") works on DBR 13.3 LTS using the connector: com.microsoft.a...
I have been using ipywidgets for quite a while in several notebooks in Databricks, but today things have completely stopped working with the following error;Uncaught Error: Script error for "@jupyter-widgets/base" http://requirejs.org/docs/errors.htm...
Hi @GeorgeD,
First, ensure that you’re using a compatible version of JupyterLab with ipywidgets. As of ipywidgets 7.6 or newer, it should work seamlessly with JupyterLab 3.0 or newer without any additional steps.To verify if the necessary extension ...
