- 1800 Views
- 1 replies
- 0 kudos
Azure DevOps load sequence
Hi Expert,How we can setup multiple notebook in a sequence order in flow for an example 1 pipeline have notebook1 - sequence 1,Notebook2- Sequence 2(in 1pipeline only)
- 1800 Views
- 1 replies
- 0 kudos
- 0 kudos
Not sure how to approach your challenge but something you can is to use the Databricks Job Scheduler or if you want an external solution in Azure you can call several notebooks from DataFactory.
- 0 kudos
- 1241 Views
- 0 replies
- 0 kudos
DBR 14.1 Pyspark Join on df1["col1"] = df2["col1"] syntax fails
HelloAfter upgrading my cluster from DBR 12 to 14.1 I got a MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION on some of my Joinsdf1.join( df2, [df1["name"] == df2["name"], df1["age"] == df2["age"]], 'left_outer' )I resolved it by...
- 1241 Views
- 0 replies
- 0 kudos
- 963 Views
- 0 replies
- 0 kudos
Seeking Advice on Optimizing Spark Scripts for Efficient Data Processing
Hello community!I'm currently working on Spark scripts for data processing and facing some performance challenges. Any tips or suggestions on optimizing code for better efficiency? Your expertise is highly appreciated! paybyplatema Thanks.
- 963 Views
- 0 replies
- 0 kudos
- 1041 Views
- 0 replies
- 0 kudos
pushing .ipynb from azure DB to github converts file to .py
I was trying to push .ipynb file to github from Azure DB to github and appears that original file is converted to source code as .py.Why does databricks do this and how can I control which ones to do or not ?I need to keep some files as .ipynb.Thanks...
- 1041 Views
- 0 replies
- 0 kudos
- 1687 Views
- 0 replies
- 0 kudos
Getting the Databricks login screen as response to API call from Azure Data Factory
Hi,I'm trying to call the DLT api to kickoff my delta live table flow with a web API call block from Azure Data Factory. I have two environments: one DEV and one PROD.The DEV environment works fine, the response is giving me the update_id, but the PR...
- 1687 Views
- 0 replies
- 0 kudos
- 2000 Views
- 0 replies
- 0 kudos
Upload file from REST API Response directly to ADLS
i have usecase to call rest API and then return response file with base64Is it possible save the response directly to ADLS without convert it to file first ?
- 2000 Views
- 0 replies
- 0 kudos
- 18738 Views
- 2 replies
- 2 kudos
Resolved! Chrome browser is very slow
Hi. Recently, I've found that Databricks is really slow when editing notebooks, such as adding cells, copying and pasting text or cells, etc. It's just the past few weeks actually. I'm using Chrome version 118.0.5993.118. Everything else with Chrome ...
- 18738 Views
- 2 replies
- 2 kudos
- 2 kudos
It seems related to the notebook length (number of cells). The notebook that was really slow had about 40-50 cells, which I've done before without issue. Anyway after starting a new notebook using Chrome, it seems useable again. So without a specific...
- 2 kudos
- 2429 Views
- 0 replies
- 1 kudos
Not able to use graphframes library in databricks.
I have installed graphframes library from maven repository in the cluster (13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12), Standard DS4_v2). The library that I have installed is graphframes:graphframes:0.8.3-spark3.5-s_2.13.I can import the graph...
- 2429 Views
- 0 replies
- 1 kudos
- 1445 Views
- 0 replies
- 0 kudos
Unable to run query- Insert into with 'Nan' value in SQL Editor. Getting the error.
Unable to run query- Insert into with 'Nan' value in SQL Editor.Query :-Insert into ABC with values('xyz',123,Nan);Error :-org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation cannot be cast to org.apache.spark.sql.execution.datasources...
- 1445 Views
- 0 replies
- 0 kudos
- 1790 Views
- 0 replies
- 1 kudos
How to visualize result from DESCRIBE DETAIL statement
Hi all,In Lakeview Dashboard, I would like to visualize some delta table's info returned from the SQL statement 'DESCRIBE DETAIL'. In 'Data' tab, the dataset which contained that statement returned all detail info of my delta table. But the visualiz...
- 1790 Views
- 0 replies
- 1 kudos
- 1477 Views
- 0 replies
- 0 kudos
Can't see system catalog from Tableau
Hi Team, I'm trying to connect the system (system.billing.usage table) catalog (unity is enabled) in my workspace from Tableau. Im using Tableau version 2023.1 and ODBC driver version 2.7.5.1012-osx. I was able to create a connection but when Im conn...
- 1477 Views
- 0 replies
- 0 kudos
- 2474 Views
- 2 replies
- 0 kudos
DataBricks Certification Exam Got Suspended without no reason PLEASE HELP!
Hi @Cert-TeamMy test got suspend without any reason, the support guy had secured the area and I had also showed my entire desk but still they suspended my test, this is a huge loss please help me and reschedule my exam.I have also tried to raised a t...
- 2474 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Cert-Team,Still not able to raise ticket, after submitting the info for the ticket I'm not receiving any confirmation mail. Please help me to reschedule the exam as there this suspension is done without any reason and I haven't done anything whic...
- 0 kudos
- 1827 Views
- 0 replies
- 0 kudos
Using Window with a dynamic size do not work
HelloI having an issue with using a window in pyspark. I created a simple example that reproduce the error, basically i want to define a window with a dynamic size (say from another column ) instead of a fix value from pyspark.sql import SparkSessio...
- 1827 Views
- 0 replies
- 0 kudos
- 1995 Views
- 0 replies
- 0 kudos
Unable to clone Bitbucket repository on Databricks Legacy
For the past 3 weeks, I have been unable to clone the Bitbucket repository into Databricks legacy.I got the following error message "Error Creating RepoThere was an error performing the operation. Please try again or open a support ticket."I would ap...
- 1995 Views
- 0 replies
- 0 kudos
- 9280 Views
- 2 replies
- 0 kudos
Duplicate queryId in the query history api response
Hello Experts,I was using the `/api/2.0/sql/history/queries` end-point to find the historical stats for queries on the warehouse endpoint.I am surprised to see exact rows duplicated in the response, even though they have different `next_page_token` H...
- 9280 Views
- 2 replies
- 0 kudos
- 0 kudos
Sorry for formatting. This editor drives me crazy. Python code once again:#Install SDK first %pip install databricks-sdk --upgrade #Restart Python kernel dbutils.library.restartPython() #List your queries history: from databricks.sdk import Worksp...
- 0 kudos
