I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...
As of now, if I try to list the jobs via "list job" API then there is a limit of 25 jobs only.Is there a way to list all the available/visible jobs to a user?
Hi @Saurabh Verma​, We haven’t heard from you on the last response from @Arvind Ravish​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to other...
My Azure Databricks workspace was decommissioned. I forgot to copy files stored in the DatabricksRoot storage (dbfs:/FileStore/...).Can the workspace be recommissioned/restored? Is there any way to get my data back?Also, is there any difference betwe...
I want to use databricks inside vscode and I therefore need Databricks-connect I configure my settings using databricks-connect configure as follows: Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]Databricks Token [<my token>]Cl...
In case it helps anyone, I ran into this issue and had to remove the trailing / from the host name. It used to work fine with the trailing / so something must have changed.
I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...
Hi @Lloyd Vickery​, We haven’t heard from you on the last response from @Werner Stinckens​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to ot...
I know how to do it with spark, and read/write tables (like https://docs.microsoft.com/en-gb/azure/databricks/data/data-sources/sql-databases#python-example )But this time, I need to only update a field of a specific row in a table. I do not think I ...
thanks for the link.I am maybe wrong, but they describe how to connect with spark. They do not provide a connection engine that we could use directly (like with pyodbc) or an engine that we could use in pandas, for example.
HI there,I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy. In ML-08 - Hyperopt notebook, I see the following error in cmd 13. best_hyperparam = fmin(fn=objectiv...
Per the documentation, a workspace is limited to 1k concurrent job runs.Can somebody clarify how the concurrency limit is set i.e:Is it 1k concurrent runs across all jobs in the workspaceIs it 1k concurrent runs for a single jobAlso, is there any way...
Hi @Kaniz Fatma​ Weners gave some great suggestions on the issue we are dealing with. But some confirmation on the question from data bricks side would be much appreciated
I'm trying to configure audit logs for databricks and as a part of it I need to configure S3 storage using endpoint storage-configurations.But for some reason I'm getting invalid endpoint. I'm using following URL with basicAuth.https://accounts.cloud...
Hey there @Aman Sehgal​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...
Using above configuration in cluster, when I run databricks job parallelly with multiple request at a same time, then I am getting mount/unmount issue. For an example : When I make three request to databricks job , it run 3 jobs parallelly but somet...
Hi @rahul upadhyay​, We haven’t heard from you on the last response from @Prabakar Ammeappin​ . If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.
Hi,Wanted to ask, I have been experiencing databricks_error = user initiated cluster terminationdatabricks_error_message: timeout while placing nodesand soetimes it stays stuck on "finding new nodes, instances if neccessary" for hours..and it goes of...
Hi @Ghazanfar Uruj​ , We haven’t heard from you on the last response from @Atanu Sarkar​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to oth...
I created a table that aggregates data by year and week of year and display this in a chart over time. As part of the query, I order by the year, then week columns. In the visualization on the query (in the SQL editor), I disabled the sort, because i...
Hi @Erik Louie​ , We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will res...
Hi @Manav Mishra​, We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will re...
I signed up with databricks free trial but I am not be able to create new cluster , could you help me in this regard ?When i create cluster nothing happens and it terminates automatically , I'm on aws
