Hey,We have a use-case where we want to clone a git repository in Azure DevOps to a storage container (Blob storage).When I'm trying to run the "git clone" command to local storage I keep getting `Operation not supported` error.Git is installed and I...
When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while...
Hi,I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-for-configuring-file-notification-for-adls-gen2-and-azure-b...
Request #00482566Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I had some network issues and lighting issues. My test was stopped in the middle and I was connected with the proctor for reviewing. As r...
Hi @Kaniz @Sujitha @APadmanabhan @Cert-Team @Cert-Bricks @Cert-TeamOPS I have been waiting for more than 40+ hours since I raised my ticket. Till now I dint get any response from the support team nor from anyone. Can you please escalate this issue...
or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues
Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...
We use Databricks widgets in our python notebooks to pass parameters in jobs but also for when we are running the notebooks manually (outside of a job context) for various reasons. We're a small team, but I've noticed that when I create a notebook an...
@shanebo425 You can add your widgets to the code, ex:dbutils.widgets.text("test", "") dbutils.widgets.get("test") Remember that the cell with widget needs to be run in order for widgets to be actually visible in a notebook.
I want to ingest multiple XML files with varying but similar structures without defining a schema.For example: <?xml version="1.0" encoding="ISO-8859-1"?> <LIEFERUNG> <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH </NAME> <ST...
@Retired_mod Thanks a lot.I found an issue in from_xml function.I posted above: SELECT from_xml(CONCAT('<ABSENDER>', ABSENDER, '</ABSENDER>'), schema_of_xml(' <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH</NAME> <STRASSE>M...
Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....
I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...
Hello,I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.Looks like all required permission are provided to the service principle: 
Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...
Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
Hi All,I am running a Spark job using cluster with 8 executor with 8 cores each. The job involves execution of UDF. The job processes rows in few 100 thousands. When I run the job, each executor is assigned 8 job each. Usually the job succeeds in les...
​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing...
We are running a job on a cluster with DBR 10.4 LTS Spark 3.2.1 and scala 2.12. Cluster is using 4 workers (spot instances) .The driver is not a spot instanceRandomly ( only on one environment and not all executions), we are getting the following err...
I using Notebooks to do some transformations I install a new whl: %pip install --force-reinstall /Workspace/<my_lib>.whl %restart_python Then I successfully import the installed lib from my_lib.core import test However when I run my code with fo...
Hi all,we are currently running into the following issuewe are using serverless SQL warehousein a JAVA application we are using the latest Databricks JDBC driver (v2.6.36)we are querying the warehouse with a collect_list function, which should return...
Hey Wilco, The answer is no, ODBC/JDBC don't support complex types so these need to be compressed into strings over the wire (usually in JSON representation) and rehydrated on the client side into a complex object.
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
