- 1053 Views
- 1 replies
- 1 kudos
Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions
or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues
- 1053 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...
- 1 kudos
- 1933 Views
- 2 replies
- 0 kudos
Saving Widgets to Git
We use Databricks widgets in our python notebooks to pass parameters in jobs but also for when we are running the notebooks manually (outside of a job context) for various reasons. We're a small team, but I've noticed that when I create a notebook an...
- 1933 Views
- 2 replies
- 0 kudos
- 0 kudos
@shanebo425 You can add your widgets to the code, ex:dbutils.widgets.text("test", "") dbutils.widgets.get("test") Remember that the cell with widget needs to be run in order for widgets to be actually visible in a notebook.
- 0 kudos
- 9171 Views
- 2 replies
- 0 kudos
XML DLT Autoloader - Ingestion of XML Files
I want to ingest multiple XML files with varying but similar structures without defining a schema.For example: <?xml version="1.0" encoding="ISO-8859-1"?> <LIEFERUNG> <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH </NAME> <ST...
- 9171 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod Thanks a lot.I found an issue in from_xml function.I posted above: SELECT from_xml(CONCAT('<ABSENDER>', ABSENDER, '</ABSENDER>'), schema_of_xml(' <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH</NAME> <STRASSE>M...
- 0 kudos
- 6470 Views
- 8 replies
- 3 kudos
Resolved! How to preserve my database when the cluster is terminated?
Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...
- 6470 Views
- 8 replies
- 3 kudos
- 3 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....
- 3 kudos
- 12309 Views
- 8 replies
- 12 kudos
Resolved! I cannot see the Hive databases or tables once I terminate the cluster and use another cluster.
I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....
- 12309 Views
- 8 replies
- 12 kudos
- 12 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...
- 12 kudos
- 1504 Views
- 1 replies
- 0 kudos
Databricks Auto Loader authorization exception
Hello,I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.Looks like all required permission are provided to the service principle: 
- 1504 Views
- 1 replies
- 0 kudos
- 3770 Views
- 3 replies
- 0 kudos
Primary Key not working as expected on Unity Catalog delta tables
Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...
- 3770 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 5211 Views
- 0 replies
- 1 kudos
Spark tasks getting stick on one executor
Hi All,I am running a Spark job using cluster with 8 executor with 8 cores each. The job involves execution of UDF. The job processes rows in few 100 thousands. When I run the job, each executor is assigned 8 job each. Usually the job succeeds in les...
- 5211 Views
- 0 replies
- 1 kudos
- 8924 Views
- 7 replies
- 10 kudos
Resolved! How to use dbx for local development.
​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing...
- 8924 Views
- 7 replies
- 10 kudos
- 584 Views
- 0 replies
- 0 kudos
Random NoClassDefFound Error when running job
We are running a job on a cluster with DBR 10.4 LTS Spark 3.2.1 and scala 2.12. Cluster is using 4 workers (spot instances) .The driver is not a spot instanceRandomly ( only on one environment and not all executions), we are getting the following err...
- 584 Views
- 0 replies
- 0 kudos
- 1352 Views
- 1 replies
- 0 kudos
ModuleNotFoundError when run with foreachBatch on serverless mode
I using Notebooks to do some transformations I install a new whl: %pip install --force-reinstall /Workspace/<my_lib>.whl %restart_python Then I successfully import the installed lib from my_lib.core import test However when I run my code with fo...
- 1352 Views
- 1 replies
- 0 kudos
- 2482 Views
- 2 replies
- 0 kudos
SQL Warehouse: Retrieving SQL ARRAY Type via JDBC driver
Hi all,we are currently running into the following issuewe are using serverless SQL warehousein a JAVA application we are using the latest Databricks JDBC driver (v2.6.36)we are querying the warehouse with a collect_list function, which should return...
- 2482 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Wilco, The answer is no, ODBC/JDBC don't support complex types so these need to be compressed into strings over the wire (usually in JSON representation) and rehydrated on the client side into a complex object.
- 0 kudos
- 3374 Views
- 2 replies
- 0 kudos
Resolved! ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)should one create it in the work space manually via UI? and how?would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.
- 3374 Views
- 2 replies
- 0 kudos
- 0 kudos
I am experiencing significant delay on my streaming. I am using changefeed connector. Its processing streaming batch very frequently but experiencing sudden halt and shows no active stage for longer time. I observed below exception continuously promp...
- 0 kudos
- 7045 Views
- 2 replies
- 4 kudos
Resolved! Streaming Delta Live Tables
I'm a little confused about how streaming works with DLT. My first questions is what is the difference in behavior if you set the pipeline mode to "Continuous" but in your notebook you don't use the "streaming" prefix on table statements, and simila...
- 7045 Views
- 2 replies
- 4 kudos
- 4 kudos
Is it possible to have custom upserts in streaming tables in a delta live tables pipeline?Use case: I am trying to maintain a valid session based on timestamp column and want to upsert to the target table.Tried going through the documentations but dl...
- 4 kudos
- 1007 Views
- 2 replies
- 0 kudos
Unable to execute update statement through Databricks Notebook
I am unable to execute update statements through Databricks Notebook, getting this error message "com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runti...
- 1007 Views
- 2 replies
- 0 kudos
- 0 kudos
This is resolved, this happens when a Column in the table has a GENERATED BY DEFAULT AS IDENTITY defined. When you remove this column, it works fine
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
