- 1181 Views
- 2 replies
- 0 kudos
Reprocess of old data stored in adls
Hi,We have a requirement fir a scenario to reprocess old data using data factory pipeline.Here are the detailsStorage in ADLSGEN2Landing zone(where the data will be stored in the same format as we get from source),Data will be loaded from sql server ...
- 1181 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod I just posted a possible solution for the above problem and it has been rejected community moderator without any explanation. This has happened to me twice in past as well.Can you please help in this case.
- 0 kudos
- 1556 Views
- 0 replies
- 0 kudos
running git clone from databricks notebook
Hey,We have a use-case where we want to clone a git repository in Azure DevOps to a storage container (Blob storage).When I'm trying to run the "git clone" command to local storage I keep getting `Operation not supported` error.Git is installed and I...
- 1556 Views
- 0 replies
- 0 kudos
- 6633 Views
- 5 replies
- 1 kudos
Resolved! I am trying to use Databricks Autoloader with File Notification Mode
When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while...
- 6633 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi,I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-for-configuring-file-notification-for-adls-gen2-and-azure-b...
- 1 kudos
- 1514 Views
- 3 replies
- 1 kudos
Databricks Certified Associate Developer Exam Got Suspended. Require support for the same.
Request #00482566Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I had some network issues and lighting issues. My test was stopped in the middle and I was connected with the proctor for reviewing. As r...
- 1514 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Kaniz @Sujitha @APadmanabhan @Cert-Team @Cert-Bricks @Cert-TeamOPS I have been waiting for more than 40+ hours since I raised my ticket. Till now I dint get any response from the support team nor from anyone. Can you please escalate this issue...
- 1 kudos
- 5787 Views
- 2 replies
- 1 kudos
i have created a materialized view table using delta live table pipeline and its not appending data
i have created a materialized view table using delta live table pipeline , for some reason it is overwriting data every day , i want it to append data to the table instead of doing full refresh suppose i had 8 million records in table and if irun the...
- 5787 Views
- 2 replies
- 1 kudos
- 1 kudos
@zero234 ,Adding some suggestion based on answers from @Retired_mod. Important point to note here: "To define a materialized view in Python, apply @table to a query that performs a static read against a data source. To define a streaming table, apply...
- 1 kudos
- 1209 Views
- 1 replies
- 1 kudos
Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions
or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues
- 1209 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...
- 1 kudos
- 2231 Views
- 2 replies
- 0 kudos
Saving Widgets to Git
We use Databricks widgets in our python notebooks to pass parameters in jobs but also for when we are running the notebooks manually (outside of a job context) for various reasons. We're a small team, but I've noticed that when I create a notebook an...
- 2231 Views
- 2 replies
- 0 kudos
- 0 kudos
@shanebo425 You can add your widgets to the code, ex:dbutils.widgets.text("test", "") dbutils.widgets.get("test") Remember that the cell with widget needs to be run in order for widgets to be actually visible in a notebook.
- 0 kudos
- 9447 Views
- 2 replies
- 0 kudos
XML DLT Autoloader - Ingestion of XML Files
I want to ingest multiple XML files with varying but similar structures without defining a schema.For example: <?xml version="1.0" encoding="ISO-8859-1"?> <LIEFERUNG> <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH </NAME> <ST...
- 9447 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod Thanks a lot.I found an issue in from_xml function.I posted above: SELECT from_xml(CONCAT('<ABSENDER>', ABSENDER, '</ABSENDER>'), schema_of_xml(' <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH</NAME> <STRASSE>M...
- 0 kudos
- 7238 Views
- 8 replies
- 3 kudos
Resolved! How to preserve my database when the cluster is terminated?
Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...
- 7238 Views
- 8 replies
- 3 kudos
- 3 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....
- 3 kudos
- 14266 Views
- 8 replies
- 12 kudos
Resolved! I cannot see the Hive databases or tables once I terminate the cluster and use another cluster.
I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....
- 14266 Views
- 8 replies
- 12 kudos
- 12 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...
- 12 kudos
- 1661 Views
- 1 replies
- 0 kudos
Databricks Auto Loader authorization exception
Hello,I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.Looks like all required permission are provided to the service principle: 
- 1661 Views
- 1 replies
- 0 kudos
- 4310 Views
- 3 replies
- 0 kudos
Primary Key not working as expected on Unity Catalog delta tables
Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...
- 4310 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 5947 Views
- 0 replies
- 1 kudos
Spark tasks getting stick on one executor
Hi All,I am running a Spark job using cluster with 8 executor with 8 cores each. The job involves execution of UDF. The job processes rows in few 100 thousands. When I run the job, each executor is assigned 8 job each. Usually the job succeeds in les...
- 5947 Views
- 0 replies
- 1 kudos
- 10249 Views
- 7 replies
- 10 kudos
Resolved! How to use dbx for local development.
​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing...
- 10249 Views
- 7 replies
- 10 kudos
- 674 Views
- 0 replies
- 0 kudos
Random NoClassDefFound Error when running job
We are running a job on a cluster with DBR 10.4 LTS Spark 3.2.1 and scala 2.12. Cluster is using 4 workers (spot instances) .The driver is not a spot instanceRandomly ( only on one environment and not all executions), we are getting the following err...
- 674 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
