- 819 Views
- 1 replies
- 0 kudos
- 819 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mitashi12 it's possible as long as you're on DBR 14.3+. Limitations with Databricks Connect for Python .
- 0 kudos
- 1284 Views
- 0 replies
- 0 kudos
Problem using from_avro function
Hello everyone,I need your help with a topic that has been preoccupying me for a few days."from_avro" function gives me a strange result when I pass it the json schema of a Kafka topic.=================================================================...
- 1284 Views
- 0 replies
- 0 kudos
- 1339 Views
- 2 replies
- 0 kudos
Merge operation with ouputMode update in autoloader databricks
Hi team,I am trying to do merge operation along with outputMode('update') and foreachmode byusing below code but it is not updating data could you please any help on this?output=(casting_df.writeStream.format('delta').trigger(availableNow=True).optio...
- 1339 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @db_knowledge Please try .foreachBatch(upsertToDelta) instead of creating the lambda inside it. Best, Alessandro
- 0 kudos
- 1013 Views
- 0 replies
- 0 kudos
Generate embeddings from third party API in Delta Live Tables
HiWe currently have a Delta Live Tables flow through which textual data is flowing. As a final enrichment step we would also want to generate embeddings using a third party api provider (probably Voyage.AI). They support batch embedding which would g...
- 1013 Views
- 0 replies
- 0 kudos
- 1663 Views
- 2 replies
- 0 kudos
Reprocess of old data stored in adls
Hi,We have a requirement fir a scenario to reprocess old data using data factory pipeline.Here are the detailsStorage in ADLSGEN2Landing zone(where the data will be stored in the same format as we get from source),Data will be loaded from sql server ...
- 1663 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod I just posted a possible solution for the above problem and it has been rejected community moderator without any explanation. This has happened to me twice in past as well.Can you please help in this case.
- 0 kudos
- 1892 Views
- 0 replies
- 0 kudos
running git clone from databricks notebook
Hey,We have a use-case where we want to clone a git repository in Azure DevOps to a storage container (Blob storage).When I'm trying to run the "git clone" command to local storage I keep getting `Operation not supported` error.Git is installed and I...
- 1892 Views
- 0 replies
- 0 kudos
- 7789 Views
- 5 replies
- 1 kudos
Resolved! I am trying to use Databricks Autoloader with File Notification Mode
When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while...
- 7789 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi,I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-for-configuring-file-notification-for-adls-gen2-and-azure-b...
- 1 kudos
- 1930 Views
- 3 replies
- 1 kudos
Databricks Certified Associate Developer Exam Got Suspended. Require support for the same.
Request #00482566Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I had some network issues and lighting issues. My test was stopped in the middle and I was connected with the proctor for reviewing. As r...
- 1930 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Kaniz @Sujitha @APadmanabhan @Cert-Team @Cert-Bricks @Cert-TeamOPS I have been waiting for more than 40+ hours since I raised my ticket. Till now I dint get any response from the support team nor from anyone. Can you please escalate this issue...
- 1 kudos
- 1515 Views
- 1 replies
- 1 kudos
Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions
or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues
- 1515 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...
- 1 kudos
- 2931 Views
- 2 replies
- 0 kudos
Saving Widgets to Git
We use Databricks widgets in our python notebooks to pass parameters in jobs but also for when we are running the notebooks manually (outside of a job context) for various reasons. We're a small team, but I've noticed that when I create a notebook an...
- 2931 Views
- 2 replies
- 0 kudos
- 0 kudos
@shanebo425 You can add your widgets to the code, ex:dbutils.widgets.text("test", "") dbutils.widgets.get("test") Remember that the cell with widget needs to be run in order for widgets to be actually visible in a notebook.
- 0 kudos
- 9916 Views
- 2 replies
- 0 kudos
XML DLT Autoloader - Ingestion of XML Files
I want to ingest multiple XML files with varying but similar structures without defining a schema.For example: <?xml version="1.0" encoding="ISO-8859-1"?> <LIEFERUNG> <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH </NAME> <ST...
- 9916 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod Thanks a lot.I found an issue in from_xml function.I posted above: SELECT from_xml(CONCAT('<ABSENDER>', ABSENDER, '</ABSENDER>'), schema_of_xml(' <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH</NAME> <STRASSE>M...
- 0 kudos
- 8403 Views
- 8 replies
- 3 kudos
Resolved! How to preserve my database when the cluster is terminated?
Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...
- 8403 Views
- 8 replies
- 3 kudos
- 3 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....
- 3 kudos
- 17112 Views
- 8 replies
- 12 kudos
Resolved! I cannot see the Hive databases or tables once I terminate the cluster and use another cluster.
I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....
- 17112 Views
- 8 replies
- 12 kudos
- 12 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...
- 12 kudos
- 2070 Views
- 1 replies
- 0 kudos
Databricks Auto Loader authorization exception
Hello,I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.Looks like all required permission are provided to the service principle: 
- 2070 Views
- 1 replies
- 0 kudos
- 5300 Views
- 3 replies
- 0 kudos
Primary Key not working as expected on Unity Catalog delta tables
Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...
- 5300 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
