cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Sricharan05
by • New Contributor III
  • 1099 Views
  • 3 replies
  • 1 kudos

Databricks Certified Associate Developer Exam Got Suspended. Require support for the same.

Request #00482566Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I had some network issues and lighting issues. My test was stopped in the middle and I was connected with the proctor for reviewing. As r...

Data Engineering
Reply
  • 1099 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Sricharan05
New Contributor III
  • 1 kudos

Hi @Kaniz @Sujitha @APadmanabhan @Cert-Team @Cert-Bricks @Cert-TeamOPS    I have been waiting for more than 40+ hours since I raised my ticket. Till now I dint get any response from the support team nor from anyone. Can you please escalate this issue...

  • 1 kudos
2 More Replies
zero234
by • New Contributor III
  • 4433 Views
  • 2 replies
  • 1 kudos

i have created a materialized view table using delta live table pipeline and its not appending data

i have created a materialized view table using delta live table pipeline , for some reason it is overwriting data every day , i want it to append data to the table instead of doing full refresh suppose i had 8 million records in table and if irun the...

Data Engineering
Reply
  • 4433 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
kulkpd
Contributor
  • 1 kudos

@zero234 ,Adding some suggestion based on answers from @Retired_mod. Important point to note here: "To define a materialized view in Python, apply @table to a query that performs a static read against a data source. To define a streaming table, apply...

  • 1 kudos
1 More Replies
alonisser
by • Contributor
  • 896 Views
  • 1 replies
  • 1 kudos

Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions

or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues

Data Engineering
Reply
  • 896 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
alonisser
Contributor
  • 1 kudos

Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...

  • 1 kudos
dzsuzs
by • New Contributor II
  • 1257 Views
  • 2 replies
  • 1 kudos

OOM Issue in Streaming with foreachBatch()

I have a stateless streaming application that uses foreachBatch. This function executes between 10-400 times each hour based on custom logic.  The logic within foreachBatch includes: collect() on very small DataFrames (a few megabytes) --> driver mem...

Data Engineering
Reply
  • 1257 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
xorbix_rshiva
Contributor
  • 1 kudos

From the information you provided, your issue might be resolved by setting a watermark on the streaming dataframe. The purpose of watermarks is to set a maximum time for records to be retained in state. Without a watermark, records in your state will...

  • 1 kudos
1 More Replies
shanebo425
by • New Contributor III
  • 1629 Views
  • 2 replies
  • 0 kudos

Saving Widgets to Git

We use Databricks widgets in our python notebooks to pass parameters in jobs but also for when we are running the notebooks manually (outside of a job context) for various reasons. We're a small team, but I've noticed that when I create a notebook an...

Data Engineering
Reply
  • 1629 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@shanebo425 You can add your widgets to the code, ex:dbutils.widgets.text("test", "") dbutils.widgets.get("test") Remember that the cell with widget needs to be run in order for widgets to be actually visible in a notebook.

  • 0 kudos
1 More Replies
avrm91
by • Contributor
  • 8991 Views
  • 2 replies
  • 0 kudos

XML DLT Autoloader - Ingestion of XML Files

I want to ingest multiple XML files with varying but similar structures without defining a schema.For example:   <?xml version="1.0" encoding="ISO-8859-1"?> <LIEFERUNG> <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH </NAME> <ST...

Data Engineering
Reply
  • 8991 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
avrm91
Contributor
  • 0 kudos

@Retired_mod Thanks a lot.I found an issue in from_xml function.I posted above: SELECT from_xml(CONCAT('<ABSENDER>', ABSENDER, '</ABSENDER>'), schema_of_xml(' <ABSENDER> <RZLZ>R00000001</RZLZ> <NAME>Informatik GmbH</NAME> <STRASSE>M...

  • 0 kudos
1 More Replies
daindana
by • New Contributor III
  • 5802 Views
  • 8 replies
  • 3 kudos

Resolved! How to preserve my database when the cluster is terminated?

Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...

Data Engineering
Reply
  • 5802 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
dhpaulino
New Contributor II
  • 3 kudos

 As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....

  • 3 kudos
7 More Replies
lnsnarayanan
by • New Contributor II
  • 10809 Views
  • 8 replies
  • 12 kudos

Resolved! I cannot see the Hive databases or tables once I terminate the cluster and use another cluster.

I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....

Data Engineering
Reply
  • 10809 Views
  • 8 replies
  • 12 kudos
View Replies
Latest Reply
dhpaulino
New Contributor II
  • 12 kudos

As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...

  • 12 kudos
7 More Replies
v01d
by • New Contributor III
  • 1373 Views
  • 1 replies
  • 0 kudos

Databricks Auto Loader authorization exception

Hello,I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.Looks like all required permission are provided to the service principle: 

Screenshot_2024-06-01_at_14_32_06.png
Data Engineering
Reply
  • 1373 Views
  • 1 replies
  • 0 kudos
View Replies
AkasBala
by • New Contributor III
  • 3063 Views
  • 3 replies
  • 0 kudos

Primary Key not working as expected on Unity Catalog delta tables

Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...

Data Engineering
Reply
  • 3063 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
paranoid_jvm
by • New Contributor II
  • 4206 Views
  • 0 replies
  • 1 kudos

Spark tasks getting stick on one executor

Hi All,I am running a Spark job using cluster with 8 executor with 8 cores each. The job involves execution of UDF. The job processes rows in few 100 thousands. When I run the job, each executor is assigned 8 job each. Usually the job succeeds in les...

Data Engineering
Reply
  • 4206 Views
  • 0 replies
  • 1 kudos
Erik
by • Valued Contributor III
  • 8268 Views
  • 7 replies
  • 10 kudos

Resolved! How to use dbx for local development.

​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing...

Data Engineering
Reply
  • 8268 Views
  • 7 replies
  • 10 kudos
View Replies
Latest Reply
FeliciaWilliam
Contributor
  • 10 kudos

I found answers to my questions here

  • 10 kudos
6 More Replies
abaet
by • New Contributor
  • 509 Views
  • 0 replies
  • 0 kudos

Random NoClassDefFound Error when running job

We are running a job on a cluster with DBR 10.4 LTS Spark 3.2.1 and scala 2.12. Cluster is using 4 workers (spot instances) .The driver is not a spot instanceRandomly ( only on one environment and not all executions), we are getting the following err...

Data Engineering
Reply
  • 509 Views
  • 0 replies
  • 0 kudos
fury-kata
by • New Contributor II
  • 1161 Views
  • 1 replies
  • 0 kudos

ModuleNotFoundError when run with foreachBatch on serverless mode

I using Notebooks to do some transformations I install a new whl:  %pip install --force-reinstall /Workspace/<my_lib>.whl %restart_python  Then I  successfully import the installed lib  from my_lib.core import test  However when I run my code with fo...

Data Engineering
Reply
  • 1161 Views
  • 1 replies
  • 0 kudos
View Replies
wilco
by • New Contributor II
  • 2176 Views
  • 2 replies
  • 0 kudos

SQL Warehouse: Retrieving SQL ARRAY Type via JDBC driver

Hi all,we are currently running into the following issuewe are using serverless SQL warehousein a JAVA application we are using the latest Databricks JDBC driver (v2.6.36)we are querying the warehouse with a collect_list function, which should return...

Data Engineering
Reply
  • 2176 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
KTheJoker
Databricks Employee
  • 0 kudos

Hey Wilco, The answer is no, ODBC/JDBC don't support complex types so these need to be compressed into strings over the wire (usually in JSON representation) and rehydrated on the client side into a complex object.

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels