cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
silvadev
by • New Contributor III
  • 8523 Views
  • 1 replies
  • 0 kudos

Resolved! MongoDB Spark Connector v10.x read error on Databricks 13.x

I have facing a error when I am trying to read data from any MongoDB collection using MongoDB Spark Connector v10.x on Databricks v13.x.The below error appear to start at line #113 of MongoDB Spark Connector Library (v10.2.0):  java.lang.NoSuchMethod...

Data Engineering
mongodb
spark
Reply
  • 8523 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
silvadev
New Contributor III
  • 0 kudos

The problem was fixed in Databricks Runtime 13.3 LTS.

  • 0 kudos
jonathan-dufaul
by • Valued Contributor
  • 3889 Views
  • 2 replies
  • 0 kudos

Resolved! Error updating workflow, webhook not found?

I have no idea what this error means or what it could mean. When I'm trying to save a workflow I get a popup saying this:

image
Data Engineering
Reply
  • 3889 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Robin_LOCHE
New Contributor II
  • 0 kudos

I had the same issue, thanks for the info! Apparently it's also possible to fix it by removing all the actual notification in the interface (the bugged one is not displayed, but if you remove everything for some reason it removes the bugged one too)....

  • 0 kudos
1 More Replies
DJey
by • New Contributor III
  • 13269 Views
  • 7 replies
  • 0 kudos

connect to azure sql database from databricks using service principal

Hi All, Can someone please help me with the Python code to connect Azure SQL Database to Databricks using Service Principle instead of directly passing username and password. I'm using above code but getting above error. Refer Screenshot 2.Please hel...

DJey_0-1688048752356.png DJey_1-1688048784120.png
Data Engineering
Azure Databricks
Azure SQL Database
Databricks
Reply
  • 13269 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Joe_Suarez
New Contributor III
  • 0 kudos

First, you need to create a service principal in Azure and grant it the necessary permissions to access your Azure SQL Database to do crm data enrichment. You can do this using the Azure CLI or the Azure Portal. Ensure that your Databricks cluster ha...

  • 0 kudos
6 More Replies
User16776430979
by • New Contributor III
  • 47080 Views
  • 3 replies
  • 5 kudos

Best practices around bronze/silver/gold (medallion model) data lake classification?

What's the best way to organize our data lake and delta setup? We’re trying to use the bronze, silver and gold classification strategy. The main question is how do we know what classification the data is inside Databricks if there’s no actual physica...

Data Engineering
Reply
  • 47080 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

with Unity taking into account, it is certainly a good idea to think about your physical data storage.As you cannot have overlap between volumes and tables this can become cumbersome.F.e. we used to store delta tables of a data object in the same dir...

  • 5 kudos
2 More Replies
JessGa
by • New Contributor II
  • 2741 Views
  • 3 replies
  • 0 kudos

Requesting an exam reattempt

Hi DB SupportI gave Databricks Certified Associate Engineering Exam today but missed by just by one percent. I got 68.88% and pass is 70%.I am planning to reattempt this exam in coming days and was hoping you could help, Could you kindly give me anot...

Data Engineering
Reply
  • 2741 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Littlereb5
New Contributor II
  • 0 kudos

I recommend reaching out to Databricks directly or checking their official certification website for information on retake policies, voucher availability, and any discounts or promotions they may offer for reattempts.

  • 0 kudos
2 More Replies
Gilg
by • Contributor II
  • 5711 Views
  • 0 replies
  • 0 kudos

Pivot in Databricks SQL

Hi Team,I have a table that has a key column (column name) and value column (value of the column name). These values are generated dynamically and wanted to pivot the table.Question 1: Is there a way that we can do this without specifying all the col...

Gilg_0-1695088239719.png
Data Engineering
Reply
  • 5711 Views
  • 0 replies
  • 0 kudos
MauiWarrior
by • New Contributor
  • 5185 Views
  • 0 replies
  • 0 kudos

Installing fpp3 R package on Databricks

In R notebook I am running:     install.packages('fpp3', dependencies = TRUE) And getting back errors:     ERROR: dependency ‘vctrs’ is not available for package ‘slider’I then install 'vctrs' and it again generates similar error that some package is...

Data Engineering
Reply
  • 5185 Views
  • 0 replies
  • 0 kudos
data_turtle
by • New Contributor
  • 1280 Views
  • 0 replies
  • 0 kudos

How do I get AWS costs from my SQL Warehouses?

Hi,How do I find the AWS associated costs from my databricks SQL warehouse usage? I tried using tags but they didn't show up in the AWS cost explorer.My use case is I am running some DBT - Databricks jobs and I want to find the cost for certain jobs....

Data Engineering
Reply
  • 1280 Views
  • 0 replies
  • 0 kudos
Sabtout
by • New Contributor II
  • 2014 Views
  • 1 replies
  • 0 kudos

Using Python UDF in Delta live table

Hello,I tried running a python UDF in a Delta Live Table workflow in Advanced mode but it did not run and gave the "Python UDF is not supported in your environment" error.Can I get a clear picture if the Python External UDFs are supported or not?

Data Engineering
Reply
  • 2014 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Sabtout
New Contributor II
  • 0 kudos

Hi @Retired_mod I ran this SQL query in my Catalog (I'm using Unity Catalog) :CREATE OR REPLACE FUNCTION cat_projint_dev.silver.GetEditor(prompt STRING)RETURNS STRINGLANGUAGE PYTHONAS $$print(prompt)$$ Then I ran a Delta Live Table workflow using Uni...

  • 0 kudos
Direo
by • Contributor
  • 6140 Views
  • 1 replies
  • 0 kudos

Unexpected performance behaviors due to changes in the Spark engine or Databricks runtime

Hi!We have recently upgraded our cluster from Databricks Runtime 10.4 LTS which includes Apache Spark 3.2.1 to to Databricks Runtime 13.3 LTSincludes Apache Spark 3.2.1 powered by Apache Spark 3.3.0 and noticed that one of our jobs runtime has dramat...

Data Engineering
Cluster upgrade
crossjoin
disk keeps expanding
spark 3.2.1 vs spark 3.3.0
Reply
  • 6140 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Direo
Contributor
  • 0 kudos

Seems that broadcasting the smaller table in crossjoin did the magic.

  • 0 kudos
Prank
by • New Contributor III
  • 6378 Views
  • 10 replies
  • 7 kudos

Facing issue in the below mentioned code: dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().get('browserHostName').get() while executing through workflow, but it's totally working fine with workspace.

Please do let me know how to resolve this issue?

Data Engineering
Reply
  • 6378 Views
  • 10 replies
  • 7 kudos
View Replies
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 7 kudos

@Prank  why do you want the browser hostname?

  • 7 kudos
9 More Replies
alvaro_databric
by • New Contributor III
  • 1527 Views
  • 1 replies
  • 0 kudos

Noisy error in Databricks.

Hello all, I need some help figuring out the exact cause of this error. While in the middle of a complex scala transformation notebook in Azure Databricks, from time to time i get an error which looks like this, can someone help?Job aborted due to st...

Data Engineering
Reply
  • 1527 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello all, I need some help figuring out the exact cause of this error. While in the middle of a complex scala transformation notebook in Azure Databricks, from time to time i get an error which looks like this, can someone help?Job aborted due to st...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Gilg
by • Contributor II
  • 10540 Views
  • 0 replies
  • 0 kudos

Azure Service Bus

Hi Team,My client has Azure Service Bus and wants to do streaming using DLT. In my previous client we use Event Hub to get the data. Not sure if we can do this with Service Bus.Thanks in advance,G

Data Engineering
Reply
  • 10540 Views
  • 0 replies
  • 0 kudos
AMadan
by • New Contributor III
  • 14535 Views
  • 1 replies
  • 2 kudos

temp tables in Databricks

Hey Team,We are in a migration phase from SQL server to Databricks.In our sql server databricks, lot of times, #temp tables created and dropped like typical Datawarehousing sql.In Databricks, we can do 2 things. I want to check which one is best appr...

Data Engineering
Reply
  • 14535 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
berserkersap
Contributor
  • 2 kudos

Just to add a bit more from experienceIf you are doing to use the temp tables too many times, then using temp views is going to slow down your cluster . It is due to lazy evaluation unless you persist that view (dataframe cache or globaltempview).Eve...

  • 2 kudos
berserkersap
by • Contributor
  • 4441 Views
  • 0 replies
  • 1 kudos

Speed Up JDBC Write from Databricks Notebook to MS SQL Server

Hello Everyone,I have a use case where I need to write a delta table from DataBricks to a SQL Server Table using Pyspark/ python/ spark SQL .The delta table I am writing contains around 3 million records and the SQL Server Table is neither partitione...

Data Engineering
JDBC
MS SQL Server
pyspark
Table Write
Reply
  • 4441 Views
  • 0 replies
  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels