cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Gilg
by Contributor II
  • 5475 Views
  • 0 replies
  • 0 kudos

Pivot in Databricks SQL

Hi Team,I have a table that has a key column (column name) and value column (value of the column name). These values are generated dynamically and wanted to pivot the table.Question 1: Is there a way that we can do this without specifying all the col...

Gilg_0-1695088239719.png
  • 5475 Views
  • 0 replies
  • 0 kudos
MauiWarrior
by New Contributor
  • 5153 Views
  • 0 replies
  • 0 kudos

Installing fpp3 R package on Databricks

In R notebook I am running:     install.packages('fpp3', dependencies = TRUE) And getting back errors:     ERROR: dependency ‘vctrs’ is not available for package ‘slider’I then install 'vctrs' and it again generates similar error that some package is...

  • 5153 Views
  • 0 replies
  • 0 kudos
data_turtle
by New Contributor
  • 1250 Views
  • 0 replies
  • 0 kudos

How do I get AWS costs from my SQL Warehouses?

Hi,How do I find the AWS associated costs from my databricks SQL warehouse usage? I tried using tags but they didn't show up in the AWS cost explorer.My use case is I am running some DBT - Databricks jobs and I want to find the cost for certain jobs....

  • 1250 Views
  • 0 replies
  • 0 kudos
Sabtout
by New Contributor II
  • 1959 Views
  • 1 replies
  • 0 kudos

Using Python UDF in Delta live table

Hello,I tried running a python UDF in a Delta Live Table workflow in Advanced mode but it did not run and gave the "Python UDF is not supported in your environment" error.Can I get a clear picture if the Python External UDFs are supported or not?

  • 1959 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sabtout
New Contributor II
  • 0 kudos

Hi @Retired_mod I ran this SQL query in my Catalog (I'm using Unity Catalog) :CREATE OR REPLACE FUNCTION cat_projint_dev.silver.GetEditor(prompt STRING)RETURNS STRINGLANGUAGE PYTHONAS $$print(prompt)$$ Then I ran a Delta Live Table workflow using Uni...

  • 0 kudos
Direo
by Contributor
  • 6037 Views
  • 1 replies
  • 0 kudos

Unexpected performance behaviors due to changes in the Spark engine or Databricks runtime

Hi!We have recently upgraded our cluster from Databricks Runtime 10.4 LTS which includes Apache Spark 3.2.1 to to Databricks Runtime 13.3 LTSincludes Apache Spark 3.2.1 powered by Apache Spark 3.3.0 and noticed that one of our jobs runtime has dramat...

Data Engineering
Cluster upgrade
crossjoin
disk keeps expanding
spark 3.2.1 vs spark 3.3.0
  • 6037 Views
  • 1 replies
  • 0 kudos
Latest Reply
Direo
Contributor
  • 0 kudos

Seems that broadcasting the smaller table in crossjoin did the magic.

  • 0 kudos
Prank
by New Contributor III
  • 6122 Views
  • 10 replies
  • 7 kudos
  • 6122 Views
  • 10 replies
  • 7 kudos
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 7 kudos

@Prank  why do you want the browser hostname?

  • 7 kudos
9 More Replies
alvaro_databric
by New Contributor III
  • 1482 Views
  • 1 replies
  • 0 kudos

Noisy error in Databricks.

Hello all, I need some help figuring out the exact cause of this error. While in the middle of a complex scala transformation notebook in Azure Databricks, from time to time i get an error which looks like this, can someone help?Job aborted due to st...

  • 1482 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello all, I need some help figuring out the exact cause of this error. While in the middle of a complex scala transformation notebook in Azure Databricks, from time to time i get an error which looks like this, can someone help?Job aborted due to st...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Gilg
by Contributor II
  • 10363 Views
  • 0 replies
  • 0 kudos

Azure Service Bus

Hi Team,My client has Azure Service Bus and wants to do streaming using DLT. In my previous client we use Event Hub to get the data. Not sure if we can do this with Service Bus.Thanks in advance,G

  • 10363 Views
  • 0 replies
  • 0 kudos
AMadan
by New Contributor II
  • 13966 Views
  • 1 replies
  • 2 kudos

temp tables in Databricks

Hey Team,We are in a migration phase from SQL server to Databricks.In our sql server databricks, lot of times, #temp tables created and dropped like typical Datawarehousing sql.In Databricks, we can do 2 things. I want to check which one is best appr...

  • 13966 Views
  • 1 replies
  • 2 kudos
Latest Reply
berserkersap
Contributor
  • 2 kudos

Just to add a bit more from experienceIf you are doing to use the temp tables too many times, then using temp views is going to slow down your cluster . It is due to lazy evaluation unless you persist that view (dataframe cache or globaltempview).Eve...

  • 2 kudos
berserkersap
by Contributor
  • 4266 Views
  • 0 replies
  • 1 kudos

Speed Up JDBC Write from Databricks Notebook to MS SQL Server

Hello Everyone,I have a use case where I need to write a delta table from DataBricks to a SQL Server Table using Pyspark/ python/ spark SQL .The delta table I am writing contains around 3 million records and the SQL Server Table is neither partitione...

Data Engineering
JDBC
MS SQL Server
pyspark
Table Write
  • 4266 Views
  • 0 replies
  • 1 kudos
p-kolluri
by New Contributor II
  • 1404 Views
  • 2 replies
  • 0 kudos

Error while installing SAT tool in azure databricks

Hi, was running the security analysis initializer notebook, am I'm getting an error in cmd 8 for import_dashboard_template notebook. That is notebook no.5. First got an error for cmd 8  for this line "workspacedf = spark.sql("select * from `global_te...

  • 1404 Views
  • 2 replies
  • 0 kudos
Latest Reply
victormarcel09
New Contributor II
  • 0 kudos

Hii,You can double-check the structure and contents of the json_ object to ensure it contains the expected keys and values. If you still encounter issues, verify the data types and perform necessary type conversions to ensure compatibility.

  • 0 kudos
1 More Replies
rahulverma22
by New Contributor
  • 763 Views
  • 1 replies
  • 0 kudos

Data Engineer Associate V3 exam got suspended

Hi team,My Databricks Certified Data Engineer Associate exam got suspended within 15 minutes. I had also shown my exam room to the proctor. My exam got suspended due to eye movement. I was not moving my eyes away from laptop screen. Some questions ar...

  • 763 Views
  • 1 replies
  • 0 kudos
Latest Reply
JodySF
Databricks Employee
  • 0 kudos

Hi @rahulverma22 , Please refer to this post for guidance on how to proceed: https://community.databricks.com/t5/certifications/suspended-exam-don-t-worry-our-support-team-can-help/m-p/44469#M352

  • 0 kudos
YosepWijaya
by New Contributor
  • 28071 Views
  • 4 replies
  • 1 kudos

How can I embed image to the cell using markdown or code?

I have been trying to embed the image from the dbfs location, when I run the code, the image is unknown or question mark. I have tried following code: The path of the file is dbfs:/FileStore/tables/svm.jpgdisplayHTML("<img src ='dbfs:/FileStore/tabl...

  • 28071 Views
  • 4 replies
  • 1 kudos
Latest Reply
AlexBrulliea
New Contributor II
  • 1 kudos

Is there a way to embed an image from mounted storage into my markdown cell? Or can this only be done using the dbfs files?

  • 1 kudos
3 More Replies
MounicaVemulapa
by New Contributor III
  • 6419 Views
  • 4 replies
  • 0 kudos

How to handle multilines coming from CSV file in a quoted string

How to handle multilines coming from CSV file in a quoted string

  • 6419 Views
  • 4 replies
  • 0 kudos
Latest Reply
mathan_pillai
Databricks Employee
  • 0 kudos

Hi, Can you try escape parameter & quote parameter to indicate which characters need to be ignored. The escape character within the quotes will be ignored. you can specify the newline character, that it needs to be ignored. please refer to below doc...

  • 0 kudos
3 More Replies
alvaro_databric
by New Contributor III
  • 13601 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks Python Import Bug

Hello,I am reaching to the community in order to shed some light into a Bug I have been encountering recently. The two setups are as follows:SETUP-1 (WORKS):Python Notebook in Workspace FileSystem (this is Shared/folder/...)Python custom wheel librar...

  • 13601 Views
  • 3 replies
  • 1 kudos
Latest Reply
alvaro_databric
New Contributor III
  • 1 kudos

Solution has comed with an update, as stated in Work with Python and R modules | Databricks on AWS, In Databricks Runtime 13.0 and above, directories added to the Python sys.path are automatically distributed to all executors in the cluster. In Datab...

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels