cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
smart5mk
by • New Contributor III
  • 1557 Views
  • 3 replies
  • 13 kudos

Any Upcoming Updates to Data Engineer Professional Exam or Advanced Data Engineering Course?

Just failed the Data Engineer Professional exam... the Advanced Data Engineering prep course is totally inadequate, and the exam went way beyond in scope with some topics and questions. Are there any changes coming to the curriculum or the exam?

Data Engineering
Reply
  • 1557 Views
  • 3 replies
  • 13 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 13 kudos

For me, The exam actually really really hard, you must know almost all knowledgeof Databricks Lake House Platform and very deeply. Anyway, mistake make perfect keep fighting. Hope to hear from your post next time with Passed message!

  • 13 kudos
2 More Replies
shamly
by • New Contributor III
  • 2914 Views
  • 2 replies
  • 3 kudos

spark exception error while reading a parquet file

when I try to read parquet file from Azure datalake container from databricks, I am getting spark exception. Below is my queryimport pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt...

Data Engineering
Reply
  • 2914 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
DavideAnghileri
Contributor
  • 3 kudos

Hi @shamly pt​ , more info are needed to solve the issue. However common problems are:The storage is not mountThat file doesn't exists in the mounted storageAlso, there is no need to use an f-string if there are no curly brackets with expressions in ...

  • 3 kudos
1 More Replies
db-avengers2rul
by • Contributor II
  • 2404 Views
  • 8 replies
  • 18 kudos

Code snippet error from course - Databricks Academy - Delta Lake Rapid Start with Python

Dear Team,While i was doing hands on practice from the course - Delta Lake Rapid Start with Pythonhttps://customer-academy.databricks.com/learn/course/97/delta-lake-rapid-start-with-pythoni have come across false as the output dbutils.fs.rm(health_t...

Data Engineering
Reply
  • 2404 Views
  • 8 replies
  • 18 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 18 kudos

Could you give more description about your issue (screenshot or something). Hope to help you find the issue?

  • 18 kudos
7 More Replies
rajat1
by • New Contributor
  • 12143 Views
  • 3 replies
  • 2 kudos

How to convert dataframe (df), to a excel file that I can share with my colleagues ?

I am working on microsoft azure databrick, I have a final dataframe of shape (3276*23) , I want to share it in form of excel file? How can I do it ( I am using ->df.to_excel('fileOutput.xlsx', sheet_name = 'Sheet1', index = False) , command is runn...

Data Engineering
Reply
  • 12143 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

You could try this way, convert Pyspark Dataframe to Pandas Dataframe then export to excel file.

  • 2 kudos
2 More Replies
LPlates
by • New Contributor III
  • 9591 Views
  • 3 replies
  • 1 kudos

Resolved! How do you read an Excel spreadsheet with Databricks

My cluster has Scala 2.12I've installed Maven Library com.crealytics:spark-excel_2.12:0.14.0I get an error java.lang.IllegalStateException: Cannot get a STRING value from a NUMERIC cellwhen trying to execute the following%pythonexcelFileName="/mnt/dl...

Data Engineering
Reply
  • 9591 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Another way also help for your case is usign Pandas to read excel then convert Pandas Dataframe to Pyspark Dataframe

  • 1 kudos
2 More Replies
JanakaNaw
by • New Contributor II
  • 3765 Views
  • 9 replies
  • 3 kudos

Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received

Hello, I passed Databricks Certified Data Engineer Associate on 28th October 2022, but I haven't received my certificate/badge yet. Please help me with this. Best Regards,Janaka Nawarathna.

Data Engineering
Reply
  • 3765 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Have you received your badge?

  • 3 kudos
8 More Replies
Ryan_Chynoweth
by • Honored Contributor III
  • 6613 Views
  • 3 replies
  • 7 kudos

Resolved! Best language to use

Databricks supports SQL, Scala, Python, and R. Is there a most performant language to use on Databricks? I know SQL well but would like to get into one of the other languages and don't know which to focus on.

Data Engineering
Reply
  • 6613 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

It total depends on you? BTW, you can choose Python and SQL

  • 7 kudos
2 More Replies
NOOR_BASHASHAIK
by • Contributor
  • 538 Views
  • 0 replies
  • 2 kudos

Databricks SQL endpoint authentication

Hi all​I have a requirement that goes like this:Users from a particular software that doesn't have out-of-the-box integration with Databricks click on a dashboard, the button click then sends an SQL query​ to Databricks (user gets authenticated in Da...

Data Engineering
Reply
  • 538 Views
  • 0 replies
  • 2 kudos
Ruby8376
by • Valued Contributor
  • 596 Views
  • 0 replies
  • 2 kudos

Tableau analytics integration with databricks delta lake

Hii there!!Currently, we are exploring options for reporting on Salesforce. We extract data from salesforce via databricks and store it in delta lake.Is there a connector by which data can be pulled from databricks into Tableau/CRM analytics??I know ...

Data Engineering
Reply
  • 596 Views
  • 0 replies
  • 2 kudos
karthik_p
by • Esteemed Contributor
  • 720 Views
  • 1 replies
  • 7 kudos

accounts.cloud.databricks.com

How to Enable Features in Data bricks to explicitly opt-in As a Databricks user, sometimes we may want to use account features that are supported for all workspaces under that account. once we enable them, then users from workspaces will be able to u...

image
Data Engineering
Reply
  • 720 Views
  • 1 replies
  • 7 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 7 kudos

Thank you for this amazing post @karthik p​! Keep sharing such valuable information on the Community.

  • 7 kudos
wyzer
by • Contributor II
  • 3795 Views
  • 2 replies
  • 12 kudos

Resolved! Add the creation date of a parquet file into a DataFrame

Currently I load multiple parquet file with this code:df = spark.read.parquet("/mnt/dev/bronze/Voucher/*/*")(Inside the Voucher folder, there is one folder by date. Each one containing one parquet file)How can I add a column into this DataFrame, that...

Data Engineering
Reply
  • 3795 Views
  • 2 replies
  • 12 kudos
View Replies
Latest Reply
wyzer
Contributor II
  • 12 kudos

Thanks @Michail Karamanos​ 

  • 12 kudos
1 More Replies
Yaswanth
by • New Contributor III
  • 16469 Views
  • 5 replies
  • 18 kudos

Resolved! How can Delta table protocol version be downgraded from higher version to lower version the table properties minReader from 2 to 1 and MaxWriter from 5 to 3.

Is there a possibility to downgrade the Delta Table protocol versions minReader from 2 to 1 and maxWriter from 5 to 3? I have set the TBL properties to 2 and 5 and columnmapping mode to rename the columns in the DeltaTable but the other users are rea...

Data Engineering
Reply
  • 16469 Views
  • 5 replies
  • 18 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 18 kudos

Hi @Yaswanth velkur, We haven’t heard from you since the last response from @Youssef Mrini​ and me​​, and I was checking back to see if our suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 18 kudos
4 More Replies
chiragnayyar
by • Contributor
  • 970 Views
  • 1 replies
  • 3 kudos

Resolved! In person Databricks meetup in Singapore?

Hi I would like to know if anyone interested to volunteer in person Databricks meetup.Please share your thoughts, and we can talk further about the logistics Thank you

Data Engineering
Reply
  • 970 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Nice initiative @Chirag Nayyar​!

  • 3 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels