cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
antoniosisba96
by • New Contributor II
  • 3042 Views
  • 4 replies
  • 4 kudos

Passed Data Engineer Associate Exam but received twice the Lakehouse Accreditation

Hi all,today (27/12/22 14:00 Rome Time Zone) I passed the Data Engineer Associate exam, but I received the badge of Lakehouse Fundamentals (second time).   My email address is: sisbarra@gmail.comMy company address is: antonio.sisbarra@nttdata.comCan ...

Exam Passed Digital Badge Wrong Twice the same Badge
Data Engineering
Reply
  • 3042 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Nadia1
Databricks Employee
  • 4 kudos

Hello Antonio,I deleted the badge under: antonio.sisbarra@nttdata.com . You are good to go.Thanks!

  • 4 kudos
3 More Replies
Riddhi
by • New Contributor III
  • 6626 Views
  • 9 replies
  • 14 kudos

Resolved! Databricks Lakehouse Fundamentals Accreditation V2 badge/certificate not received.

Hello, this is regarding Databricks Lakehouse Fundamentals Accreditation V2. I haven't received my badge/certificate. I also raised a ticket but haven't received any response. My request id is #00248504.​. Kindly help me out with this.

Databricks Lakehouse Exam Score Screenshot 1
Data Engineering
Reply
  • 6626 Views
  • 9 replies
  • 14 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 14 kudos

@Jose Gonzalez​ ,@Vidula Khanna​ ,I also did not received my badge in portalThis is my portal link- https://credentials.databricks.com/profile/aviralbhardwaj143185/walletMy case number is - 00250939Also, my points are not updating in the reward port...

  • 14 kudos
8 More Replies
debanjan89
by • New Contributor II
  • 2898 Views
  • 3 replies
  • 2 kudos

How do we concatenate some fixed string with a secret value in Spark Config in Databricks Job Cluster?

Hi Team,I am trying to configure access to adls through Service Principal through Spark Config in Databricks job cluster. like, fs.azure.account.oauth2.client.id.<adls_account_name>.dfs.core.windows.net {{secrets/scopeName/clientID}}The above stateme...

Data Engineering
Reply
  • 2898 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Manimkm08
New Contributor III
  • 2 kudos

@Kaniz Fatma​ We are blocked on this issue. Can you please look into the thread and give your suggestion to workaround it.

  • 2 kudos
2 More Replies
Mado
by • Valued Contributor II
  • 23673 Views
  • 1 replies
  • 0 kudos

Resolved! How to show all rows by "DataFrame.show()"?

Hi,DataFrame.show() has a parameter n to set "Number of rows to show".Is there any way to show all rows?

Data Engineering
Reply
  • 23673 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sher
Valued Contributor II
  • 0 kudos

Hi Medothis method will work fine df.show(df.count())

  • 0 kudos
Jyo777
by • Contributor
  • 2115 Views
  • 2 replies
  • 3 kudos

Resolved! Can't do "Full screen" while taking Databricks Apache Spark developer course.

Hi, I see the option for "Full screen" on bottom right but its disabled/inactive. Attached is the screenshot for same.Please advise as its hard to read or see contents on half screen.Thanks

Data Engineering
Reply
  • 2115 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

press F11 button it will become full screen

  • 3 kudos
1 More Replies
semi
by • New Contributor II
  • 2283 Views
  • 3 replies
  • 3 kudos

Access file location problem

import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json")   SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...

Data Engineering
Reply
  • 2283 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

Looks like it is because the oauth2client.service_account does not know about DBFS (whereas spark does).Is it an option to manage your secrets in databricks? https://docs.databricks.com/security/secrets/secrets.html

  • 3 kudos
2 More Replies
Spauk
by • New Contributor II
  • 19408 Views
  • 5 replies
  • 7 kudos

Resolved! Best Practices for naming Tables and Databases in Databricks

We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...

Data Engineering
Reply
  • 19408 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
LandanG
Databricks Employee
  • 7 kudos

Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...

  • 7 kudos
4 More Replies
jonathan-dufaul
by • Valued Contributor
  • 1900 Views
  • 3 replies
  • 3 kudos

Resolved! Why does chaining spark.read from one system/driver and .write to another system/driver take so much longer than doing each piece individually?

i am reading data from IBM DB2 and saving into a MS SQL server (the first step is moving the code itself to databricks, and then we will move the databases to databricks itself). Problem I'm running into is doing something like the below will take > ...

Data Engineering
Reply
  • 1900 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Hi, it is related to partitioning optimization. By default, the JDBC driver queries the source database with only a single thread. So write was from one partition as one partition was created, so it was using a single core. When you used pandas, it d...

  • 3 kudos
2 More Replies
J15S
by • New Contributor III
  • 1973 Views
  • 4 replies
  • 4 kudos

RStudio on Databricks user experience

Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...

Data Engineering
Reply
  • 1973 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
J15S
New Contributor III
  • 4 kudos

@Jonathan Dufault​ Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integ...

  • 4 kudos
3 More Replies
Prototype998
by • New Contributor III
  • 1446 Views
  • 0 replies
  • 0 kudos

Singleton Design Principle for pyspark database connector A singleton is a design pattern that ensures that a class has only one instance, and provide...

Singleton Design Principle for pyspark database connectorA singleton is a design pattern that ensures that a class has only one instance, and provides a global access point to that instance. Here is an example of how you could implement a singleton d...

Data Engineering
Reply
  • 1446 Views
  • 0 replies
  • 0 kudos
Jfoxyyc
by • Valued Contributor
  • 2181 Views
  • 2 replies
  • 2 kudos

How to use partial_parse.msgpack with workflow dbt task?

I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...

Data Engineering
Reply
  • 2181 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?

  • 2 kudos
1 More Replies
KVNARK
by • Honored Contributor II
  • 3513 Views
  • 4 replies
  • 6 kudos

Resolved! Connecting azure synapse through data bricks note books

Hi All, Happy new year!how can we connect to azure synapse serverless sql pool through databricks notebooks and execute DDLs

Data Engineering
Reply
  • 3513 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 6 kudos

@KVNARK .​ https://joeho.xyz/blog-posts/how-to-connect-to-azure-synapse-in-azure-databricks/

  • 6 kudos
3 More Replies
APol
by • New Contributor II
  • 2986 Views
  • 2 replies
  • 2 kudos

Read/Write concurrency issue

Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...

Data Engineering
Reply
  • 2986 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
FerArribas
Contributor
  • 2 kudos

Hi @Anastasiia Polianska​,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...

  • 2 kudos
1 More Replies
maddy_081063
by • New Contributor II
  • 4616 Views
  • 2 replies
  • 4 kudos

Is there a way to automate the Azure Databricks dashboard to schedule and send an email output with the dashboard?

Data Engineering
Reply
  • 4616 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
FerArribas
Contributor
  • 4 kudos

Hi @maddy v​ ,I recommend that you use the Databricks SQL module for this type of reports and email alerts. It is a very interesting module with multiple options for your use case.https://learn.microsoft.com/en-us/azure/databricks/sql/user/dashboards...

  • 4 kudos
1 More Replies
pvm26042000
by • New Contributor III
  • 1050 Views
  • 1 replies
  • 3 kudos

Spark SQL & Spark ML

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank you ...

Data Engineering
Reply
  • 1050 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.

  • 3 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels