cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

User16830818469
by New Contributor
  • 2505 Views
  • 2 replies
  • 0 kudos

Databricks SQL Visualizations - export

How can I export Databricks SQL Dashboards as a .pdf?

  • 2505 Views
  • 2 replies
  • 0 kudos
Latest Reply
BigMF
New Contributor III
  • 0 kudos

Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:

  • 0 kudos
1 More Replies
su
by New Contributor
  • 2520 Views
  • 3 replies
  • 0 kudos

Reading from /tmp no longer working

Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...

  • 2520 Views
  • 3 replies
  • 0 kudos
Latest Reply
Evan_From_Bosto
New Contributor II
  • 0 kudos

I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.

  • 0 kudos
2 More Replies
francisix
by New Contributor II
  • 2029 Views
  • 4 replies
  • 1 kudos

Resolved! I haven't received badge for completion

Hi,Today I completed the test for Lakehouse fundamentals by scored 85%, still I haven't received the badge through my email francis@intellectyx.comKindly let me know please !-Francis

  • 2029 Views
  • 4 replies
  • 1 kudos
Latest Reply
AdrianLobacz
Contributor
  • 1 kudos

Hi, I have same problem but you must wait up to 24 hours and you will receive the badge

  • 1 kudos
3 More Replies
satishshravan88
by New Contributor II
  • 1027 Views
  • 3 replies
  • 2 kudos

I haven't received certificate or badge after clearing data engineer associate exam

Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273]​ Mail id : satishshravan888@gmail.comThanks,Satish

  • 1027 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Satish Shravan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies
virbickt
by New Contributor III
  • 7069 Views
  • 3 replies
  • 6 kudos

Resolved! "User not authorized" error when trying to deploy access connector on Azure

Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...

  • 7069 Views
  • 3 replies
  • 6 kudos
Latest Reply
ThomasVanBilsen
New Contributor III
  • 6 kudos

Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...

  • 6 kudos
2 More Replies
aicd_de
by New Contributor III
  • 1621 Views
  • 1 replies
  • 0 kudos

Resolved! Error Using spark.catalog.dropTempView()

I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...

  • 1621 Views
  • 1 replies
  • 0 kudos
Latest Reply
aicd_de
New Contributor III
  • 0 kudos

            spark.sql("DROP TABLE "+prefix_updates)            spark.sql("DROP TABLE "+prefix_main)Fixed it for me.

  • 0 kudos
databicky
by Contributor II
  • 10877 Views
  • 12 replies
  • 4 kudos
  • 10877 Views
  • 12 replies
  • 4 kudos
Latest Reply
FerArribas
Contributor
  • 4 kudos

Hi @Hubert Dudek​,​Pandas API doesn't support abfss protocol.You have three options:​If you need to use pandas, you can write the excel to the local file system (dbfs) and then move it to ABFSS (for example with dbutils)Write as csv directly in abfss...

  • 4 kudos
11 More Replies
isaac_gritz
by Valued Contributor II
  • 9311 Views
  • 6 replies
  • 6 kudos

Local Development on Databricks

How to Develop Locally on Databricks with your Favorite IDEdbx is a Databricks Labs project that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS | Azure | GC...

  • 9311 Views
  • 6 replies
  • 6 kudos
Latest Reply
Jfoxyyc
Valued Contributor
  • 6 kudos

I'm actually not a fan of dbx. I prefer the AWS Glue interactive sessions way of using the IDE. It's exactly like the web notebook experience. I can see the reason why dbx exists, but I'd still like to use a regular notebook experience in my IDE.

  • 6 kudos
5 More Replies
antoniosisba96
by New Contributor II
  • 1705 Views
  • 4 replies
  • 4 kudos

Passed Data Engineer Associate Exam but received twice the Lakehouse Accreditation

Hi all,today (27/12/22 14:00 Rome Time Zone) I passed the Data Engineer Associate exam, but I received the badge of Lakehouse Fundamentals (second time).   My email address is: sisbarra@gmail.comMy company address is: antonio.sisbarra@nttdata.comCan ...

Exam Passed Digital Badge Wrong Twice the same Badge
  • 1705 Views
  • 4 replies
  • 4 kudos
Latest Reply
Nadia1
Honored Contributor
  • 4 kudos

Hello Antonio,I deleted the badge under: antonio.sisbarra@nttdata.com . You are good to go.Thanks!

  • 4 kudos
3 More Replies
Riddhi
by New Contributor III
  • 3678 Views
  • 9 replies
  • 14 kudos

Resolved! Databricks Lakehouse Fundamentals Accreditation V2 badge/certificate not received.

Hello, this is regarding Databricks Lakehouse Fundamentals Accreditation V2. I haven't received my badge/certificate. I also raised a ticket but haven't received any response. My request id is #00248504.​. Kindly help me out with this.

Databricks Lakehouse Exam Score Screenshot 1
  • 3678 Views
  • 9 replies
  • 14 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 14 kudos

@Jose Gonzalez​ ,@Vidula Khanna​ ,I also did not received my badge in portalThis is my portal link- https://credentials.databricks.com/profile/aviralbhardwaj143185/walletMy case number is - 00250939Also, my points are not updating in the reward port...

  • 14 kudos
8 More Replies
debanjan89
by New Contributor II
  • 1765 Views
  • 3 replies
  • 2 kudos

How do we concatenate some fixed string with a secret value in Spark Config in Databricks Job Cluster?

Hi Team,I am trying to configure access to adls through Service Principal through Spark Config in Databricks job cluster. like, fs.azure.account.oauth2.client.id.<adls_account_name>.dfs.core.windows.net {{secrets/scopeName/clientID}}The above stateme...

  • 1765 Views
  • 3 replies
  • 2 kudos
Latest Reply
Manimkm08
New Contributor III
  • 2 kudos

@Kaniz Fatma​ We are blocked on this issue. Can you please look into the thread and give your suggestion to workaround it.

  • 2 kudos
2 More Replies
Mado
by Valued Contributor II
  • 12051 Views
  • 1 replies
  • 0 kudos

Resolved! How to show all rows by "DataFrame.show()"?

Hi,DataFrame.show() has a parameter n to set "Number of rows to show".Is there any way to show all rows?

  • 12051 Views
  • 1 replies
  • 0 kudos
Latest Reply
sher
Valued Contributor II
  • 0 kudos

Hi Medothis method will work fine df.show(df.count())

  • 0 kudos
Jyo777
by Contributor
  • 1167 Views
  • 2 replies
  • 3 kudos

Resolved! Can't do "Full screen" while taking Databricks Apache Spark developer course.

Hi, I see the option for "Full screen" on bottom right but its disabled/inactive. Attached is the screenshot for same.Please advise as its hard to read or see contents on half screen.Thanks

  • 1167 Views
  • 2 replies
  • 3 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

press F11 button it will become full screen

  • 3 kudos
1 More Replies
semi
by New Contributor II
  • 1281 Views
  • 3 replies
  • 3 kudos

Access file location problem

import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json")   SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...

  • 1281 Views
  • 3 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

Looks like it is because the oauth2client.service_account does not know about DBFS (whereas spark does).Is it an option to manage your secrets in databricks? https://docs.databricks.com/security/secrets/secrets.html

  • 3 kudos
2 More Replies
Spauk
by New Contributor II
  • 9157 Views
  • 5 replies
  • 7 kudos

Resolved! Best Practices for naming Tables and Databases in Databricks

We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...

  • 9157 Views
  • 5 replies
  • 7 kudos
Latest Reply
LandanG
Honored Contributor
  • 7 kudos

Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...

  • 7 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels