cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Anonymous
by Not applicable
  • 1444 Views
  • 2 replies
  • 16 kudos

Resolved! How to access data files for Databricks Workspace directly through Azure Blob Storage

Hi everyone,This is the first time I used Azure to deploy a Databricks (before I quite familiar use AWS for deploying Databricks). I want to view Databricks Workspace file directly from Azure portal but always don't have a permission.Could you give m...

Image
  • 1444 Views
  • 2 replies
  • 16 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 16 kudos

@Jensen Ackles​ read document as link below :https://docs.databricks.com/external-data/azure-storage.htmli dont know whats steps ur done, hope this can help u in this case

  • 16 kudos
1 More Replies
TariqueAnwer
by New Contributor II
  • 3121 Views
  • 5 replies
  • 3 kudos

Pyspark CSV Incorrect Count

B1123451020-502,"","{""m"": {""difference"": 60}}","","","",2022-02-12T15:40:00.783Z B1456741975-266,"","{""m"": {""difference"": 60}}","","","",2022-02-04T17:03:59.566Z B1789753479-460,"","",",","","",2022-02-18T14:46:57.332Z B1456741977-123,"","{""...

  • 3121 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Tarique Anwer​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 3 kudos
4 More Replies
ravikanthranjit
by New Contributor III
  • 3017 Views
  • 6 replies
  • 14 kudos

Vacuum on external tables that we mount on ADLS

Want to know the best process of removal of files on ADLS after Optimize and Vacuum Dry run is completed

  • 3017 Views
  • 6 replies
  • 14 kudos
Latest Reply
Anonymous
Not applicable
  • 14 kudos

Hi @Ravikanth Narayanabhatla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...

  • 14 kudos
5 More Replies
Matt101122
by Contributor
  • 2259 Views
  • 4 replies
  • 4 kudos

Resolved! Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

We are having an issue with calls to /api/2.1/jobs/create taking between 30 and 60 seconds to complete. Calls to /api/2.1/jobs/list complete as expected in about 6 seconds so the issue seems to be only with the job create API. This seems to only be h...

  • 2259 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Matthew Dalesio​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 4 kudos
3 More Replies
ebg
by New Contributor III
  • 3232 Views
  • 3 replies
  • 10 kudos

I have an Azure storage account, and I need to list the containers on that storage account using scala from databricks notebook

I tried to run azure-cli on databricks (i am using Credential passthrough and my account needs MFA ) and do the following code:az storage container list --account-name "account_name" --auth-mode login --query "[].name" --output tsvHowever, it outputs...

  • 3232 Views
  • 3 replies
  • 10 kudos
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @elias bou ghosn​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 10 kudos
2 More Replies
AJDJ
by New Contributor III
  • 2572 Views
  • 5 replies
  • 10 kudos

Cost as per the Databricks demo

Hi there,I came across this Databricks demo from the below link. https://youtu.be/BqB7YQ1-KKcKindly Fastforward to time 16:30 or 16:45 of the video and watch few mins of the video related to cost. My understanding is the data is in the lake and datab...

  • 2572 Views
  • 5 replies
  • 10 kudos
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @AJ DJ​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 10 kudos
4 More Replies
smart5mk
by New Contributor III
  • 1663 Views
  • 3 replies
  • 13 kudos

Any Upcoming Updates to Data Engineer Professional Exam or Advanced Data Engineering Course?

Just failed the Data Engineer Professional exam... the Advanced Data Engineering prep course is totally inadequate, and the exam went way beyond in scope with some topics and questions. Are there any changes coming to the curriculum or the exam?

  • 1663 Views
  • 3 replies
  • 13 kudos
Latest Reply
Anonymous
Not applicable
  • 13 kudos

For me, The exam actually really really hard, you must know almost all knowledgeof Databricks Lake House Platform and very deeply. Anyway, mistake make perfect keep fighting. Hope to hear from your post next time with Passed message!

  • 13 kudos
2 More Replies
shamly
by New Contributor III
  • 3058 Views
  • 2 replies
  • 3 kudos

spark exception error while reading a parquet file

when I try to read parquet file from Azure datalake container from databricks, I am getting spark exception. Below is my queryimport pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt...

  • 3058 Views
  • 2 replies
  • 3 kudos
Latest Reply
DavideAnghileri
Contributor
  • 3 kudos

Hi @shamly pt​ , more info are needed to solve the issue. However common problems are:The storage is not mountThat file doesn't exists in the mounted storageAlso, there is no need to use an f-string if there are no curly brackets with expressions in ...

  • 3 kudos
1 More Replies
db-avengers2rul
by Contributor II
  • 2639 Views
  • 8 replies
  • 18 kudos

Code snippet error from course - Databricks Academy - Delta Lake Rapid Start with Python

Dear Team,While i was doing hands on practice from the course - Delta Lake Rapid Start with Pythonhttps://customer-academy.databricks.com/learn/course/97/delta-lake-rapid-start-with-pythoni have come across false as the output dbutils.fs.rm(health_t...

  • 2639 Views
  • 8 replies
  • 18 kudos
Latest Reply
Anonymous
Not applicable
  • 18 kudos

Could you give more description about your issue (screenshot or something). Hope to help you find the issue?

  • 18 kudos
7 More Replies
rajat1
by New Contributor
  • 12679 Views
  • 3 replies
  • 2 kudos

How to convert dataframe (df), to a excel file that I can share with my colleagues ?

I am working on microsoft azure databrick, I have a final dataframe of shape (3276*23) , I want to share it in form of excel file? How can I do it ( I am using ->df.to_excel('fileOutput.xlsx', sheet_name = 'Sheet1', index = False) , command is runn...

  • 12679 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

You could try this way, convert Pyspark Dataframe to Pandas Dataframe then export to excel file.

  • 2 kudos
2 More Replies
LPlates
by New Contributor III
  • 10073 Views
  • 3 replies
  • 1 kudos

Resolved! How do you read an Excel spreadsheet with Databricks

My cluster has Scala 2.12I've installed Maven Library com.crealytics:spark-excel_2.12:0.14.0I get an error java.lang.IllegalStateException: Cannot get a STRING value from a NUMERIC cellwhen trying to execute the following%pythonexcelFileName="/mnt/dl...

  • 10073 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Another way also help for your case is usign Pandas to read excel then convert Pandas Dataframe to Pyspark Dataframe

  • 1 kudos
2 More Replies
JanakaNaw
by New Contributor II
  • 4063 Views
  • 9 replies
  • 3 kudos

Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received

Hello, I passed Databricks Certified Data Engineer Associate on 28th October 2022, but I haven't received my certificate/badge yet. Please help me with this. Best Regards,Janaka Nawarathna.

  • 4063 Views
  • 9 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Have you received your badge?

  • 3 kudos
8 More Replies
Ryan_Chynoweth
by Esteemed Contributor
  • 6992 Views
  • 3 replies
  • 7 kudos

Resolved! Best language to use

Databricks supports SQL, Scala, Python, and R. Is there a most performant language to use on Databricks? I know SQL well but would like to get into one of the other languages and don't know which to focus on.

  • 6992 Views
  • 3 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

It total depends on you? BTW, you can choose Python and SQL

  • 7 kudos
2 More Replies
NOOR_BASHASHAIK
by Contributor
  • 579 Views
  • 0 replies
  • 2 kudos

Databricks SQL endpoint authentication

Hi all​I have a requirement that goes like this:Users from a particular software that doesn't have out-of-the-box integration with Databricks click on a dashboard, the button click then sends an SQL query​ to Databricks (user gets authenticated in Da...

  • 579 Views
  • 0 replies
  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels