- 4864 Views
- 4 replies
- 3 kudos
Pyspark CSV Incorrect Count
B1123451020-502,"","{""m"": {""difference"": 60}}","","","",2022-02-12T15:40:00.783Z B1456741975-266,"","{""m"": {""difference"": 60}}","","","",2022-02-04T17:03:59.566Z B1789753479-460,"","",",","","",2022-02-18T14:46:57.332Z B1456741977-123,"","{""...
- 4864 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Tarique Anwer​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 1420 Views
- 0 replies
- 1 kudos
access to my account
access to my accountI can't access to my accountI am getting password reset link in my email but The password reset doesn't work.Invalid email address or passwordNote: Emails/usernames are case-sensitive
- 1420 Views
- 0 replies
- 1 kudos
- 5450 Views
- 6 replies
- 14 kudos
Vacuum on external tables that we mount on ADLS
Want to know the best process of removal of files on ADLS after Optimize and Vacuum Dry run is completed
- 5450 Views
- 6 replies
- 14 kudos
- 14 kudos
Hi @Ravikanth Narayanabhatla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 14 kudos
- 4101 Views
- 4 replies
- 4 kudos
Resolved! Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?
We are having an issue with calls to /api/2.1/jobs/create taking between 30 and 60 seconds to complete. Calls to /api/2.1/jobs/list complete as expected in about 6 seconds so the issue seems to be only with the job create API. This seems to only be h...
- 4101 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Matthew Dalesio​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 4 kudos
- 4898 Views
- 3 replies
- 10 kudos
I have an Azure storage account, and I need to list the containers on that storage account using scala from databricks notebook
I tried to run azure-cli on databricks (i am using Credential passthrough and my account needs MFA ) and do the following code:az storage container list --account-name "account_name" --auth-mode login --query "[].name" --output tsvHowever, it outputs...
- 4898 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @elias bou ghosn​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 10 kudos
- 3646 Views
- 2 replies
- 6 kudos
Cost as per the Databricks demo
Hi there,I came across this Databricks demo from the below link. https://youtu.be/BqB7YQ1-KKcKindly Fastforward to time 16:30 or 16:45 of the video and watch few mins of the video related to cost. My understanding is the data is in the lake and datab...
- 3646 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @AJ DJ​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 6 kudos
- 3129 Views
- 3 replies
- 13 kudos
Any Upcoming Updates to Data Engineer Professional Exam or Advanced Data Engineering Course?
Just failed the Data Engineer Professional exam... the Advanced Data Engineering prep course is totally inadequate, and the exam went way beyond in scope with some topics and questions. Are there any changes coming to the curriculum or the exam?
- 3129 Views
- 3 replies
- 13 kudos
- 13 kudos
For me, The exam actually really really hard, you must know almost all knowledgeof Databricks Lake House Platform and very deeply. Anyway, mistake make perfect keep fighting. Hope to hear from your post next time with Passed message!
- 13 kudos
- 1706 Views
- 1 replies
- 6 kudos
ML inference server (with REST api) how can I specify a GPU cluster.?
is there an API for that? as I couldn't find a way to do this through the UIClassic serving for now (didn't get access to the new "serverless" offering)
- 1706 Views
- 1 replies
- 6 kudos
- 6 kudos
any clues?
- 6 kudos
- 4700 Views
- 2 replies
- 3 kudos
spark exception error while reading a parquet file
when I try to read parquet file from Azure datalake container from databricks, I am getting spark exception. Below is my queryimport pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt...
- 4700 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @shamly pt​ , more info are needed to solve the issue. However common problems are:The storage is not mountThat file doesn't exists in the mounted storageAlso, there is no need to use an f-string if there are no curly brackets with expressions in ...
- 3 kudos
- 4564 Views
- 8 replies
- 18 kudos
Code snippet error from course - Databricks Academy - Delta Lake Rapid Start with Python
Dear Team,While i was doing hands on practice from the course - Delta Lake Rapid Start with Pythonhttps://customer-academy.databricks.com/learn/course/97/delta-lake-rapid-start-with-pythoni have come across false as the output dbutils.fs.rm(health_t...
- 4564 Views
- 8 replies
- 18 kudos
- 18 kudos
Could you give more description about your issue (screenshot or something). Hope to help you find the issue?
- 18 kudos
- 17741 Views
- 2 replies
- 1 kudos
How to convert dataframe (df), to a excel file that I can share with my colleagues ?
I am working on microsoft azure databrick, I have a final dataframe of shape (3276*23) , I want to share it in form of excel file? How can I do it ( I am using ->df.to_excel('fileOutput.xlsx', sheet_name = 'Sheet1', index = False) , command is runn...
- 17741 Views
- 2 replies
- 1 kudos
- 1 kudos
You could try this way, convert Pyspark Dataframe to Pandas Dataframe then export to excel file.
- 1 kudos
- 14125 Views
- 2 replies
- 1 kudos
Resolved! How do you read an Excel spreadsheet with Databricks
My cluster has Scala 2.12I've installed Maven Library com.crealytics:spark-excel_2.12:0.14.0I get an error java.lang.IllegalStateException: Cannot get a STRING value from a NUMERIC cellwhen trying to execute the following%pythonexcelFileName="/mnt/dl...
- 14125 Views
- 2 replies
- 1 kudos
- 1 kudos
Another way also help for your case is usign Pandas to read excel then convert Pandas Dataframe to Pyspark Dataframe
- 1 kudos
- 6862 Views
- 9 replies
- 3 kudos
Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received
Hello, I passed Databricks Certified Data Engineer Associate on 28th October 2022, but I haven't received my certificate/badge yet. Please help me with this. Best Regards,Janaka Nawarathna.
- 6862 Views
- 9 replies
- 3 kudos
- 10807 Views
- 3 replies
- 7 kudos
Resolved! Best language to use
Databricks supports SQL, Scala, Python, and R. Is there a most performant language to use on Databricks? I know SQL well but would like to get into one of the other languages and don't know which to focus on.
- 10807 Views
- 3 replies
- 7 kudos
- 977 Views
- 0 replies
- 2 kudos
Databricks SQL endpoint authentication
Hi all​I have a requirement that goes like this:Users from a particular software that doesn't have out-of-the-box integration with Databricks click on a dashboard, the button click then sends an SQL query​ to Databricks (user gets authenticated in Da...
- 977 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
