- 1444 Views
- 2 replies
- 16 kudos
Resolved! How to access data files for Databricks Workspace directly through Azure Blob Storage
Hi everyone,This is the first time I used Azure to deploy a Databricks (before I quite familiar use AWS for deploying Databricks). I want to view Databricks Workspace file directly from Azure portal but always don't have a permission.Could you give m...
- 1444 Views
- 2 replies
- 16 kudos
- 16 kudos
@Jensen Ackles​ read document as link below :https://docs.databricks.com/external-data/azure-storage.htmli dont know whats steps ur done, hope this can help u in this case
- 16 kudos
- 3121 Views
- 5 replies
- 3 kudos
Pyspark CSV Incorrect Count
B1123451020-502,"","{""m"": {""difference"": 60}}","","","",2022-02-12T15:40:00.783Z B1456741975-266,"","{""m"": {""difference"": 60}}","","","",2022-02-04T17:03:59.566Z B1789753479-460,"","",",","","",2022-02-18T14:46:57.332Z B1456741977-123,"","{""...
- 3121 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Tarique Anwer​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 3017 Views
- 6 replies
- 14 kudos
Vacuum on external tables that we mount on ADLS
Want to know the best process of removal of files on ADLS after Optimize and Vacuum Dry run is completed
- 3017 Views
- 6 replies
- 14 kudos
- 14 kudos
Hi @Ravikanth Narayanabhatla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 14 kudos
- 2259 Views
- 4 replies
- 4 kudos
Resolved! Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?
We are having an issue with calls to /api/2.1/jobs/create taking between 30 and 60 seconds to complete. Calls to /api/2.1/jobs/list complete as expected in about 6 seconds so the issue seems to be only with the job create API. This seems to only be h...
- 2259 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Matthew Dalesio​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 4 kudos
- 3232 Views
- 3 replies
- 10 kudos
I have an Azure storage account, and I need to list the containers on that storage account using scala from databricks notebook
I tried to run azure-cli on databricks (i am using Credential passthrough and my account needs MFA ) and do the following code:az storage container list --account-name "account_name" --auth-mode login --query "[].name" --output tsvHowever, it outputs...
- 3232 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @elias bou ghosn​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 10 kudos
- 2572 Views
- 5 replies
- 10 kudos
Cost as per the Databricks demo
Hi there,I came across this Databricks demo from the below link. https://youtu.be/BqB7YQ1-KKcKindly Fastforward to time 16:30 or 16:45 of the video and watch few mins of the video related to cost. My understanding is the data is in the lake and datab...
- 2572 Views
- 5 replies
- 10 kudos
- 10 kudos
Hi @AJ DJ​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 10 kudos
- 1663 Views
- 3 replies
- 13 kudos
Any Upcoming Updates to Data Engineer Professional Exam or Advanced Data Engineering Course?
Just failed the Data Engineer Professional exam... the Advanced Data Engineering prep course is totally inadequate, and the exam went way beyond in scope with some topics and questions. Are there any changes coming to the curriculum or the exam?
- 1663 Views
- 3 replies
- 13 kudos
- 13 kudos
For me, The exam actually really really hard, you must know almost all knowledgeof Databricks Lake House Platform and very deeply. Anyway, mistake make perfect keep fighting. Hope to hear from your post next time with Passed message!
- 13 kudos
- 940 Views
- 1 replies
- 6 kudos
ML inference server (with REST api) how can I specify a GPU cluster.?
is there an API for that? as I couldn't find a way to do this through the UIClassic serving for now (didn't get access to the new "serverless" offering)
- 940 Views
- 1 replies
- 6 kudos
- 6 kudos
any clues?
- 6 kudos
- 3058 Views
- 2 replies
- 3 kudos
spark exception error while reading a parquet file
when I try to read parquet file from Azure datalake container from databricks, I am getting spark exception. Below is my queryimport pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt...
- 3058 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @shamly pt​ , more info are needed to solve the issue. However common problems are:The storage is not mountThat file doesn't exists in the mounted storageAlso, there is no need to use an f-string if there are no curly brackets with expressions in ...
- 3 kudos
- 2639 Views
- 8 replies
- 18 kudos
Code snippet error from course - Databricks Academy - Delta Lake Rapid Start with Python
Dear Team,While i was doing hands on practice from the course - Delta Lake Rapid Start with Pythonhttps://customer-academy.databricks.com/learn/course/97/delta-lake-rapid-start-with-pythoni have come across false as the output dbutils.fs.rm(health_t...
- 2639 Views
- 8 replies
- 18 kudos
- 18 kudos
Could you give more description about your issue (screenshot or something). Hope to help you find the issue?
- 18 kudos
- 12679 Views
- 3 replies
- 2 kudos
How to convert dataframe (df), to a excel file that I can share with my colleagues ?
I am working on microsoft azure databrick, I have a final dataframe of shape (3276*23) , I want to share it in form of excel file? How can I do it ( I am using ->df.to_excel('fileOutput.xlsx', sheet_name = 'Sheet1', index = False) , command is runn...
- 12679 Views
- 3 replies
- 2 kudos
- 2 kudos
You could try this way, convert Pyspark Dataframe to Pandas Dataframe then export to excel file.
- 2 kudos
- 10073 Views
- 3 replies
- 1 kudos
Resolved! How do you read an Excel spreadsheet with Databricks
My cluster has Scala 2.12I've installed Maven Library com.crealytics:spark-excel_2.12:0.14.0I get an error java.lang.IllegalStateException: Cannot get a STRING value from a NUMERIC cellwhen trying to execute the following%pythonexcelFileName="/mnt/dl...
- 10073 Views
- 3 replies
- 1 kudos
- 1 kudos
Another way also help for your case is usign Pandas to read excel then convert Pandas Dataframe to Pyspark Dataframe
- 1 kudos
- 4063 Views
- 9 replies
- 3 kudos
Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received
Hello, I passed Databricks Certified Data Engineer Associate on 28th October 2022, but I haven't received my certificate/badge yet. Please help me with this. Best Regards,Janaka Nawarathna.
- 4063 Views
- 9 replies
- 3 kudos
- 6992 Views
- 3 replies
- 7 kudos
Resolved! Best language to use
Databricks supports SQL, Scala, Python, and R. Is there a most performant language to use on Databricks? I know SQL well but would like to get into one of the other languages and don't know which to focus on.
- 6992 Views
- 3 replies
- 7 kudos
- 579 Views
- 0 replies
- 2 kudos
Databricks SQL endpoint authentication
Hi all​I have a requirement that goes like this:Users from a particular software that doesn't have out-of-the-box integration with Databricks click on a dashboard, the button click then sends an SQL query​ to Databricks (user gets authenticated in Da...
- 579 Views
- 0 replies
- 2 kudos