- 24935 Views
- 5 replies
- 5 kudos
What are the best practices for spark DataFrame caching?
Hi,When caching a DataFrame, I always use "df.cache().count()".However, in this reference, it is suggested to save the cached DataFrame into a new variable:When you cache a DataFrame create a new variable for it cachedDF = df.cache(). This will allow...
- 24935 Views
- 5 replies
- 5 kudos
- 5 kudos
In addition to other comments, I will just add that make sure you do the cache only when necessary. i.e. if you need to save a data frame for a time being to be referenced later in the code, then you should consider doing a cache. But if your code ha...
- 5 kudos
- 2635 Views
- 0 replies
- 2 kudos
Databricks Community rewards portal is down
When can we expect the Databricks Community rewards portal to be up and running. The page shows the warning message as website is under construction. Attached the screen shot of the message for your reference. Kindly resolve the issue and reload the ...
- 2635 Views
- 0 replies
- 2 kudos
- 6221 Views
- 4 replies
- 3 kudos
Resolved! What is the best approach to display DataFrame without re-executing the logic each time we display?
Hi,I have a DataFrame and different transformations are applied on the DataFrame. I want to display DataFrame after several transformations to check the results. However, according to the Reference, every time I try to display results, it runs the e...
- 6221 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks.In this reference, it is suggested to save the cached DataFrame into a new variable:When you cache a DataFrame create a new variable for it cachedDF = df.cache(). This will allow you to bypass the problems that we were solving in our example, ...
- 3 kudos
- 3881 Views
- 4 replies
- 3 kudos
How to read gcs paths with square barkets?
Hi! I'm trying to read a file using Scala from gcs that has square brackets in the file path.I keep getting the following error:URISyntaxException: Illegal character in path at index 209I tried putting an extra front slash in front of them but it sti...
- 3881 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Kaniz ! Thank you for your help.However, when I try using you're code I still get an error: "URISyntaxException: Illegal character in path at index "I'm trying to read a txt file. This is the file path: "gs://my-bucket/my Data/sparkTests/GM-1220,...
- 3 kudos
- 4749 Views
- 0 replies
- 0 kudos
List all delta tables in a database with total size, last snapshot size and user using python/sql
I am trying to list all delta tables in a database and retrieve the following columns: `totalsizeinbyte`, `sizeinbyte` (i.e. the size of last snap shot size) and `created_by` (`lastmodified_by` could also work). Checking online I came across the foll...
- 4749 Views
- 0 replies
- 0 kudos
- 4548 Views
- 6 replies
- 0 kudos
Resolved! Databricks Certified Data Engineer Associate (Version 3) Exam suspended
Hi Team, My data bricks exam got suspended on 19th July 2023 and it is still in the suspended state. I have raised a support request using the below link https://help.databricks.com/s/contact-us?ReqType=training and my request Id is #00347940 I wo...
- 4548 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi again, I have notified the team that handles these requests, and they will get back to you.I appreciate your patience.
- 0 kudos
- 2881 Views
- 0 replies
- 0 kudos
Group by using a filter/parameter in SQL dashboard?
Hey all,I'm trying to build a SQL dashboard where the consumer can select a date range, but then can also select whether they want to see the results grouped by day/week/month over that timeframe. I think this is possible for a notebook dashboard by ...
- 2881 Views
- 0 replies
- 0 kudos
- 2222 Views
- 1 replies
- 1 kudos
Uable to Connect to S3
ExecutionException: java.nio.file.AccessDeniedException: s3a://bucketName: Forbidden;Hadoop 3.3.4, aws-sdk-java/1.12.189 Linux/5.15.0-1038-aws OpenJDK_64-Bit_Server_VM/25.362-b09 java/1.8.0_362 scala/2.12.14 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg...
- 2222 Views
- 1 replies
- 1 kudos
- 1 kudos
https://kb.databricks.com/en_US/security/forbidden-access-to-s3-datahttps://kb.databricks.com/en_US/delta/s3-permissions-delta You can check this two. see if this helps.
- 1 kudos
- 894 Views
- 0 replies
- 0 kudos
Failed to submit the feedback survey on time for certification voucher.
I attend the data bricks training on Apr 2023. I failed to submit the feedback survey on time for certification voucher. Will it be possible to get the offer voucher now?
- 894 Views
- 0 replies
- 0 kudos
- 1569 Views
- 0 replies
- 0 kudos
Not able to catch structured streaming exception
I would like to catch and handle an exception in a structured streaming job.The databricks notebook still displays the exception, regardless of added exception handling (see attached screenshot)I guess that the exception is displayed by the cell outp...
- 1569 Views
- 0 replies
- 0 kudos
- 845 Views
- 0 replies
- 0 kudos
Hi all
great to be here
- 845 Views
- 0 replies
- 0 kudos
- 26437 Views
- 5 replies
- 8 kudos
How to run a different python version then the default one on databricks cluster
Hi All, I recently have a use case where i need to run something on the databricks cluster and the bare requirement is that python version shoulde be 3.8 and DBR version should be 11+. I am confused on how to handle this. Is there anyway in which we ...
- 26437 Views
- 5 replies
- 8 kudos
- 8 kudos
Hi @deficiant_codge Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 8 kudos
- 1476 Views
- 1 replies
- 0 kudos
Using Dolly for healthcare providers
Databricks Data+AI Summit has been very interesting. New(ish) user here with a background on healthcare. I see huge efficiency gains for healthcare providers in using specialized LLMs to get a synopsis of patient history. Would a provider need to tra...
- 1476 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey everyone! As a healthcare enthusiast, I completely agree with Dave-Griffith's point about the potential efficiency gains in using specialized LLMs for patient history. It's amazing how advanced technology like Dolly can revolutionize healthcare.T...
- 0 kudos
- 1582 Views
- 2 replies
- 5 kudos
Unable to login into partner account
@Anonymous , @Retired_mod , Could you please help me?Several days ago I was able to log into Databricks Partner portal but now I can't and even resetting password is not helping.Thanks in advance.
- 1582 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @asgalantsevepam Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
- 5 kudos
- 2583 Views
- 3 replies
- 4 kudos
Databricks Certification Exam got suspended
I had an issue when I was answering an exam for a certification. The exam was interrupted due to an issue on secure browser software, the screen became black and I could not finish the exame because I could not see the rest of the questions in the qu...
- 2583 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @ari_reyes Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 4 kudos
