- 670 Views
- 1 replies
- 0 kudos
Suspension of Data Engineer Professional exam
Hi Databricks TeamI had scheduled my exam on 6th sep 2023, during exam same pop up came up, stating that I am looking in some other direction. I told them that my laptop mouse is not working properly, so I was looking at it. But still they suspended ...
- 670 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @priyakant1 ,Have you got any response from the team, like did they reschedule your exam?
- 0 kudos
- 1878 Views
- 3 replies
- 1 kudos
Resolved! My exam has suspended , Need help Urgently (21/08/2023)
Hello Team,I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file a complain...
- 1878 Views
- 3 replies
- 1 kudos
- 1 kudos
Sub: My exam Datbricks Data Engineer Associate got suspended_need immediate help please (10/09/2023)I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showin...
- 1 kudos
- 2280 Views
- 2 replies
- 1 kudos
Resolved! Records are missing while filtering the dataframe in multithreading
Hi, I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below. Test data: I have configuration like below based on column 4: If colum...
- 2280 Views
- 2 replies
- 1 kudos
- 1 kudos
Looks like you are comparing to strings like "1", not values like 1 in your filter condition. It's hard to say, there are some details missing like the rest of the code and the DF schema, and what output you are observing.
- 1 kudos
- 1162 Views
- 2 replies
- 0 kudos
Increase cores for Spark History Server
By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (Number of threads that will be used by history server to process event logs)How can we increase the number of cores for Spark History Server ?
- 1162 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @VMeghraj, To increase the number of cores for Spark History Server, you can modify the spark.history.fs.numReplayThreads Configuration parameter. You can set the desired number of cores by modifying the value of spark.history.fs.numReplayThreads...
- 0 kudos
- 1160 Views
- 1 replies
- 0 kudos
Databricks Rstudio Init Script Deprecated
OK so I'm trying to use Open Source Rstudio on Azure Databricks.I'm following the instructions here: https://learn.microsoft.com/en-us/azure/databricks/sparkr/rstudio#install-rstudio-server-open-source-editionI've installed the necessary init script ...
- 1160 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @meystingray, The error message you're encountering is indicating that the init script path is not absolute. According to the Databricks documentation, init scripts should be stored as workspace files. Here's how you can do it. 1. Store your ini...
- 0 kudos
- 5636 Views
- 1 replies
- 0 kudos
Is it good to process files in multithreading?
Hi,I need to process nearly 30 files from different locations and insert records to RDS.I am using multi-threading to process these files parallelly like below. def process_files(file_path): <process files here> 1. Find bad records based on fie...
- 5636 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Policepatil , - The approach of parallel processing files can increase the overall speed of the operation.- Multi-threading can optimize CPU usage but not necessarily make I/O operations faster.- I/O operations like reading and writing files are...
- 0 kudos
- 1314 Views
- 2 replies
- 0 kudos
Data Insertion
Scenario: Data from blob storage to SQL db once a week.I have 15(from current date to next 15 days) days data into the blob storage, stored date wise in parquet format, and after seven days the next 15 days data will be inserted. Means till 7th day t...
- 1314 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @bachan, Based on your scenario, you might consider using Azure Data Factory (ADF) for your data pipeline. Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. Here ...
- 0 kudos
- 2823 Views
- 2 replies
- 0 kudos
Server error: OK - Notebook
Hi I am currently having a weird notebook behavior. Every time I write, I am getting the following error. My gut feeling is that it causes by the Auto-save feature.Cheers,Gil
- 2823 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Gilg , Based on the given information, it seems that the error you are experiencing is related to notebook autosaving. The error message "Failed to save revision: Notebook size exceeds limit" indicates that the notebook size is too large to be a...
- 0 kudos
- 4911 Views
- 1 replies
- 0 kudos
Databricks Terraform Cluster Issue.
Error: default auth: cannot configure default credentials. Config: token=***. Env: DATABRICKS_TOKENon cluster.tf line 27, in data “databricks_spark_version” “latest_lts”:27: data “databricks_spark_version” “latest_lts” {
- 4911 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Simon_T , Based on the given error message and the provided information, it seems that the default authentication credentials are not properly configured for Databricks. To resolve this issue, you need to set up the authentication using a person...
- 0 kudos
- 675 Views
- 1 replies
- 0 kudos
Databricks data engineer associate Exam got suspended.
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...
- 675 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sivaji Sorry to hear you had a bad experience, and that you got a slow response here in the community. I see that you have taken and passed the exam, Congratulations!For the future, our support team handles cases from here first so it tends to be...
- 0 kudos
- 1305 Views
- 1 replies
- 0 kudos
Records are missing while creating new dataframe from one big dataframe using filter
Hi,I have data in file like belowI have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, ea...
- 1305 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Policepatil , The issue you're experiencing with missing records could be due to a variety of reasons. It could be related to how Spark handles data partitioning, or it might be due to some data quality issues in your input files. One possible ex...
- 0 kudos
- 962 Views
- 1 replies
- 0 kudos
Github "Danger Zone"
There is a "Danger zone" appearing in the Github indicating that the repositories I share on Databricks should be Suspended and possibly that Databricks should be uninstalled. This may be something standard in Github. Has anyone run across it?
- 962 Views
- 1 replies
- 0 kudos
- 0 kudos
It's not telling you that you should do these things. It's telling you that you may break stuff by doing these things. Yes the "Danger Zone" is a thing on Github, it tries to warn you before you do things like click to delete a repo.
- 0 kudos
- 282 Views
- 0 replies
- 0 kudos
Notebook Langchain ModuleNotFoundError Problem
Specific line of code:from langchain.retrievers.merger_retriever import MergerRetrieverAll other langchain import works when this is commented out. Same line works on my local VS Code.Appreciate any help with this issue. Thanks!
- 282 Views
- 0 replies
- 0 kudos
- 425 Views
- 0 replies
- 0 kudos
How to switch from free trial to Community Edition of Databricks in my Azure workspace?
hello,I am trying to switch into Databricks Community Edition after a 14 day trial. I was able to register, but when I try to start my new cluster, I get an error message, "Cluster start feature is currently disabled, and the cluster does not run".In...
- 425 Views
- 0 replies
- 0 kudos
- 2771 Views
- 3 replies
- 3 kudos
Resolved! Databricks community edition still available?
Is Databricks platform still available in its Community edition (outside Azure, AWS or GCP)? Can someone share the updated link?Thanks,Elisa
- 2771 Views
- 3 replies
- 3 kudos
- 3 kudos
Look : it is still available but afaik always linked to azure/gcp/aws
- 3 kudos
