cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

PragyaS
by New Contributor
  • 745 Views
  • 1 replies
  • 0 kudos

Creating Zip file on Azure storage explorer

I need to create zip file which will contain csv files which I have created from the dataframe. But I am unable to create valid zip file which will be containing all csv files. Is it possible to create zip file from code in databricks on azure storag...

  • 745 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @PragyaS,  References: Creating Zip file on Azure storage explorer - Databricks CommunityHow to zip files (on Azure Blob Storage) with shutil in DatabricksWorking with Unity Catalog in Azure DatabricksAnnouncing the General Availability of Unity C...

  • 0 kudos
nileshtiwaari
by New Contributor
  • 356 Views
  • 1 replies
  • 0 kudos

Spark structured streaming

hi,could someone please help me with this code :-input parameter df is a spark structured streaming dataframe def apply_duplicacy_check(df, duplicate_check_columns):    if len(duplicate_check_columns) == 0:         return None, df    valid_df = df.dr...

  • 356 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @nileshtiwaari , The error message you’re encountering indicates that the .exceptAll() operation is not supported on streaming DataFrames. In structured streaming, certain operations have limitations due to the nature of streaming data. To addr...

  • 0 kudos
bamvallar
by New Contributor
  • 352 Views
  • 1 replies
  • 0 kudos
  • 352 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @bamvallar, Welcome to the Databricks Community! We're thrilled to have you here and thank you for attending the DAIS and visiting our community booth. Your participation means a lot to us, and we're excited to see the contributions you'll bring ...

  • 0 kudos
chand20025
by New Contributor
  • 312 Views
  • 1 replies
  • 0 kudos

Does databricks  support spearman rank correlation

  while working on spark scala. I tried working on code using pearson correaltion it works but when I  tried to work on spearman rank correlation it displayed error like Currently only the calculation of the Pearson Correlation coefficient is support...

Screenshot (21).png
  • 312 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @chand20025,  It seems that Databricks currently only supports the calculation of the Pearson Correlation coefficient and not the Spearman rank correlation. If you’re specifically looking to calculate Spearman rank correlation, you might need to e...

  • 0 kudos
269violetta
by New Contributor
  • 316 Views
  • 1 replies
  • 0 kudos

Hi all, I recently enabled SSO on my Databricks account. Now, when a user signs in they see "No work

Hi all, I recently enabled SSO on my Databricks account. Now, when a user signs in they see "No workspaces have been enabled for your account", which is the expected behavior as I haven't created any workspaces yet

  • 316 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @269violetta, Thank you for sharing your concern with us!   To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours)....

  • 0 kudos
269violetta
by New Contributor
  • 642 Views
  • 1 replies
  • 0 kudos

Databricks summit 2024 unity catalog migration!!

Databricks summit 2024 unity catalog migration!!

  • 642 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @269violetta, At the Data + AI Summit 2024, Databricks introduced exciting advancements for Unity Catalog, their unified governance solution for data and AI. Here are the highlights: Open Sourcing Unity Catalog: Databricks has open-sourced Un...

  • 0 kudos
scharly3
by New Contributor II
  • 3216 Views
  • 3 replies
  • 0 kudos

Error: Folder xxxx@xxx.com is protected

Hello, On Azure Databricks i'm trying to remove a folder on the Repos folder using the following command : databricks workspace delete "/Repos/xxx@xx.com"I got the following error message:databricks workspace delete "/Repos/xxxx@xx.com"Error: Folder ...

  • 3216 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

The repo/username folder is your home folder and it is a system folder. Therefore, the folder cannot be deleted or renamed

  • 0 kudos
2 More Replies
ganemouni
by New Contributor
  • 574 Views
  • 1 replies
  • 0 kudos

Facing issue with Databricks Context Limit Exceeded

We have a use case , where there may be chance of 200 jobs executing at once. But Few notebooks are failing with an issue "run failed with error message Too many execution contexts are open right now.(Limit set currently to 150)." Can anyone help how...

ganemouni_0-1718803036914.png
  • 574 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Hello, you can refer to the following KB article for information and best practices in regards the issue you are facing: https://kb.databricks.com/en_US/notebooks/too-many-execution-contexts-are-open-right-now  Best practices Use a job cluster instea...

  • 0 kudos
Jagadish_Vela
by New Contributor II
  • 907 Views
  • 3 replies
  • 0 kudos

Issue with importing data into Power BI from a view containing 100 million records in ADB

Hello, everyone. I'm trying to import data into Power BI from a view in Azure Databricks that contains over 10 million records. Every time it's failing with this following error"Failed to save modifications to the server. Error returned: 'OLE DB or O...

ADB Token Error.png
  • 907 Views
  • 3 replies
  • 0 kudos
Latest Reply
riquedan
New Contributor II
  • 0 kudos

Users have the option of either using OAuth or switching to Personal Access Token as a solution. I am sorry to say that this is the only method I am aware of at the present. hi 

  • 0 kudos
2 More Replies
karola61
by New Contributor II
  • 408 Views
  • 1 replies
  • 0 kudos

What is the vector database to generate in DataBricks?

What is the vector database to generate in DataBricks?

  • 408 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

it is used for RAG models (generative AI) and contains embeddings used by those models.https://docs.databricks.com/en/generative-ai/vector-search.html

  • 0 kudos
sucharita-das
by New Contributor II
  • 659 Views
  • 1 replies
  • 0 kudos

Certification exam

Hello Team, Today my exam was scheduled at 7:30 AM. No one contacted me before the exam for any validation/guideline. About one hour after, the proctor suspended my exam. Please check the video. I was constantly looking at the screen only. I have rai...

Community Platform Discussions
Databricks Professional
exam suspended
  • 659 Views
  • 1 replies
  • 0 kudos
Latest Reply
sucharita-das
New Contributor II
  • 0 kudos

Hi Databricks community and @Cert-Team,It is more than 24 hours and I did not get any update after my Databricks Professional Exam got suspended.Please help me to sort this out so that I can complete the exam. It is very important for me to complete ...

  • 0 kudos
subahan_dbricks
by New Contributor II
  • 995 Views
  • 4 replies
  • 1 kudos

Databricks DataEngineering Learning

I am new to Databricks. I am trying to do the lab work given Databricks DataEngineering course, at workbook 4.1 I am getting below error. Please help to resolve.Expected one and only one cluster definition.Edit the config via the JSON interface to re...

  • 995 Views
  • 4 replies
  • 1 kudos
Latest Reply
subahan_dbricks
New Contributor II
  • 1 kudos

I am unable to resolve this. Any help would be appreciated. Thanks

  • 1 kudos
3 More Replies
Uderson
by New Contributor
  • 944 Views
  • 1 replies
  • 0 kudos

Vector Database

What is the vector database to generate in DataBricks?

  • 944 Views
  • 1 replies
  • 0 kudos
Latest Reply
mhiltner
Contributor III
  • 0 kudos

Not sure if i understood the question. If you want to use Databricks Vector Database, just go to your table > create vector search index. First you need to create a Vector Search Endpoint (compute > vector search), and you need to have an enabled ser...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors