Hello all,I've been experiencing the error described below, where I try to query a table from Snowflake which is about ~5.5B rows and ~30columns, and it fails almost systematically; specifically, either the Spark Job doesn't even start or I get the ...
Hey there @hamzatazib96​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
Hello all,We are working on one of the client requirements to implement suitable data encryption in Azure Databricks.We should be able to encrypt and decrypt the data based on the access, we explored fernet library but client denied it saying it degr...
Hi @purushotham Chanda​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...
I recently tried to create my account with Databricks Community Edition. I have singed up for it and received verification email. After that I have to reset my password. But while doing so I am always getting the following error. Can someone help me ...
I have a few suggestions for UI improvement on Databricks console -- Or maybe if anyone has figured out a way (using greasemonkey or similar scripts) to make some changes to Databricks UI -- i would like to know. # 1 - Workspace NavigationCan we have...
Great ideas.I know that regarding #1 new file manager is in development.#3 I also proposed when we discussed possible improvements.@Lindsay Olson​ @Jose Gonzalez​ @Prabakar Ammeappin​ maybe we can push it as user feedback as that are great ideas with...
We have a couple sources we'd already set up to stream to prod using a 3p system. Is there a way to sync this directly to our dev workspace to build pipelines? eg. directly connecting to a cluster in prod and pull with a job cluster, dump to S3 and u...
Hi @Erik Louie​ , We haven't heard from you on the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to oth...
So, I have a super simple left join from one table to another it's purpose to retrieve the date of birth for a customer from the customer ID FK in the transaction table to the customer ID PK in the customer table. A customer will have several transac...
Hi @Faye Hughes​ Thank you so much for getting back to us. It's really great of you to send in the solution and mark the answer as best. We really appreciate your time.Wish you a great Databricks journey ahead!
Hi all,I have a Delta Table as a Spark Streaming source.This table contains signals on row level -> each signal is one append to the source table that creates a new version in the delta transaction history.I am not really sure now how Spark streaming...
Hey there @Michael Galli​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
Hi @James Owen​, We haven’t heard from you on the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to...
I have a delta table already created, now I want to enable the change data feed. I read that I have to set delta.enableChangeDataFeed property to true. But however, this cannot be done using the Scala API. I tried using this but it didn't work. I am ...
Hi @Yatharth Maheshwari​, We haven’t heard from you on the last response from @Jose Gonzalez​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpf...
I have 1 role with AWS root account and 1 role wit AWS non-root account. How do I tell Dbricks to use specific role for cluster creation ? Please guide me here or if any documentation will also suffice . Thanks.
Hi @Anandkumar Ravikumar​ , We haven't heard from you on the last response from @Aman Sehgal​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to...
Hi @Thao Ton​, We haven't heard from you on the last response from @Hubert Dudek​ and @Aman Sehgal​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community as it can be he...
I am using the schema evolution in the delta table and the code is written in databricks notebook. df.write
.format("delta")
.mode("append")
.option("mergeSchema", "true")
.partitionBy("date")
.save(path)But I ...
Hi, I am currently trying to learn Databricks and going through tutorials and learning materials. I came across this link https://databricks.com/discover/pages/getting-started-with-delta-live-tablesWhile I get most of what is described in page, I fin...
I am try to get the percentile values on different splits but I got that the result of Databricks PERCENTILE_DISC() function is not accurate . I have run the same query on MS SQL but getting different result set.Here are both result sets for Pyspark ...
