Hi Team,In Azure Databricks, we currently use Service Principal when creating Mount Points to Azure storage ( ADLS Gen1, ADLS Gen 2 and Azure Blob Storage).As part of S360 action to eliminate SPN secrets, we were asked to move to SPN+certificate / MS...
What’s a good way to map ddiiferent datasets all to a standard set of variables. For example in table 1 there is an ‘user_number’ field. And table 2 has the same field but it’s labeled ‘user_id’. They are both the same, and I want to plug both into a...
Lot of new announcements. Love the world we are getting into.
Lot of exciting announcements. Love the GPU’s being used for Photon
enjoying the conference and learning about open source unity
Thoughts on setting up labs for IT training and Hackathons (coding, not infiltrating) in underserved communities?
Hey ya'll!I've just started to dabble with Databricks recently and decided a fraud-detection pipeline would be a cool project to implement. Let me know what ya'll think about the article. Also would love more smaller scale project ideas I could work ...
hi team,Anyone can guide me for certification renewal process?
@Yogic24 It's in certification FAQ. https://www.databricks.com/learn/certification/faq#certificationsTo recertify, you will need to take the full current live exam.
Enjoying the conference and learning a lot as a new user to Databricks!
I have a pickle file "vectorizer.pkl" and I am currently facing an inconsistent behavior when trying to load that file. Sometimes it gets loaded successfully and sometimes I face an error. Here is how I am trying to load the file:from joblib import l...
I have a simple python script which have been running fine on my cluster but recently the same script gets stuck at map. So I tried creating a new cluster with less resources and tried to run the same script over that and it ran just fine.Here are th...
I agree with @raphaelblg. Most likely you're running out of memory. Multiprocessing or threadpools unfortunately do not benefit from extra workers as they only run on your driver node. This is very annoying and not a very known fact. Spark driver als...
How to remove duplicates in streaming query on the basis of some id?
@nileshtiwaari Are you refering to Strucutred Streaming or DLT?In case of Structured Streaming: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplicationAbout DLT, here's a thread from a couple of months...
Hello, While reviewing Unity Catalog to better understand its limitations, I came across the following statement:Overwrite mode for DataFrame write operations into Unity Catalog is supported only for Delta tables, not for other file formats. The user...
Looking forward to the amazing updates with Gen AI during tomorrow's keynote
Learned about Managed tables
Delta Sharing is pretty sweet!
