On Video 4 of the Advanced Data Engineering with Databricks course at 3:08 the presenter says 'No one else can do what we can with a single solution' . This is far from truth, Palantir foundry is miles ahead of databricks in Data Governance , Ease of...
Hi Team,Could you please share the best practices for error handling in Databricks for the following: 1. Notebook level 2.Job level 3. Code level(Python) 4. streaming 5. DLT & Autoloader Kindly suggest details around Error handling...
Hi all,For training purposes, I have cloned a repo from John Snow Labs into my Databricks account and am working in the notebook that you can review at https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/03.0.SparkNLP_Pretr...
Try changing the magic command for that cell from %md to %md-sandbox to see if that helps the image to render appropriately.
Is there a way for me to get Type 2 SCD changes without using streaming tables?I'm worried streaming tables may have limitations that interfere with adoption.
I am trying to work through Tutorial: Query data from a Notebook.Access errors are defeating my attempts. Steps to reproduce:sign up for free trial through Databricks website. The path skipped the subscription-selection step and defaulted the trial t...
Done, thanks!
Hi,what mlflow version is used for the "Databricks Managed MLflow"Is there a workarround for MLflow CVE-2023-6015 (vulnerabilities) ?RegardsTimo
I am looking to run spark-sql-perf on a databricks cluster I am not sure how to create a session that add spark-sql-perf jar dependencies
I am currently using a personal computer cluster [13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)] on GCP attached to a notebook. After running a few command lines without an issue, I end up getting this errorInternal error. Attach your notebook t...
Hello,at some point I tested Databricks for a potential customer and, after the test, I cancelled the subscription.I read that it is not possible to resubscribe with the same e-mail address. Therefore, my idea would be to delete the account I created...
I have a similar issue. I subscribed to Databricks using AWS account email. I cancelled it later. Now I want to start using Databricks on AWS again using the same email id and with pay as you go plan. But there is no way to re-subscribe. If this can...
Hi I am just getting started in databricks would appreciate some help here.I have 10TB TPCDS in S3 i a hive partition structure.My goal is to benchmark a data bricks cluster on this data.after setting all IAM credentials according to this https://doc...
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
Hi Expert,How we can setup multiple notebook in a sequence order in flow for an example 1 pipeline have notebook1 - sequence 1,Notebook2- Sequence 2(in 1pipeline only)
Not sure how to approach your challenge but something you can is to use the Databricks Job Scheduler or if you want an external solution in Azure you can call several notebooks from DataFactory.
HelloAfter upgrading my cluster from DBR 12 to 14.1 I got a MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION on some of my Joinsdf1.join( df2, [df1["name"] == df2["name"], df1["age"] == df2["age"]], 'left_outer' )I resolved it by...
Hello community!I'm currently working on Spark scripts for data processing and facing some performance challenges. Any tips or suggestions on optimizing code for better efficiency? Your expertise is highly appreciated! paybyplatema Thanks.
I was trying to push .ipynb file to github from Azure DB to github and appears that original file is converted to source code as .py.Why does databricks do this and how can I control which ones to do or not ?I need to keep some files as .ipynb.Thanks...
