Hello,Is it possible to utilize S3 tags when writing a DataFrame with PySpark? Or is the only option to write the dataframe and then use boto3 to tag all the files?More information about S3 object tagging is here: Amazon S3 Object Tagging.Thank you.
Do I need to save the data locally and run the plotting locally as well or does anyone have a smart solution to this
As stated. My company changed name and the email address has migrated. I need to change it to the new one. And there is no way to open a support ticket to address that from what I saw (Standard Plan)Please do not tell me to contact AWS as they have ...
Hi,I am quite new to working with Databricks in VS code. I am trying to figure out the best way to plot my data, when running on a cluster. I would like to have the possibility to zoom and move the plot as I have when plotting locally with Matplotlib...
Hi all!I am using databricks-connect to develop and test pyspark code pure python (not notebook) files in my local IDE, running on a Databricks cluster. These files are part of a deployment setup with dbx so that they are run as tasks in a workflow.E...
Thanks for your response @Retired_mod . Can you elaborate on the difference between your suggestion and the code I provided? i.e. what would your if-else look like?
Hi Team, I am getting error that voucher code is invalid error when trying to register for "Databricks Certified Associate Data Engineer Associate. I got this issue once page was reloaded due to slowness of the internet before checkout. and the vouch...
Hello - I have a foreign catalog which I can access fine in SQL. However, I can't access it from from python notebook.i.e. this works just fine if I have notebook using a Pro SQL Warehouse%sqlUSE CATALOG <my_foreign_catalog_name>;USE SCHEMA public;S...
Hi, Are you using this in single user cluster? Also, please tag @Debayan with your next response so that I will get notified.
Regarding --files option in spark submit task of Databricks jobs, would like to understand how it works and what is the syntax to pass multiple files to --files? I tried using --files and --py-files and my understanding is, it should make available t...
Hi, could you please check if this helps: https://docs.databricks.com/en/files/index.html Also please tag @Debayan​ with your next response which will notify me, Thank you!
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-hive to Databricks?Regards,Phanindra
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-SQOOP to Databricks?
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-ooze jobs to Databricks?
Hi, You can try checking the below resources on Hadoop migration: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-HDFS to Databricks?
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
I am using a notebook to copy over my database on a schedule (I had no success connecting through the Data Explorer UI). When I run the notebook on its own, it works. When I run it as a scheduled job, I get this error. org.apache.spark.SparkSQLExcept...
Hi, the error code is minimal, could you please post the whole error if that is possible? Also please tag @Debayan​ with your next response which will notify me, Thank you!
Need help recovering account owner.Problem: Account owner cannot sign in with its password after SSO was configured. Account owner is a DL for team ownership so it doesn't have an AWS account and can't configure in AD group since it has "+" in the em...
Resolved by temporarily disabling SSO with Active directory that wasn't allowing an email to be created with "+"
We have our DLT pipeline and we need to import our custom libraries packaged in wheel files.We are on Azure DBX and we are using Az DevOps CI/CD to build and deploy the wheel packages on our DBX environment. In the top of our DLT notebook we are impo...
"context_based_upload_for_execute": truein projects.json allowed the code to run - but ended withRuntimeError: Cannot start a remote Spark session because there is a regular Spark session already running.
