Hi,I am quite new to working with Databricks in VS code. I am trying to figure out the best way to plot my data, when running on a cluster. I would like to have the possibility to zoom and move the plot as I have when plotting locally with Matplotlib...
Hi Team, I am getting error that voucher code is invalid error when trying to register for "Databricks Certified Associate Data Engineer Associate. I got this issue once page was reloaded due to slowness of the internet before checkout. and the vouch...
It seems that when I am connecting to Databricks Warehouse, it is using the default catalog which is hive_metastore. Is there a way to define unity catalog to be the default?I know I can run the queryUSE CATALOG MAINAnd then the current session will ...
Thanks Brian2. Is there an equivalent config parameter for a SQL Warehouse?
create table SalesReport(TerritoryName NVARCHAR(50), ProductName NVARCHAR(100), TotalSales DECIMAL(10,2), PreviousYearSales DECIMAL(10,2), GrowthRate DECIMAL(10,2)); create table ErrorLog( ErrorID int, ErrorMessage nvarchar(max),ErrorDate datetime);...
I would like to know if there is any difference if I save dataframe during tranformation to itself as first code or to new dataframe as second example.Thankslog_df = log_df.withColumn("process_timestamp",from_utc_timestamp(lit(current_timestamp()),"E...
Hi fellasi am working on databricks using icebergat first i have configured my notebook as belowspark.conf.set("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkCatalog")spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")s...
Hi,we are using databricks jdbc https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.33it seems like there is a thread leakage when getConnection failscould anyone advice?can be reproduced with @Test void databricksThreads() {...
Hi,none of the above suggestion will not work...we already contacted databricks jdbc team, thread leakage was confirmed and was fixed in version 2.6.34https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.34this leakage still exist if...
Hello - I have a foreign catalog which I can access fine in SQL. However, I can't access it from from python notebook.i.e. this works just fine if I have notebook using a Pro SQL Warehouse%sqlUSE CATALOG <my_foreign_catalog_name>;USE SCHEMA public;S...
Hi, Are you using this in single user cluster? Also, please tag @Debayan with your next response so that I will get notified.
Hi,I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below. Test data: I have configuration like below based on column 4: If column 4=0:...
Regarding --files option in spark submit task of Databricks jobs, would like to understand how it works and what is the syntax to pass multiple files to --files? I tried using --files and --py-files and my understanding is, it should make available t...
Hi, could you please check if this helps: https://docs.databricks.com/en/files/index.html Also please tag @Debayan​ with your next response which will notify me, Thank you!
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-hive to Databricks?Regards,Phanindra
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-SQOOP to Databricks?
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-ooze jobs to Databricks?
Hi, You can try checking the below resources on Hadoop migration: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-...
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-HDFS to Databricks?
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
I am using a notebook to copy over my database on a schedule (I had no success connecting through the Data Explorer UI). When I run the notebook on its own, it works. When I run it as a scheduled job, I get this error. org.apache.spark.SparkSQLExcept...
Hi, the error code is minimal, could you please post the whole error if that is possible? Also please tag @Debayan​ with your next response which will notify me, Thank you!
