- 2200 Views
- 1 replies
- 1 kudos
Resolved! threads leakage when getConnection fails
Hi,we are using databricks jdbc https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.33it seems like there is a thread leakage when getConnection failscould anyone advice?can be reproduced with @Test void databricksThreads() {...
- 2200 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,none of the above suggestion will not work...we already contacted databricks jdbc team, thread leakage was confirmed and was fixed in version 2.6.34https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.34this leakage still exist if...
- 1 kudos
- 16138 Views
- 2 replies
- 0 kudos
Resolved! Access Foreign Catalog using Python in Notebook
Hello - I have a foreign catalog which I can access fine in SQL. However, I can't access it from from python notebook.i.e. this works just fine if I have notebook using a Pro SQL Warehouse%sqlUSE CATALOG <my_foreign_catalog_name>;USE SCHEMA public;S...
- 16138 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Are you using this in single user cluster? Also, please tag @Debayan with your next response so that I will get notified.
- 0 kudos
- 1277 Views
- 0 replies
- 0 kudos
Missing records while using limit in multithreading
Hi,I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below. Test data: I have configuration like below based on column 4: If column 4=0:...
- 1277 Views
- 0 replies
- 0 kudos
- 2988 Views
- 1 replies
- 0 kudos
--files in spark submit task
Regarding --files option in spark submit task of Databricks jobs, would like to understand how it works and what is the syntax to pass multiple files to --files? I tried using --files and --py-files and my understanding is, it should make available t...
- 2988 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, could you please check if this helps: https://docs.databricks.com/en/files/index.html Also please tag @Debayan​ with your next response which will notify me, Thank you!
- 0 kudos
- 1574 Views
- 1 replies
- 0 kudos
Hive Migration best practices
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-hive to Databricks?Regards,Phanindra
- 1574 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
- 0 kudos
- 1944 Views
- 1 replies
- 0 kudos
Sqoop Migration best practices
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-SQOOP to Databricks?
- 1944 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
- 0 kudos
- 1278 Views
- 1 replies
- 0 kudos
OOZE Jobs migration to databricks
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-ooze jobs to Databricks?
- 1278 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources on Hadoop migration: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-...
- 0 kudos
- 2605 Views
- 1 replies
- 0 kudos
HDFS to Databricks
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-HDFS to Databricks?
- 2605 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
- 0 kudos
- 1661 Views
- 1 replies
- 0 kudos
Notebook runs with error when run as a job
I am using a notebook to copy over my database on a schedule (I had no success connecting through the Data Explorer UI). When I run the notebook on its own, it works. When I run it as a scheduled job, I get this error. org.apache.spark.SparkSQLExcept...
- 1661 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, the error code is minimal, could you please post the whole error if that is possible? Also please tag @Debayan​ with your next response which will notify me, Thank you!
- 0 kudos
- 1346 Views
- 1 replies
- 0 kudos
Suspension of Data Engineer Professional exam
Hi Databricks TeamI had scheduled my exam on 6th sep 2023, during exam same pop up came up, stating that I am looking in some other direction. I told them that my laptop mouse is not working properly, so I was looking at it. But still they suspended ...
- 1346 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @priyakant1 ,Have you got any response from the team, like did they reschedule your exam?
- 0 kudos
- 3027 Views
- 1 replies
- 0 kudos
My exam has suspended , Need help Urgently (21/08/2023)
Hello Team,I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file a complain...
- 3027 Views
- 1 replies
- 0 kudos
- 0 kudos
Sub: My exam Datbricks Data Engineer Associate got suspended_need immediate help please (10/09/2023)I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showin...
- 0 kudos
- 3490 Views
- 1 replies
- 1 kudos
Resolved! Records are missing while filtering the dataframe in multithreading
Hi, I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below. Test data: I have configuration like below based on column 4: If colum...
- 3490 Views
- 1 replies
- 1 kudos
- 1 kudos
Looks like you are comparing to strings like "1", not values like 1 in your filter condition. It's hard to say, there are some details missing like the rest of the code and the DF schema, and what output you are observing.
- 1 kudos
- 10026 Views
- 1 replies
- 0 kudos
Resolved! Recover Account Owner
Need help recovering account owner.Problem: Account owner cannot sign in with its password after SSO was configured. Account owner is a DL for team ownership so it doesn't have an AWS account and can't configure in AD group since it has "+" in the em...
- 10026 Views
- 1 replies
- 0 kudos
- 0 kudos
Resolved by temporarily disabling SSO with Active directory that wasn't allowing an email to be created with "+"
- 0 kudos
- 4255 Views
- 1 replies
- 0 kudos
Increase cores for Spark History Server
By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (Number of threads that will be used by history server to process event logs)How can we increase the number of cores for Spark History Server ?
- 4255 Views
- 1 replies
- 0 kudos
- 3280 Views
- 1 replies
- 0 kudos
Will the cells in the notebook keep running even if the browser is closed?
The Execute ML Model Pipeline celll has finished running, it took 2.27 days to finish running. However, the code in the following cell called Process JSON Output needs to take a very long time again to run. Can I simply close the browser and shut dow...
- 3280 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I have just tested it internally, even if the browser is closed the notebook keeps on running. You can start with a quick job to test it. Also, please tag @Debayan with your next response so that I will get notified.
- 0 kudos
