- 2308 Views
- 11 replies
- 1 kudos
Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...
- 2308 Views
- 11 replies
- 1 kudos
Latest Reply
Hi @Rahul K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...
10 More Replies
- 909 Views
- 1 replies
- 0 kudos
I am using the databricks jdbc driver to access a delta lake. The database URL specifies transportMode=http. I have experimented with setting different values of fetchSize on the java.sqlPreparedStatement object and have monitored memory use within m...
- 909 Views
- 1 replies
- 0 kudos
Latest Reply
I think there is one spark configuration but I forgot right now Pelase try to utilized this doc maybe you get something- https://spark.apache.org/docs/latest/configuration.html
- 20688 Views
- 6 replies
- 8 kudos
let's suppose there is a database db, inside that so many tables are there and , i want to get the size of tables . how to get in either sql, python, pyspark.even if i have to get one by one it's fine.
- 20688 Views
- 6 replies
- 8 kudos
Latest Reply
@Raman Gupta​ - could you please try the below %python
spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()
5 More Replies
- 4703 Views
- 4 replies
- 0 kudos
After the new release of numpy 1.22.0 on Dec 31st, Databricks failed with this error for my existing Databricks Notebook Version 10.1 and numpy 1.20.0Qn: Why did the earlier releases after 1.20.0 uptil 1.22.0 did not raise the same exception. ?
- 4703 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Janani Mohan​ Hope you are doing well.Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
3 More Replies
- 5556 Views
- 4 replies
- 2 kudos
Hi,Is there a way to increase the cells' font size and set it as a default setting ?Thanks.Best Damien
- 5556 Views
- 4 replies
- 2 kudos
Latest Reply
Hi @Damien Sicard​ , As @werners has stated, you can zoom your browser.
3 More Replies
- 5794 Views
- 1 replies
- 0 kudos
Spark by default uses 200 partitions when doing transformations. The 200 partitions might be too large if a user is working with small data, hence it can slow down the query. Conversely, the 200 partitions might be too small if the data is big. So ho...
- 5794 Views
- 1 replies
- 0 kudos
Latest Reply
You could tweak the default value 200 by changing spark.sql.shuffle.partitions configuration to match your data volume. Here is a sample python code for calculating the valueHowever if you have multiple workloads with different data volumes, instead ...
