- 4620 Views
- 1 replies
- 0 kudos
Resolved! what are the join hints, available in spark 3.0, and how does it help compare to pervious spark version
what are the join hints, available in spark 3.0, and how does it help compare to pervious spark version
- 4620 Views
- 1 replies
- 0 kudos
- 0 kudos
4 types of join hints in Spark 3.0BROADCASTMERGESHUFFLE_HASHSHUFFLE_REPLICATE_NLMay be good idea to enable Adaptive Query Execution which speeds up Spark SQL join during run timeIn Spark 3.0, Adaptive Query Execution comes with below featuresDynamica...
- 0 kudos
- 3425 Views
- 1 replies
- 0 kudos
How is the photon engine different to catalyst optimizer
How is the photon engine different to catalyst optimizer
- 3425 Views
- 1 replies
- 0 kudos
- 0 kudos
I got this question from some customers and I want ti clarify here tooI think we are conflating two things:Catalyst optimizer is about coming up "Steps to take to execute the query". For example, the optimizer will decide how and when to do the join...
- 0 kudos
- 1041 Views
- 0 replies
- 0 kudos
databricks.com
Can I create a Delta Lake table on Databricks and query it with open-source Spark?Yes, in order to do this, you would install Open Source Spark and Delta Lake, both are open source. Delta Engine, which is only available on Databricks, will make delta...
- 1041 Views
- 0 replies
- 0 kudos
- 970 Views
- 0 replies
- 0 kudos
Data scientist Job Profile will be relevant in the future? By seeing current features in Databricks like AUTO ML, I am assuming that should the Data ...
Data scientist Job Profile will be relevant in the future?By seeing current features in Databricks like AUTO ML, I am assuming that should the Data scientist job will be mostly automated and sooner the data scientist in the company will start decl...
- 970 Views
- 0 replies
- 0 kudos
- 1343 Views
- 0 replies
- 0 kudos
Spark 3.0 Pandas UDF Â Old vs New Pandas UDF interfaceThis slide shows the difference between the old and the new interface. The same here. The new int...
Spark 3.0 Pandas UDF Old vs New Pandas UDF interfaceThis slide shows the difference between the old and the new interface. The same here. The new interface can also be used for the existing Grouped Aggregate Pandas UDFs. In addition, the old Pandas U...
- 1343 Views
- 0 replies
- 0 kudos
- 1511 Views
- 0 replies
- 0 kudos
Cluster Sizees on DB sql Cluster size Driver size Worker count 2X-Small i3.2xlarge 1 X-Small i3.2xlarge 2 Small i3.4xlarge 4 Med...
Cluster Sizees on DB sql Cluster size Driver size Worker count 2X-Small i3.2xlarge 1 X-Small i3.2xlarge 2 Small i3.4xlarge 4 Medium i3.8xlarge 8 Large i3.8xlarge 16 X-Large i3.16xlarge 32 2X-Large i3.16xlarge...
- 1511 Views
- 0 replies
- 0 kudos
- 2974 Views
- 0 replies
- 0 kudos
Muti Cluster Load balancing Multi-cluster Load Balancing: the minimum and maximum number of clusters over which queries sent to the endpoint are distr...
Muti Cluster Load balancingMulti-cluster Load Balancing: the minimum and maximum number of clusters over which queries sent to the endpoint are distributed. The default is Off with a maximum of 1 cluster. When set to On, the default is minimum 1 clus...
- 2974 Views
- 0 replies
- 0 kudos
- 1321 Views
- 0 replies
- 0 kudos
When would I want to change the Isolation Level of my Delta table?
I read this article in the docs about isolation levels, but I am not sure if I should be specifying this for my Delta tables. What situation would I want to change the isolation level from the default?
- 1321 Views
- 0 replies
- 0 kudos
- 3453 Views
- 1 replies
- 0 kudos
Can you restrict the type of clusters users are allowed to create?
I would like to make it so users can only create job clusters and not interactive clusters. Is it possible to do this in a workspace?
- 3453 Views
- 1 replies
- 0 kudos
- 0 kudos
This can be accomplished with cluster policies. You can use a policy similar to this example to restrict certain users or groups to only have permission to create job clusters.
- 0 kudos
- 37939 Views
- 1 replies
- 0 kudos
Resolved! What's the difference between mode("append") and mode("overwrite") on my Delta table
I would like to know the difference between .mode("append") and .mode("overwrite") when writing my Delta table
- 37939 Views
- 1 replies
- 0 kudos
- 0 kudos
Mode "append" atomically adds new data to an existing Delta table and "overwrite" atomically replaces all of the data in a table.
- 0 kudos
- 4414 Views
- 1 replies
- 0 kudos
Resolved! Where does the schema for a Delta table set reside?
I would like to know where can I find the current schema information from my Delta table.
- 4414 Views
- 1 replies
- 0 kudos
- 0 kudos
The table name, path, database info are stored in Hive metastore, the actual schema is stored in the "_delta_log" directory that should be in the root path location where you Delta table is stored.
- 0 kudos
- 9052 Views
- 1 replies
- 0 kudos
Resolved! How can I read a specific Delta table part file?
is there a way to read a specific part off a delta table? When I try to read the parquet file as parquet I get an error in the notebook that I’m using the incorrect format as it’s part of a delta table. I just want to read a single Parquet file, not ...
- 9052 Views
- 1 replies
- 0 kudos
- 0 kudos
Disable Delta format to read as Parquet you need to set to false the following Spark settings:>> SET spark.databricks.delta.formatCheck.enabled=false OR>> spark.conf.set("spark.databricks.delta.formatCheck.enabled", "false")its not recommended to re...
- 0 kudos
- 4507 Views
- 1 replies
- 0 kudos
Resolved! should I run ANALYZE TABLE on Delta tables?
I would like to know if it recommended to run Analyze table on Delta tables or not. If not, why?
- 4507 Views
- 1 replies
- 0 kudos
- 0 kudos
You can run ANALYZE TABLE on Delta tables only on Databricks Runtime 8.3 and above. For more details please refer to the docs: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-analyze-table.html
- 0 kudos
- 3486 Views
- 1 replies
- 1 kudos
Download private repo from GitHub Enterprise in Databricks notebook
We are trying to download our repository which is hosted on GitHub Enterprise to use its python libraries in our notebooks.Earlier we had issues with downloading our repository using the repos feature in Databricks platform since only notebooks can b...
- 3486 Views
- 1 replies
- 1 kudos
- 1 kudos
To fix the issue, we need to pass the token in the header itself git clone https://<token>:x-oauth-basic@github.com/owner/repo.gitExample:%sh git clone https://<token>@github.com/darshanbargal4747/databricks.git
- 1 kudos
- 2029 Views
- 1 replies
- 0 kudos
Unable to use on prem Mysql server as we are not able to resolve the hostname
while connecting from notebook, it returns the error unable to resolve name.
- 2029 Views
- 1 replies
- 0 kudos
- 0 kudos
Since we are unable to resolve hostname, it point towards the DNS issue. We can use custom dns using init script and add in the cluster:%scala dbutils.fs.put("/databricks/<directory>/dns-masq.sh";,""" #!/bin/bash #####################################...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
