- 4637 Views
- 2 replies
- 0 kudos
VACUUM during read/write
Is it safe to run VACUUM on a Delta Lake table while data is being added to it at the same time? Will it impact the job result/performance?
- 4637 Views
- 2 replies
- 0 kudos
- 0 kudos
In the vast majority of cases, yes, it is safe to run VACUUM while data is concurrently being appended or updated to the same table. This is because VACUUM deletes data files no longer referenced by a Delta table's transaction log and does not effect...
- 0 kudos
- 4573 Views
- 2 replies
- 0 kudos
How does running VACUUM on Delta Lake tables effect read/write performance?
If I don't run VACUUM on a Delta Lake table, will that make my read performance slower?
- 4573 Views
- 2 replies
- 0 kudos
- 0 kudos
VACUUM has no effect on read/write performance to that table. Never running VACUUM on a table will not make read/write performance to a Delta Lake table any slower.If you run VACUUM very infrequently, your VACUUM runtimes themselves may be pretty hig...
- 0 kudos
- 2353 Views
- 1 replies
- 1 kudos
Can I have a Databricks Cluster that is only 1 node?
Yes you can create a "Single Node" Cluster, https://docs.databricks.com/clusters/single-node.html . It is currently not recommended to use "Single Node" cluster for streaming workloads
- 2353 Views
- 1 replies
- 1 kudos
- 1 kudos
Single Node clusters should not be used for production workloads involving streaming queries, or complex computations. The intention here is to bring up the Spark cluster for all kinds of workloads
- 1 kudos
- 1354 Views
- 1 replies
- 0 kudos
- 1354 Views
- 1 replies
- 0 kudos
- 0 kudos
Not yet but it is on the roadmap. Currently, only available in Databricks SQL
- 0 kudos
- 1281 Views
- 0 replies
- 0 kudos
- 1281 Views
- 0 replies
- 0 kudos
- 2105 Views
- 1 replies
- 1 kudos
- 2105 Views
- 1 replies
- 1 kudos
- 1 kudos
https://docs.databricks.com/dev-tools/api/latest/scim/scim-users.html#create-user
- 1 kudos
- 3344 Views
- 1 replies
- 0 kudos
Resolved! What is the difference between spark.sessionState.catalog.listTables vs spark.catalog.listTables
I see a significant performance difference when calling spark.sessionState.catalog.list compared to spark.catalog.list. Is that expected?
- 3344 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sessionState.catalog.listTables is a more lazy implementation.. it does not pull the column details when listing the tables. Hence it's faster. Whereas catalog.listTables will pull the column details as well. If the database has many Delta tabl...
- 0 kudos
- 7175 Views
- 1 replies
- 0 kudos
Resolved! How to list all Delta tables in a Database?
I wanted to get a list of all the Delta tables in a Database. What is the easiest way of getting it.
- 7175 Views
- 1 replies
- 0 kudos
- 0 kudos
Below code, the snippet can be used to list down the tables in a databaseval db = "database_name" spark.sessionState.catalog.listTables(db).map(table=>spark.sessionState.catalog.externalCatalog.getTable(table.database.get,table.table)).filter(x=>x....
- 0 kudos
- 43478 Views
- 3 replies
- 1 kudos
- 43478 Views
- 3 replies
- 1 kudos
- 1 kudos
This is by design and working as expected. Spark writes the data distributedly. use of coalesce (1) can help to generate one file, however this solution is not scalable for large data set as it involves bringing the data to one single task.
- 1 kudos
- 1610 Views
- 1 replies
- 1 kudos
- 1610 Views
- 1 replies
- 1 kudos
- 1 kudos
Photon is supported for batch workloads today and is the standard on Databricks SQL clusters and available as an option for Automated and Interactive clusters. And photon is in public preview today so available as an option for everyone. See this lin...
- 1 kudos
- 1729 Views
- 2 replies
- 0 kudos
- 1729 Views
- 2 replies
- 0 kudos
- 0 kudos
Delta has significant value beyond the DML/ACID capabilities. Delta's data organization strategies that @Ryan Chynoweth​ mentions also offer an advantage even for read-only use cases for querying and joining the data. Delta also supports in-place con...
- 0 kudos
- 4031 Views
- 1 replies
- 0 kudos
- 4031 Views
- 1 replies
- 0 kudos
- 0 kudos
This spark-salesforce connector looks like an option to query this data via SOQL/SAQL and brought into Databricks/Spark
- 0 kudos
- 1391 Views
- 1 replies
- 0 kudos
- 1391 Views
- 1 replies
- 0 kudos
- 0 kudos
There's actually several options here!AWSIf you'd like a very quick setup but full featured environment for your org, use the AWS quickstart: https://aws.amazon.com/quickstart/architecture/databricks/If you're solo exploring, you can use Databricks c...
- 0 kudos
- 1710 Views
- 1 replies
- 0 kudos
Setting cluster settings through SCIM
is there a way to set the following cluster settings through the SCIM? I am not seeing anything in the API docs that would suggest it is possible but I want to double check here.Enable credential passthroughSingle User AccessPermission settings
- 1710 Views
- 1 replies
- 0 kudos
- 0 kudos
Credential passthroughThis actually needs some setting up in AWS IAM to get started. Once you've created the right instance profiles, you'll need to add them to your Databricks workspace. There's pretty exhaustive guides here that has each of the ste...
- 0 kudos
- 1673 Views
- 1 replies
- 0 kudos
- 1673 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks jobs api can invoke code from cloud storage. But please note that it will not be stored as a Databricks notebook, rather it would be a source file or jar.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
