- 3267 Views
- 1 replies
- 0 kudos
Resolved! Delta metadata caching
I understand the Delta caching for the data files. Do we have anything similar for the metadata files. Will the delta metadata get cached in the Delta caching
- 3267 Views
- 1 replies
- 0 kudos
- 0 kudos
The Delta logs - JSON files will be cached on the Driver (in memory) for Delta if they are small enough (<10 MB). They are not stored in the Delta cache. Before every query Delta checks if the snapshot is stale or has to be re-built.
- 0 kudos
- 2685 Views
- 1 replies
- 0 kudos
Resolved! If I create a Feature Store, how is the underlying data actually saved?
And do I have any control over where and how it's saved?
- 2685 Views
- 1 replies
- 0 kudos
- 0 kudos
The offline store is backed by Delta tables . In AWS we support Amazon Aurora (MySQL-compatible) & Amazon RDS MySQL and in Azure we support Azure Database for MySQL and Azure SQL Database as as online stores https://docs.microsoft.com/en-us/azure/d...
- 0 kudos
- 2850 Views
- 1 replies
- 0 kudos
Resolved! Delta Streaming and Optimize
I have a master delta table that is continuously getting written by a streaming job. I have optimize writes enabled and in addition, I run the OPTIMIZE command every 3 hours. However, I think the downstream streaming jobs which are streaming the data...
- 2850 Views
- 1 replies
- 0 kudos
- 0 kudos
This is working as expected. For Delta streaming, the data files created in the first place will be used for streaming. The optimized files are not considered the downstream streaming job. This is the reason it's not recommended to run VACUUM with f...
- 0 kudos
- 5014 Views
- 1 replies
- 1 kudos
Impact on Databricks objects after a user is deleted
What happens to resources (notebooks, jobs, clusters etc.) owned by a user when a user is deleted? The underlying problem we are trying to solve is that we want to automatically delete users through SCIM when the user leaves the company so that the u...
- 5014 Views
- 1 replies
- 1 kudos
- 1 kudos
When you remove a user from Databricks, a special backup folder is created in the workspace. This backup folder contains all of the deleted user’s content.W.r.t clusters/jobs, an admin can grant permission to other users.
- 1 kudos
- 3760 Views
- 1 replies
- 1 kudos
Resolved! How to run commands on the executor
Using %sh, I am able to run commands on the notebook and get output. How can i run a command on the executor and get the output. I want to avoid using the Spark API's
- 3760 Views
- 1 replies
- 1 kudos
- 1 kudos
It's not possible to use %sh to run commands on the executor. The below code can be used to run commands on the executor and get the outputvar res=sc.runOnEachExecutor[String]({ () => import sys.process._ var cmd_Result=Seq("bash", "-c", "h...
- 1 kudos
- 21270 Views
- 1 replies
- 1 kudos
Resolved! Views vs Materialized Delta Tables
Is there general guidance around using views vs creating Delta tables? For example, I need to do some filtering and make small tweaks to a few columns for use in another application. Is there a downside of using a view here?
- 21270 Views
- 1 replies
- 1 kudos
- 1 kudos
Views won't duplicate the data so if you are just filtering columns or rows or making small tweaks then views might be a good option. Unless, of course, the filtering is really expensive or you are doing a lot of calculations, then materialize the vi...
- 1 kudos
- 3169 Views
- 1 replies
- 1 kudos
- 3169 Views
- 1 replies
- 1 kudos
- 1 kudos
All three options are secure ways to store secrets. Databricks secrets has the additional functionality of redaction , so it is convenient sometimes. Also in azure, you have the ability to use azure KV as the backend for secrets.
- 1 kudos
- 5551 Views
- 1 replies
- 1 kudos
Resolved! Classpath issues when running spark-submit
How to identify the jars used to load a particular class. I am sure I packed the classes correctly in my application jar. However, looks like the class is loaded from a different jar. I want to understand the details so that I can ensure to use the r...
- 5551 Views
- 1 replies
- 1 kudos
- 1 kudos
Adding the below configurations at the cluster level can help to print more logs to identify the jars from which the class is loaded. spark.executor.extraJavaOptions=-verbose:class spark.driver.extraJavaOptions=-verbose:class
- 1 kudos
- 2477 Views
- 1 replies
- 0 kudos
Resolved! Cannot upload libraries on UI
When trying to upload libraries on UI it fails.
- 2477 Views
- 1 replies
- 0 kudos
- 0 kudos
One corner case scenario where we can hit this issue is if there is /<shard name>/0/Filestore/jars file in the root bucket of the workspace. Once you remove the file, the upload should work fine.
- 0 kudos
- 5605 Views
- 3 replies
- 0 kudos
Resolved! Frequent spot loss of driver nodes resulting in failed jobs when using spot fleet pools
When using spot fleet pools to schedule jobs, driver and worker nodes are provisioned from the spot pools and we are noticing jobs failing with the below exception when there is a driver spot loss. Share best practices around using fleet pools with 1...
- 5605 Views
- 3 replies
- 0 kudos
- 0 kudos
In this scenario, the driver node is reclaimed by AWS. Databricks started preview of hybrid pools feature which would allow you to provision driver node from a different pool. We recommend using on-demand pool for driver node to improve reliability i...
- 0 kudos
- 3897 Views
- 1 replies
- 1 kudos
Resolved! Databricks Vs Yarn - Resource Utilization
I have a spark-submit application that worked fine with 8GB executor memory in yarn. I am testing the same job against the Databricks cluster with the same executor memory. However, the jobs are running slower in Databricks.
- 3897 Views
- 1 replies
- 1 kudos
- 1 kudos
This is not an Apple to Apple comparison. When you set 8GB as the executor memory in Yarn, then the container that is launched to run the executor JVM is getting 8GB of memory. Accordingly, the Xmx value of the heap is calculated. In Databricks, when...
- 1 kudos
- 4341 Views
- 2 replies
- 0 kudos
- 4341 Views
- 2 replies
- 0 kudos
- 0 kudos
You can disable download button for notebook results which exports results as csv from admin console - > workspace settings -> advanced section
- 0 kudos
- 3888 Views
- 2 replies
- 0 kudos
- 3888 Views
- 2 replies
- 0 kudos
- 0 kudos
When a cluster is attached to a pool, cluster nodes are created using the the pool’s idle instances which help to reduce cluster start and auto-scaling times .If you are using pools and looking to reduce start time for all scenarios, then you should ...
- 0 kudos
- 2760 Views
- 2 replies
- 0 kudos
- 2760 Views
- 2 replies
- 0 kudos
- 0 kudos
If you are looking for incrementally loading data from Azure SQL, checkout one of our technology partners that support change-data-capture or setup debezium for sql-server. These solutions could land data in a streaming fashion to kafka/kinesis/even...
- 0 kudos
- 3243 Views
- 1 replies
- 0 kudos
Resolved! Can I use OSS Spark History Server to view the EventLogs
Is it possible to run the OSS SPark history server and view the spark event logs.
- 3243 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, it's possible. The OSS Spark history server can read the Spark event logs generated on a Databricks cluster. Using Cluster log delivery, the SPark logs can be written to any arbitrary location. Event logs can be copied from there to the storage ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
