- 621 Views
- 1 replies
- 0 kudos
I want to know how databricks maintain data recency in databricks
- 621 Views
- 1 replies
- 0 kudos
Latest Reply
When using delta tables in databricks, you have the advantage of delta cache which accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format. At the beginning of each query delta tables au...
- 797 Views
- 1 replies
- 0 kudos
Even though the NPIP is more secure as the network traffic travel through Microsoft backbone network why it is optional , it should be mandatory, is there some limitataion or a case where we may not able to use NPIP .
- 797 Views
- 1 replies
- 0 kudos
Latest Reply
NPIP / secure cluster connectivity requires a NAT gateway (or similar appliance) for outbound traffic from your workspace’s subnets to the Azure backbone and public network. This incurs a small additional cost. Also, it is worth mentioning that ne...
- 887 Views
- 2 replies
- 0 kudos
Suppose I am not interested in checkpoints, how can I disable Checkpoints write in delta
- 887 Views
- 2 replies
- 0 kudos
Latest Reply
Writing statistics in a checkpoint has a cost which is visible usually only for very large tables. However it is worth mentioning that, this statistics would be very useful for data skipping which speeds up subsequent operations. In Databricks Runti...
1 More Replies
- 846 Views
- 1 replies
- 0 kudos
Where do you specify what database the DLT tables land in?
- 846 Views
- 1 replies
- 0 kudos
Latest Reply
The target key, when creating the pipeline specifies the database that the tables get published to. Documented here - https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables
- 1170 Views
- 1 replies
- 0 kudos
Specifically, we have in mind:* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)* When we want to test an API change, build a Docker image with the relevant changes in a Jar file* Update the job configur...
- 1170 Views
- 1 replies
- 0 kudos
Latest Reply
>Where do we put custom Jar files when building the Docker image? /databricks/jars>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to tak...
- 1192 Views
- 1 replies
- 0 kudos
For Delta tables, among Z-order and Partioning which is recommended technique for efficient Data Skipping
- 1192 Views
- 1 replies
- 0 kudos
Latest Reply
Partition pruning is the most efficient way to ensure Data skipping. However, choosing the right column for partitioning is very important. It's common to see choosing the wrong column for partitioning can cause a large number of small file problems ...
- 767 Views
- 2 replies
- 0 kudos
if I might need a dashboard to see increase in number of rows on day to day basis, also a dashboard that shows size of Parquet/Delta files in my Lake?
- 767 Views
- 2 replies
- 0 kudos
Latest Reply
val db = "database_name"
spark.sessionState.catalog.listTables(db).map(table=>spark.sessionState.catalog.externalCatalog.getTable(table.database.get,table.table)).filter(x=>x.provider.toString().toLowerCase.contains("delta"))The above code snippet wi...
1 More Replies
- 893 Views
- 2 replies
- 0 kudos
Is there anyway to add a Spark Config that reverts the default behavior when doing tables writes from Delta to Parquet in DBR 8.0+? I know you can simply specify .format("parquet") but that could involve a decent amount of code change for some client...
- 893 Views
- 2 replies
- 0 kudos
- 588 Views
- 1 replies
- 0 kudos
If you have a user that is removed from the workspace, are the tokens they've created automatically invalidated?
- 588 Views
- 1 replies
- 0 kudos
Latest Reply
Yes, PAT tokens will be invalid if a user is removed since those tokens are attached to their current credentials and access.
