- 1051 Views
- 1 replies
- 0 kudos
I am trying to migrate my workload to another workspace ( from ST to E2), I am planning to use data bricks sync, but still I am not sure, will it migrate everything like , currents, user , groups, job, notebook etc or has some limitations which I s...
- 1051 Views
- 1 replies
- 0 kudos
Latest Reply
Here is the support matrix for import/export operations for databricks-syncAlso checkout https://github.com/databrickslabs/migrate
- 858 Views
- 1 replies
- 0 kudos
I want to know how databricks maintain data recency in databricks
- 858 Views
- 1 replies
- 0 kudos
Latest Reply
When using delta tables in databricks, you have the advantage of delta cache which accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format. At the beginning of each query delta tables au...
- 1038 Views
- 1 replies
- 0 kudos
Even though the NPIP is more secure as the network traffic travel through Microsoft backbone network why it is optional , it should be mandatory, is there some limitataion or a case where we may not able to use NPIP .
- 1038 Views
- 1 replies
- 0 kudos
Latest Reply
NPIP / secure cluster connectivity requires a NAT gateway (or similar appliance) for outbound traffic from your workspace’s subnets to the Azure backbone and public network. This incurs a small additional cost. Also, it is worth mentioning that ne...
- 1144 Views
- 2 replies
- 0 kudos
Suppose I am not interested in checkpoints, how can I disable Checkpoints write in delta
- 1144 Views
- 2 replies
- 0 kudos
Latest Reply
Writing statistics in a checkpoint has a cost which is visible usually only for very large tables. However it is worth mentioning that, this statistics would be very useful for data skipping which speeds up subsequent operations. In Databricks Runti...
1 More Replies
- 1079 Views
- 1 replies
- 0 kudos
Where do you specify what database the DLT tables land in?
- 1079 Views
- 1 replies
- 0 kudos
Latest Reply
The target key, when creating the pipeline specifies the database that the tables get published to. Documented here - https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables
- 1531 Views
- 1 replies
- 0 kudos
Specifically, we have in mind:* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)* When we want to test an API change, build a Docker image with the relevant changes in a Jar file* Update the job configur...
- 1531 Views
- 1 replies
- 0 kudos
Latest Reply
>Where do we put custom Jar files when building the Docker image? /databricks/jars>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to tak...
- 1575 Views
- 1 replies
- 0 kudos
For Delta tables, among Z-order and Partioning which is recommended technique for efficient Data Skipping
- 1575 Views
- 1 replies
- 0 kudos
Latest Reply
Partition pruning is the most efficient way to ensure Data skipping. However, choosing the right column for partitioning is very important. It's common to see choosing the wrong column for partitioning can cause a large number of small file problems ...
- 1114 Views
- 2 replies
- 0 kudos
if I might need a dashboard to see increase in number of rows on day to day basis, also a dashboard that shows size of Parquet/Delta files in my Lake?
- 1114 Views
- 2 replies
- 0 kudos
Latest Reply
val db = "database_name"
spark.sessionState.catalog.listTables(db).map(table=>spark.sessionState.catalog.externalCatalog.getTable(table.database.get,table.table)).filter(x=>x.provider.toString().toLowerCase.contains("delta"))The above code snippet wi...
1 More Replies
- 1144 Views
- 2 replies
- 0 kudos
Is there anyway to add a Spark Config that reverts the default behavior when doing tables writes from Delta to Parquet in DBR 8.0+? I know you can simply specify .format("parquet") but that could involve a decent amount of code change for some client...
- 1144 Views
- 2 replies
- 0 kudos
- 800 Views
- 1 replies
- 0 kudos
If you have a user that is removed from the workspace, are the tokens they've created automatically invalidated?
- 800 Views
- 1 replies
- 0 kudos
Latest Reply
Yes, PAT tokens will be invalid if a user is removed since those tokens are attached to their current credentials and access.