- 1461 Views
- 0 replies
- 0 kudos
- 1461 Views
- 0 replies
- 0 kudos
Forum Posts
- 2052 Views
- 1 replies
- 0 kudos
What is databricks Sync
I am trying to migrate my workload to another workspace ( from ST to E2), I am planning to use data bricks sync, but still I am not sure, will it migrate everything like , currents, user , groups, job, notebook etc or has some limitations which I s...
- 2052 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Here is the support matrix for import/export operations for databricks-syncAlso checkout https://github.com/databrickslabs/migrate
- 0 kudos
- 1880 Views
- 1 replies
- 0 kudos
How do we manage data recency in Databricks
I want to know how databricks maintain data recency in databricks
- 1880 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
When using delta tables in databricks, you have the advantage of delta cache which accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format. At the beginning of each query delta tables au...
- 0 kudos
- 1682 Views
- 1 replies
- 0 kudos
- 1682 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Start an endpointRun a queryGo to Query HistoryClick Details, Go to the Environment tabSearch sparkVersion.
- 0 kudos
- 2054 Views
- 1 replies
- 0 kudos
Why NPIP is an optional and not mandatory
Even though the NPIP is more secure as the network traffic travel through Microsoft backbone network why it is optional , it should be mandatory, is there some limitataion or a case where we may not able to use NPIP .
- 2054 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
NPIP / secure cluster connectivity requires a NAT gateway (or similar appliance) for outbound traffic from your workspace’s subnets to the Azure backbone and public network. This incurs a small additional cost. Also, it is worth mentioning that ne...
- 0 kudos
- 1582 Views
- 1 replies
- 0 kudos
- 1582 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Each local disk is 375 GB.So, for example, for n2-standard-4, it is 2 local disks. (0.75TB /2)https://databricks.com/wp-content/uploads/2021/05/GCP-Pricing-Estimator-v2.pdf?_ga=2.241263109.66068867.1623086616-828667513.1602536526
- 0 kudos
- 1258 Views
- 1 replies
- 0 kudos
- 1258 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
They are Zonal SSD Persistent Diskhttps://cloud.google.com/compute/docs/disks#introduction
- 0 kudos
- 2495 Views
- 2 replies
- 0 kudos
Don't want checkpoint in delta
Suppose I am not interested in checkpoints, how can I disable Checkpoints write in delta
- 2495 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Writing statistics in a checkpoint has a cost which is visible usually only for very large tables. However it is worth mentioning that, this statistics would be very useful for data skipping which speeds up subsequent operations. In Databricks Runti...
- 0 kudos
- 2284 Views
- 1 replies
- 0 kudos
Resolved! Delta Live Table - landing database?
Where do you specify what database the DLT tables land in?
- 2284 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The target key, when creating the pipeline specifies the database that the tables get published to. Documented here - https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables
- 0 kudos
- 3251 Views
- 1 replies
- 0 kudos
Resolved! Questions on using Docker image with Databricks Container Service
Specifically, we have in mind:* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)* When we want to test an API change, build a Docker image with the relevant changes in a Jar file* Update the job configur...
- 3251 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
>Where do we put custom Jar files when building the Docker image? /databricks/jars>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to tak...
- 0 kudos
- 13206 Views
- 1 replies
- 2 kudos
- 13206 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Use the below endpoint on your workspace. https://your-workspace-name.cloud.databricks.com/version
- 2 kudos
- 3432 Views
- 1 replies
- 0 kudos
Resolved! Z-order or Partitioning? Which is better for Data skipping?
For Delta tables, among Z-order and Partioning which is recommended technique for efficient Data Skipping
- 3432 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Partition pruning is the most efficient way to ensure Data skipping. However, choosing the right column for partitioning is very important. It's common to see choosing the wrong column for partitioning can cause a large number of small file problems ...
- 0 kudos
- 2187 Views
- 2 replies
- 0 kudos
I have several thousands of Delta tables in my Production, what is the best way to get counts
if I might need a dashboard to see increase in number of rows on day to day basis, also a dashboard that shows size of Parquet/Delta files in my Lake?
- 2187 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
val db = "database_name" spark.sessionState.catalog.listTables(db).map(table=>spark.sessionState.catalog.externalCatalog.getTable(table.database.get,table.table)).filter(x=>x.provider.toString().toLowerCase.contains("delta"))The above code snippet wi...
- 0 kudos
- 7791 Views
- 2 replies
- 0 kudos
- 7791 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
If the read stream definition has something similar to val df = spark .read .format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribePattern", "topic.*") .option("startingOffsets", "earliest")resettin...
- 0 kudos
- 2398 Views
- 2 replies
- 0 kudos
Changing default Delta behavior in DBR 8.x for writes
Is there anyway to add a Spark Config that reverts the default behavior when doing tables writes from Delta to Parquet in DBR 8.0+? I know you can simply specify .format("parquet") but that could involve a decent amount of code change for some client...
- 2398 Views
- 2 replies
- 0 kudos
Latest Reply
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
