- 2352 Views
- 2 replies
- 0 kudos
- 2352 Views
- 2 replies
- 0 kudos
- 2362 Views
- 1 replies
- 0 kudos
Resolved! If I create a shallow clone of a Delta table, then add data to the clone, where is that data stored?
Since a shallow clone only copies the metadata of the original table, I'm wondering where new data would end up. Is it even possible to add data to a shallow clone? Is the data written back to the original source file location?
- 2362 Views
- 1 replies
- 0 kudos
- 0 kudos
Shallow Clones are really useful for short-lived use cases such as testing and experimentation . It duplicates the metadata from the source table - and any new data added would go to the location specified while creating the shallow table. >Is the da...
- 0 kudos
- 11240 Views
- 1 replies
- 0 kudos
Resolved! Can I upload an Excel file to create a table in a workspace?
On the Data tab in the workspace I have the "Create Table" button which gives me the option to upload a local file as a data source. Can I upload an Excel file here? Not sure what kind of files work for this.
- 11240 Views
- 1 replies
- 0 kudos
- 0 kudos
Currently the file types supported there are CSV, JSON and Avro. You could, however upload the excel file to the dbfs path under FileStore and write code in a notebook to parse it and persist it to a table
- 0 kudos
- 2865 Views
- 1 replies
- 0 kudos
Resolved! If I create a clone of a Delta table, does it stay in sync with the original table?
Basically wondering what happens to the clone when updates are made to the original Delta table. Will the changes apply to the cloned table as well?
- 2865 Views
- 1 replies
- 0 kudos
- 0 kudos
The clone is not a replica and so updates made to the original delta table wouldn't be applies to the clone. However, shallow clones reference data files in the source directory. If you run vacuum on the source table, clients will no longer be able t...
- 0 kudos
- 2173 Views
- 1 replies
- 0 kudos
Resolved! I know my partitions are skewed, is there anything I can do to help my performance?
I know the skew in my dataset has the potential to cause issues with my job performance, so just wondering if there is anything I can do to help my performance other than repartitioning the whole dataset.
- 2173 Views
- 1 replies
- 0 kudos
- 0 kudos
For scenarios like this, it is recommend to use a cluster with Databricks Runtime 7.3 LTS or above where AQE is enabled. AQE dynamically handles skew in sort merge join and shuffle hash join by splitting (and replicating if needed) skewed tasks into ...
- 0 kudos
- 1895 Views
- 1 replies
- 0 kudos
Resolved! Do I still need to use skew join hints if I have Adaptive Query Execution enabled?
From what I have read about AQE it seems to do a lot of what skew join hints did automatically. So should I still be using skew hints in my queries? Is there harm in using them?
- 1895 Views
- 1 replies
- 0 kudos
- 0 kudos
With AQE Databricks has the most up-to-date accurate statistics at the end of a query stage and can opt for a better physical strategy and or do optimizations that used to require hints,In the case of skew join hints, is recommended to rely on AQE...
- 0 kudos
- 3056 Views
- 1 replies
- 0 kudos
What is the maximum number of clusters per workspace in Azure Databricks?
It's governed by Azure subscription limits.
- 3056 Views
- 1 replies
- 0 kudos
- 0 kudos
In addition to subscription limits, the total capacity of clusters in each workspace is a function of the masks used for the workspace's enclosing Vnet and the pair of subnets associated with each cluster in the workspace. The masks can be changed if...
- 0 kudos
- 7011 Views
- 2 replies
- 0 kudos
Resolved! Can multiple streams write to a Delta table at the same time?
Wondering if there any dangers to doing this, and if it's a best practice. I'm concerned there could be conflicts but I'm not sure how Delta would handle it.
- 7011 Views
- 2 replies
- 0 kudos
- 0 kudos
>Can multiple streams write to a Delta table at the same time?Yes delta uses optimistic concurrency control and configurable isolation levels>I'm concerned there could be conflicts but I'm not sure how Delta would handle it.Write operations can resul...
- 0 kudos
- 1875 Views
- 1 replies
- 0 kudos
What’s the best instance type to run OPTIMIZE (bin-packing and Z-Ordering) on?
I've been doing some research on optimizing data storage while implementing delta, however, I'm not sure which instance type would be best for this.
- 1875 Views
- 1 replies
- 0 kudos
- 0 kudos
OPTIMIZE as you alluded has two operations , Bin-packing and multi-dimensional clustering ( zorder)Bin-packing optimization is idempotent, meaning that if it is run twice on the same dataset, the second run has no effectZ-Ordering is not idempotent b...
- 0 kudos
- 4420 Views
- 2 replies
- 0 kudos
- 4420 Views
- 2 replies
- 0 kudos
- 0 kudos
Code from DAIS 2021: https://github.com/chengyin38/dais_2021_drifting_away
- 0 kudos
- 1857 Views
- 3 replies
- 0 kudos
What is Autolader in Databricks?
Want to Know what is Autoloader and what are its advantages
- 1857 Views
- 3 replies
- 0 kudos
- 0 kudos
The biggest advantage is the ease with which you can star ingesting data from your Cloud Storage directly into a Delta Table. You can choose Directory Listing mode or File Notification mode, depending on what fits your use case best.
- 0 kudos
- 1146 Views
- 1 replies
- 0 kudos
- 1146 Views
- 1 replies
- 0 kudos
- 0 kudos
Most possibly in future as we progress down our Roadmap.Currently it is per-workspace, and only accessible in Databricks notebooks/jobs.Please refer to our docs:https://docs.databricks.com/applications/machine-learning/feature-store.html#known-limita...
- 0 kudos
- 1520 Views
- 1 replies
- 0 kudos
- 1520 Views
- 1 replies
- 0 kudos
- 0 kudos
It's our new high-performance runtime, using a native vectorized engine developed in C++.Please see our blog for a great overview. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databri...
- 0 kudos
- 1127 Views
- 1 replies
- 0 kudos
- 1127 Views
- 1 replies
- 0 kudos
- 0 kudos
We used to require this, but starting June 9, 2021 we no longer do, and have improved our E2 security posture.See https://docs.databricks.com/administration-guide/account-api/iam-role.html for the current permissions required.
- 0 kudos
- 1487 Views
- 1 replies
- 1 kudos
- 1487 Views
- 1 replies
- 1 kudos
- 1 kudos
Use the calculator herehttps://databricks.com/product/aws-pricing/instance-typesOpen two windows side by side, pick Photon and Non-Photon instances of the same type and compare.
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 481 | |
| 349 | |
| 287 |
