- 2193 Views
- 1 replies
- 1 kudos
Receiving a "Databricks Delta is not enabled on your account" error
The team is using Databricks Light for some pipeline development and would like to leverage Delta but are running into this error? "Databricks Delta is not enabled on your account"How can we enable Delta for our account
- 2193 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks Light is the open source Apache Spark runtime and does not come with any type of client for Delta Lake pre-installed. You'll need to manually install open source Delta Lake in order to do any reads or writes.See our docs and release notes ...
- 1 kudos
- 2400 Views
- 1 replies
- 0 kudos
Resolved! What happens when you write to same Delta table from two different clusters at the same time?
- 2400 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Lake uses optimistic concurrency control to provide transactional guarantees between writes. Under this mechanism, writes operate in three stages:Read: Reads (if needed) the latest available version of the table to identify which files need to ...
- 0 kudos
- 1653 Views
- 1 replies
- 0 kudos
Resolved! Is it possible to have time travel capability but also be able to selectively vacuum ?
I would like to have time travel functionality for several months but that ends up adding up storage costs. Is there some way to have mix of vacuum and time travel?
- 1653 Views
- 1 replies
- 0 kudos
- 0 kudos
There is not a way to time travel past the vacuum retention period. If you would like to time travel back lets say 3 months then you are not able to vacuum a shorter time frame.
- 0 kudos
- 2685 Views
- 1 replies
- 0 kudos
Overwriting Delta Table Using SQL
I have a delta table that is updated nightly, that I drop and recreate at the start of each day. However, this isn't ideal because every time I drop the table I lose all the info in the transaction log. Is there a way that I can do the equivalent of:...
- 2685 Views
- 1 replies
- 0 kudos
- 0 kudos
I think you are looking for the INSERT OVERWRITE command in Spark SQL. Check out the documentation here: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-dml-insert-overwrite-table.html
- 0 kudos
- 1677 Views
- 2 replies
- 0 kudos
Can I query tables I have created in my Databricks workspace using Tableau?
I have created Delta tables in my Databricks workspace and would like to access them from Tableau. Is this possible?
- 1677 Views
- 2 replies
- 0 kudos
- 0 kudos
Yeah - here is the link with details on how to integrate with Tabelu's different products https://docs.databricks.com/integrations/bi/tableau.html
- 0 kudos
- 2137 Views
- 1 replies
- 0 kudos
Is there a way to change the default cluster setting after a notebook has been created?
When you create a notebook, you are prompted to specify a default cluster that it will connect to. Is there a way to change that setting after the notebook is created?
- 2137 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes of course, notebooks are not exclusively tied to a specific cluster so you can pick any available/visible cluster to attach the notebook on when you want to run it.Also please keep in mind that by doing this half-way through executing a notebook,...
- 0 kudos
- 3739 Views
- 1 replies
- 0 kudos
Find Databricks SQL endpoints runtime
Is there a way to find out which runtime SQL endpoints are running?
- 3739 Views
- 1 replies
- 0 kudos
- 0 kudos
In the UI, Databricks will list the running endpoints on top. Programmatically you can get information about the endpoints using the REST APIs. You will likely need to use a combo of the list endpoint to get all the endpoints. The for each endpoint u...
- 0 kudos
- 1612 Views
- 1 replies
- 1 kudos
- 1612 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, you can use the SCIM API integration to provision both users and groups. We have examples for Okta, Azure AD and OneLogin, but any SCIM-enabled IdP should suffice.
- 1 kudos
- 2028 Views
- 1 replies
- 0 kudos
Resolved! Can I schedule Databricks pools to have different minimum idle instance counts at different times of the day
I have few jobs configured to run against a pool at 10 PM every night. After running some tests, I found that increasing minimum idle instance counts improves the job latencies. However, It wouldn't be needed to have so many VMs idle at other times...
- 2028 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes you can do so programmatically using the REST APIs. You can edit the settings of a Databricks Pool by using the Instance Pool Edit endpoint and provide the min idle that you desire. This cannot be done via the web UI.
- 0 kudos
- 1243 Views
- 1 replies
- 0 kudos
- 1243 Views
- 1 replies
- 0 kudos
- 0 kudos
It does not. You will be able to retain past runs and view them for up to 60 days regardless if you updating the configuration of the job.
- 0 kudos
- 2469 Views
- 1 replies
- 0 kudos
Resolved! When should I turn on multi-cluster load balancing on SQL Endpoints?
I see the option to enable multi-cluster load balancing when creating a SQL Endpoint, but I don't know if I should be using it or not. How do I know when I should enable it?
- 2469 Views
- 1 replies
- 0 kudos
- 0 kudos
It is best to enable multi-cluster load balance on sql endpoints when a lot of users will be running queries concurrently. Load balancing will help isolate the queries and ensure the best performance for all users. If you only have a few users runnin...
- 0 kudos
- 7732 Views
- 1 replies
- 0 kudos
- 7732 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes you can. Databricks maintains a history of your job runs for up to 60 days. If you need to preserve job runs, Databricks recommends that you export results before they expire. For more information, see https://docs.databricks.com/jobs.html#export...
- 0 kudos
- 1605 Views
- 2 replies
- 0 kudos
- 1605 Views
- 2 replies
- 0 kudos
- 2156 Views
- 1 replies
- 0 kudos
Resolved! How much space does the metadata for a Delta table take up?
If you have a lot of transactions in a table it seems like the Delta log keeping track of all those transactions would get pretty large. Does the size of the metadata become a problem over time?
- 2156 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase. By default Delta h...
- 0 kudos
- 1636 Views
- 1 replies
- 0 kudos
Resolved! Delta Sharing internally?
If we don't have any datasets to be shared with external companies, does that mean Delta Sharing is not valid for our org? Is there any use case to use it internally?
- 1636 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta sharing can be done externally and internally. One use case for sharing internally would be if two separate business units would like to share data with each other without exposing their Lakehouse with the other unit.
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 456 | |
| 349 | |
| 287 |
