- 2171 Views
- 1 replies
- 0 kudos
Resolved! Is it possible to have time travel capability but also be able to selectively vacuum ?
I would like to have time travel functionality for several months but that ends up adding up storage costs. Is there some way to have mix of vacuum and time travel?
- 2171 Views
- 1 replies
- 0 kudos
- 0 kudos
There is not a way to time travel past the vacuum retention period. If you would like to time travel back lets say 3 months then you are not able to vacuum a shorter time frame.
- 0 kudos
- 3199 Views
- 1 replies
- 0 kudos
Overwriting Delta Table Using SQL
I have a delta table that is updated nightly, that I drop and recreate at the start of each day. However, this isn't ideal because every time I drop the table I lose all the info in the transaction log. Is there a way that I can do the equivalent of:...
- 3199 Views
- 1 replies
- 0 kudos
- 0 kudos
I think you are looking for the INSERT OVERWRITE command in Spark SQL. Check out the documentation here: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-dml-insert-overwrite-table.html
- 0 kudos
- 2192 Views
- 2 replies
- 0 kudos
Can I query tables I have created in my Databricks workspace using Tableau?
I have created Delta tables in my Databricks workspace and would like to access them from Tableau. Is this possible?
- 2192 Views
- 2 replies
- 0 kudos
- 0 kudos
Yeah - here is the link with details on how to integrate with Tabelu's different products https://docs.databricks.com/integrations/bi/tableau.html
- 0 kudos
- 2770 Views
- 1 replies
- 0 kudos
Is there a way to change the default cluster setting after a notebook has been created?
When you create a notebook, you are prompted to specify a default cluster that it will connect to. Is there a way to change that setting after the notebook is created?
- 2770 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes of course, notebooks are not exclusively tied to a specific cluster so you can pick any available/visible cluster to attach the notebook on when you want to run it.Also please keep in mind that by doing this half-way through executing a notebook,...
- 0 kudos
- 4482 Views
- 1 replies
- 0 kudos
Find Databricks SQL endpoints runtime
Is there a way to find out which runtime SQL endpoints are running?
- 4482 Views
- 1 replies
- 0 kudos
- 0 kudos
In the UI, Databricks will list the running endpoints on top. Programmatically you can get information about the endpoints using the REST APIs. You will likely need to use a combo of the list endpoint to get all the endpoints. The for each endpoint u...
- 0 kudos
- 2185 Views
- 1 replies
- 1 kudos
- 2185 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, you can use the SCIM API integration to provision both users and groups. We have examples for Okta, Azure AD and OneLogin, but any SCIM-enabled IdP should suffice.
- 1 kudos
- 2532 Views
- 1 replies
- 0 kudos
Resolved! Can I schedule Databricks pools to have different minimum idle instance counts at different times of the day
I have few jobs configured to run against a pool at 10 PM every night. After running some tests, I found that increasing minimum idle instance counts improves the job latencies. However, It wouldn't be needed to have so many VMs idle at other times...
- 2532 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes you can do so programmatically using the REST APIs. You can edit the settings of a Databricks Pool by using the Instance Pool Edit endpoint and provide the min idle that you desire. This cannot be done via the web UI.
- 0 kudos
- 1624 Views
- 1 replies
- 0 kudos
- 1624 Views
- 1 replies
- 0 kudos
- 0 kudos
It does not. You will be able to retain past runs and view them for up to 60 days regardless if you updating the configuration of the job.
- 0 kudos
- 3113 Views
- 1 replies
- 0 kudos
Resolved! When should I turn on multi-cluster load balancing on SQL Endpoints?
I see the option to enable multi-cluster load balancing when creating a SQL Endpoint, but I don't know if I should be using it or not. How do I know when I should enable it?
- 3113 Views
- 1 replies
- 0 kudos
- 0 kudos
It is best to enable multi-cluster load balance on sql endpoints when a lot of users will be running queries concurrently. Load balancing will help isolate the queries and ensure the best performance for all users. If you only have a few users runnin...
- 0 kudos
- 9509 Views
- 1 replies
- 0 kudos
- 9509 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes you can. Databricks maintains a history of your job runs for up to 60 days. If you need to preserve job runs, Databricks recommends that you export results before they expire. For more information, see https://docs.databricks.com/jobs.html#export...
- 0 kudos
- 2223 Views
- 2 replies
- 0 kudos
- 2223 Views
- 2 replies
- 0 kudos
- 2711 Views
- 1 replies
- 0 kudos
Resolved! How much space does the metadata for a Delta table take up?
If you have a lot of transactions in a table it seems like the Delta log keeping track of all those transactions would get pretty large. Does the size of the metadata become a problem over time?
- 2711 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase. By default Delta h...
- 0 kudos
- 2139 Views
- 1 replies
- 0 kudos
Resolved! Delta Sharing internally?
If we don't have any datasets to be shared with external companies, does that mean Delta Sharing is not valid for our org? Is there any use case to use it internally?
- 2139 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta sharing can be done externally and internally. One use case for sharing internally would be if two separate business units would like to share data with each other without exposing their Lakehouse with the other unit.
- 0 kudos
- 2049 Views
- 1 replies
- 0 kudos
Is it possible to read a Delta table directly using Koalas?
Can I read a Delta table directly using Koalas or do I need to read using Spark and then convert the Spark dataframe to a Koalas dataframe?
- 2049 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can use the "read_delta" function. Documentation.
- 0 kudos
- 2552 Views
- 1 replies
- 2 kudos
Resolved! Unable to get mlflow central model registry to work with dbconnect.
I'm working on setting up tooling to allow team members to easily register and load models from a central mlflow model registry via dbconnect. However after following the instructions at the public docs , hitting this error raise _NoDbutilsError mlfl...
- 2552 Views
- 1 replies
- 2 kudos
- 2 kudos
You could monkey patch MLFlow's _get_dbutils() with something similar to this to get this working while connecting from dbconnectspark = SparkSession.builder.getOrCreate() # monkey-patch MLFlow's _get_dbutils() def _get_dbutils(): return DBUtils(...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
