- 896 Views
- 1 replies
- 0 kudos
I have few jobs configured to run against a pool at 10 PM every night. After running some tests, I found that increasing minimum idle instance counts improves the job latencies. However, It wouldn't be needed to have so many VMs idle at other times...
- 896 Views
- 1 replies
- 0 kudos
Latest Reply
Yes you can do so programmatically using the REST APIs. You can edit the settings of a Databricks Pool by using the Instance Pool Edit endpoint and provide the min idle that you desire. This cannot be done via the web UI.
- 966 Views
- 1 replies
- 0 kudos
I see the option to enable multi-cluster load balancing when creating a SQL Endpoint, but I don't know if I should be using it or not. How do I know when I should enable it?
- 966 Views
- 1 replies
- 0 kudos
Latest Reply
It is best to enable multi-cluster load balance on sql endpoints when a lot of users will be running queries concurrently. Load balancing will help isolate the queries and ensure the best performance for all users. If you only have a few users runnin...
- 785 Views
- 1 replies
- 0 kudos
If you have a lot of transactions in a table it seems like the Delta log keeping track of all those transactions would get pretty large. Does the size of the metadata become a problem over time?
- 785 Views
- 1 replies
- 0 kudos
Latest Reply
Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase. By default Delta h...
- 570 Views
- 1 replies
- 0 kudos
If we don't have any datasets to be shared with external companies, does that mean Delta Sharing is not valid for our org? Is there any use case to use it internally?
- 570 Views
- 1 replies
- 0 kudos
Latest Reply
Delta sharing can be done externally and internally. One use case for sharing internally would be if two separate business units would like to share data with each other without exposing their Lakehouse with the other unit.
- 617 Views
- 1 replies
- 0 kudos
Can I read a Delta table directly using Koalas or do I need to read using Spark and then convert the Spark dataframe to a Koalas dataframe?
- 617 Views
- 1 replies
- 0 kudos
Latest Reply
Yes, you can use the "read_delta" function. Documentation.
- 771 Views
- 1 replies
- 2 kudos
I'm working on setting up tooling to allow team members to easily register and load models from a central mlflow model registry via dbconnect. However after following the instructions at the public docs , hitting this error raise _NoDbutilsError
mlfl...
- 771 Views
- 1 replies
- 2 kudos
Latest Reply
You could monkey patch MLFlow's _get_dbutils() with something similar to this to get this working while connecting from dbconnectspark = SparkSession.builder.getOrCreate()
# monkey-patch MLFlow's _get_dbutils()
def _get_dbutils():
return DBUtils(...
by
aladda
• Honored Contributor II
- 599 Views
- 1 replies
- 0 kudos
I see the revision_timestamp paramater on NotebookTask https://docs.databricks.com/dev-tools/api/latest/jobs.html#jobsnotebooktask. An example of how to invoke it would be helpful
- 599 Views
- 1 replies
- 0 kudos
Latest Reply
You can use the databricks built in version control feature, coupled with the NotebookTask Jobs API to specify a specific version of the notebook based on the timestamp of the save defined in unix timestamp formatcurl -n -X POST -H 'Content-Type: app...
- 755 Views
- 1 replies
- 0 kudos
I have read and heard that having too many small files can cause performance problems when reading large data sets. But how do I know if that is an issue I am facing?
- 755 Views
- 1 replies
- 0 kudos
Latest Reply
Databricks SQL endpoint has a query history section which provides additional information to debug / tune queries. One such metric under execution details is the number of files read. For ETL/Data science workloads, you could use the Spark UI of the ...
- 1462 Views
- 1 replies
- 1 kudos
In databricks is there a way to display the spark job process in a dashboard? I have a simple dashboard that displays a table, but the main spark job behind it takes 15 minutes to run. Is there a way to show the spark job progress bar in a dashboard?
- 1462 Views
- 1 replies
- 1 kudos
Latest Reply
The best way to do so would be to collect data about the job run using the REST API (runs get endpoint). This endpoint provides as much metadata as possible. You may need to use other endpoints to get the job or run ids in order to get the correct in...
- 664 Views
- 0 replies
- 0 kudos
I have a dataframe that looks like the following,+-------+--------+
|Charges| Status|
+-------+--------+
| 495.6| Denied|
|1806.28| Denied|
| 261.3|Accepted|
| 8076.5|Accepted|
|1041.24| Denied|
| 507.88| Denied|
| 208.0|Accepted|
| 152.49| ...
- 664 Views
- 0 replies
- 0 kudos
- 1119 Views
- 1 replies
- 0 kudos
I know I have the option to delete rows from a Delta table when running a merge. But I'm confused about how that would actually affect the files that contain the deleted records. Are those files deleted, or are they rewritten, or what?
- 1119 Views
- 1 replies
- 0 kudos
Latest Reply
Delta implements MERGE by physically rewriting existing files. It is implemented in two steps.Perform an inner join between the target table and source table to select all files that have matches.Perform an outer join between the selected files in t...
