- 12864 Views
- 3 replies
- 1 kudos
Schedule a Notebook Dashboard
Hey all,I have a workflow that updates a delta table, and then runs a notebook that generates a dashboard. I was hoping that by adding this second step that the dashboard would get updated to show the most current data, instead of the user needing to...
- 12864 Views
- 3 replies
- 1 kudos
- 1 kudos
@ehpogue at the end of the day I just used chatgpt to rewrite a bunch of python code into SQL and mash together all of the temporary views into one giant query. hacky but it did work
- 1 kudos
- 1541 Views
- 1 replies
- 0 kudos
Managed Table
Hi Experts,I have a workspace created and associated a metastore with it, the metastore points to a storage location USDATA and then I create two catalogs in the workspace and one is using default meta store as the external storage location and other...
- 1541 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mani2105,if i create a table in the sales catalog without specifiying any external location, will the tables created be managed and will go to the Sales storage account Yes, if you create a table in the sales catalog without specifying any exter...
- 0 kudos
- 4417 Views
- 2 replies
- 1 kudos
Resolved! Unity Catalog Metastore Details
hi,I would like to seek response to my following questions regarding Unity Catalog Metastore's path.While configuring metastore, designating a metastore storage account (in case of Azure, it's ADLS Gen2) seems to be an optional thing. In case I confi...
- 4417 Views
- 2 replies
- 1 kudos
- 1 kudos
The storage container you configure for the metastore will contain the files of managed tables and volumes. The metadata is stored in a database of the Databricks control plane.
- 1 kudos
- 7079 Views
- 1 replies
- 0 kudos
DLT full refresh
Running a task with full refresh in delta live tables removes existing data and reloads it from scratch. We are ingesting data from an event hub topic and from files. The event hub topic stores messages for seven days after arrival. If we would run a...
- 7079 Views
- 1 replies
- 0 kudos
- 0 kudos
I know it's a bit after the fact, but in case you didn't solve it, I came across this article in the Databricks documentation. You can set pipelines.reset.allowed to false on a table to prevent a full refresh of a table. Ref: https://docs.databrick...
- 0 kudos
- 2633 Views
- 1 replies
- 0 kudos
Resolved! can't create cluster
Hi,I am trying to learn databricks and created a student account In Azure.currently I have 100$ available with me.But when i try to create a basic cluster with 1DBU/hour I am getting warning and not able to create.This account may not have enough CP...
- 2633 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi This is because new accounts on azure have lower quotas: https://learn.microsoft.com/en-us/azure/quotas/quotas-overviewTry to choos another Node-Type like Standard_4Fs, which has less CPU. And you can increase th quota: https://learn.microsoft.com...
- 0 kudos
- 4973 Views
- 7 replies
- 2 kudos
Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' :
Hi there,My cluster version is 15.4 LTS, and the workspace has UC enabled. When I used the initialization script to install ODBC Driver 17 for SQL Server, there were no errors and the cluster started successfully. But when I use ODBC Driver 17 for SQ...
- 4973 Views
- 7 replies
- 2 kudos
- 2 kudos
Also is INIT script route is the only route? or is there any other option available?the reason I am asking, some time back we had a call with DBK and then they mentioned like usage of INIT script is not the right way or so.. cant recall exact explana...
- 2 kudos
- 5569 Views
- 4 replies
- 3 kudos
Resolved! Workflow concurrent runs not working as expected
Hello All,I am trying to fetch data from different sources for tables driven by a metadata table. Data will get fetched from sources using jdbc connector for each table mentioned in the metadata table. A scheduled job is responsible for fetching the ...
- 5569 Views
- 4 replies
- 3 kudos
- 3 kudos
Soo.. You use a loop to go through metadata table and then retrieve and ingest files using JDBC?If so, then the concurrent runs won't be helpful. Concurrent runs means the number of how many runs of that job can be ran side by side. For you, this wou...
- 3 kudos
- 2829 Views
- 4 replies
- 2 kudos
Resolved! Restricting access to secrets
Hi. I want to restrict access to secrets to a security group, as the secrets can be used to retrieve sensitive data only a few people should see. Up until now, we have been using KV-backed secret scopes, but as it's sufficient that Databricks has the...
- 2829 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Johan, this should work for restriction: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets.Fine granulat access based on secrets is currently not possible.BR
- 2 kudos
- 17604 Views
- 5 replies
- 2 kudos
databricks single user cluster is not able to assign service principals
I want to set the databricks cluster Access mode to single user and assign the service principal account to the user. In other words, after creating a single mode cluster, how can I access only servcie principals?
- 17604 Views
- 5 replies
- 2 kudos
- 2 kudos
So, here is an alternative to either the UI (doesn't work actually; not possible) or the CLI. Use the Databricks API.endpoint_change = f"{databricksURL}/api/2.1/clusters/edit" # change single ownerpayload_change = { "cluster_id": cluster_id ,"clu...
- 2 kudos
- 5873 Views
- 4 replies
- 0 kudos
Databricks Runtime 13.3 - can I use Databricks Connect without Unity Catalog?
Hello all,The official documentation for Databricks Connect states that, for Databricks Runtime versions 13.0 and above, my cluster needs to have Unity Catalog enabled for me to use Databricks Connect, and use a Databricks cluster through an IDE like...
- 5873 Views
- 4 replies
- 0 kudos
- 0 kudos
Did you configure databricks connect without UnitCatalog at the end? If you managed to do that can you share with me how?
- 0 kudos
- 10877 Views
- 2 replies
- 2 kudos
Identity column value of Databricks delta table is not started with 0 and increaed by 1. It always started with something like 1 or 2 and increased by 2. Below is the sample code and any logical input here is appreciated
spark.sql("CREATE TABLE integrated.TrailingWeeks(ID bigint GENERATED BY DEFAULT AS IDENTITY (START WITH 0 increment by 1) ,Week_ID int NOT NULL) USING delta OPTIONS (path 'dbfs:/<Path in Azure datalake>/delta')")
- 10877 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,When you define an identity column in Databricks with GENERATED BY DEFAULT AS IDENTITY (START WITH 0 INCREMENT BY 1), it is expected to start at 0 and increment by 1. However, due to Databricks' distributed architecture, the values may not be str...
- 2 kudos
- 1490 Views
- 2 replies
- 0 kudos
Cluster is not starting
Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...
- 1490 Views
- 2 replies
- 0 kudos
- 27661 Views
- 9 replies
- 9 kudos
Cluster occasionally fails to launch
I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...
- 27661 Views
- 9 replies
- 9 kudos
- 9 kudos
Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...
- 9 kudos
- 22315 Views
- 7 replies
- 6 kudos
Resolved! Downloading sharepoint lists using python
Hello, I am trying to download lists from SharePoint into a pandas dataframe. However I cannot get any information successfully. I have attempted many solution mentioned in stackoverflow. Below is one of those attempts: # https://pypi.org/project/sha...
- 22315 Views
- 7 replies
- 6 kudos
- 6 kudos
Hello, I have gotten the code to work by using office365 library instead.
- 6 kudos
- 9803 Views
- 5 replies
- 0 kudos
Resolved! Problem with SQL Warehouse (Serverless)
I get the following error message on the attempt to use SQL Warehouse (Serverless) compute with Materialized Views (a simple interaction, e.g. DML, UI sample lookup). The MVs are created off the back of Federated Tables (Postgresql), MVs are created ...
- 9803 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey,To clarify, as I think I'm potentially hitting Databricks unintended "functionality".Materialised Views are managed by DLT pipeline, which was deployed with DABs off CI/CD pipeline,DLT Pipeline runs a notebook with Python code creating MVs dynami...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 568 | |
| 349 | |
| 287 |
