- 2311 Views
- 2 replies
- 2 kudos
Resolved! Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture
Hi everyone,I’m working on implementing Structured Streaming in Databricks to capture Change Data Capture (CDC) as part of a Medallion Architecture (Bronze, Silver, and Gold layers). While Microsoft’s documentation provides a theoretical approach, I’...
- 2311 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @JissMathew ,Do you have access to databricks academy? I believe in their data engineering track there's pleny of example notebooks.Or you can try dbdemos. For example, here you can find demo notebook for autoloaderDatabricks Autoloader (cloudfile...
- 2 kudos
- 1645 Views
- 1 replies
- 1 kudos
Resolved! Multiple volumes from same external location?
Hey all,Do you know if it's possible to create multiple volumes referencing the same s3 bucket from the same external location?For example, if I have two workspaces (test and prod) testing different versions of pipeline code but with static data I'd ...
- 1645 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, it is a limitation, and it is not possible to create multiple volumes referencing the same S3 bucket. This restriction ensures consistency and prevents conflicts when accessing the same data source.Possible Solution:Use subdirectories within the...
- 1 kudos
- 885 Views
- 3 replies
- 0 kudos
Environment Notification / Message
Is it somehow possible to create a message or alerting for specific Databricks environments to make people more aware that they are using e.g. a PROD environment?It can be reflected in the environment name like "dev" or "prod", yes. But it would be n...
- 885 Views
- 3 replies
- 0 kudos
- 0 kudos
Seems that for Azure the process is a little bit different you might follow steps in https://learn.microsoft.com/en-us/azure/databricks/resources/ideas
- 0 kudos
- 937 Views
- 2 replies
- 0 kudos
getting job_parameters object with sql
Hey,In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.While job_parameters exists in system.workflow.job_run_timeline table, it is not populated ...
- 937 Views
- 2 replies
- 0 kudos
- 0 kudos
@yairofek wrote:Hey,In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.While job_parameters exists in system.workflow.job_run_timeline table, it ...
- 0 kudos
- 9470 Views
- 7 replies
- 6 kudos
Resolved! Secret scope with Azure RBAC
Hello!We have lots of Azure keyvaults that we use in our Azure Databricks workspaces. We have created secret scopes that are backed by the keyvaults. Azure supports two ways of authenticating to keyvaults:- Access policies, which has been marked as l...
- 9470 Views
- 7 replies
- 6 kudos
- 6 kudos
@Chamak You can find 'AzureDatabricks' in User, group or service principal assignment. You dont need to find application id, as it will automatically displayed when you add AzureDatabricks as member. cc: @daniel_sahal
- 6 kudos
- 2214 Views
- 5 replies
- 2 kudos
Resolved! NAT gateway with public IP for SCC disabled Databricks cluster
Hi Team, We need to have single public IP for all outbound traffic flowing through our Databricks cluster. The Secure Cluster Connectivity (SCC) is disabled for our cluster and currently we get dynamic public IPs assigned to the VMs under managed res...
- 2214 Views
- 5 replies
- 2 kudos
- 891 Views
- 1 replies
- 0 kudos
renaming resource group on azure is it possible?
Hello community,I deployed one month ago a resource group with a particular name, where inside there are two databricks workspaces deployed. Is it possible to rename the resource group without any problem? Or do I need to move the existed dbws to a n...
- 891 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeremy98 ,Unfortunately, you cannot rename resource group. You need to create new resource group and recreate all required resources.
- 0 kudos
- 1038 Views
- 1 replies
- 0 kudos
Resolved! System tables on workspace level
I could be mistaken, but it seem like the systems table contain data of all workspaces, even workspaces that you don't have access to. According to "least principle privilege" idea, I do not think that's a good idea.If forementioned is correct, has s...
- 1038 Views
- 1 replies
- 0 kudos
- 0 kudos
As per documentation it is confirmed that system tables include data from all workspaces in your account, but they can only be accessed by a workspace with Unity Catalog, you can restrict which admins has access to this system tables.It is not possib...
- 0 kudos
- 7630 Views
- 1 replies
- 0 kudos
Managing databricks workspace permissions
I need assistance with writing API/Python code to manage a Databricks workspace permissions database(unity catalog). The task involves obtaining a list of workspace details from the account console, which includes various details like Workspace name,...
- 7630 Views
- 1 replies
- 0 kudos
- 0 kudos
Here's a start. https://docs.databricks.com/api/workspace/workspacebindings/updatebindings As far as coding, I use CURL. See attachment as to the syntax. Note the example in the attachment is for Workspace notebooks, as opposed to Workspace envir...
- 0 kudos
- 816 Views
- 1 replies
- 0 kudos
Get hardware metrics like CPU usage, memory usage and send it to Azure Monitor
Hello guys,I would like to get hardware metrics like server load distribution, CPU utilization, memory utilization and send it to Azure Monitor. Is there any way to do this? Can you help me with this doubt?Thanks.
- 816 Views
- 1 replies
- 0 kudos
- 0 kudos
@xzero-trustx wrote:Hello guys,I would like to get hardware metrics like server load distribution, CPU utilization, memory utilization and send it to Azure Monitor. Is there any way to do this? Can you help me with this doubt?Thanks.Hello!Yes, you ca...
- 0 kudos
- 835 Views
- 1 replies
- 1 kudos
Issue running notebook from another notebook in Job cluster
Hi,I have situation where I can run my notebook without any issue when I use a 'normal' cluster. However, when I run the exact same notebook in a job cluster it fails.It fails at the point where it runs the cell:`%run ../utils/some_other_notebook`And...
- 835 Views
- 1 replies
- 1 kudos
- 1 kudos
Not sure what went wrong but after pulling the sources (notebooks) again from GIT it now works both for my 'normal' cluster and the 'job' cluster.Case closed for me...
- 1 kudos
- 1141 Views
- 2 replies
- 5 kudos
- 1141 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi,Yes, Databricks Asset Bundles (DABs) can be used with a Standard Tier Databricks workspace. The use of DABs is not directly tied to the workspace pricing tier but rather to the configuration of your workspace and integration with CI/CD pipelines.
- 5 kudos
- 730 Views
- 2 replies
- 1 kudos
Meta data for cloned table
Hi, guys!How I can find out whether a table is cloned right now - hopefully by querying some meta data (information_schema or the like)?Thanks! Sebastian
- 730 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SeBaFlu ,To determine whether a table in Databricks is a clone (created using Delta Lake's CREATE TABLE CLONE), you can use Delta Lake's metadata and DESCRIBE HISTORY command.
- 1 kudos
- 3283 Views
- 2 replies
- 0 kudos
Resolved! Download Dashboard as PDF
I see several reference in Databricks documentation to export a Dashboard as a PDF, yet I have no options for format when I download it, it creates a json file. Is there a way to download a Dashboard as a PDF?
- 3283 Views
- 2 replies
- 0 kudos
- 0 kudos
Actually you can download a Dashboard to a PDF if it is a legacy dashboard. It does not appear as an option for an AI/BI dashboard.
- 0 kudos
- 1601 Views
- 4 replies
- 0 kudos
Resolved! Databricks apps, data plane configuration not supported
Unable to create app, get 'This workspace has a data plane configuration that is not yet supported' message. Is there something specific I should look for configuration wise to correct the issue? Azure hosted. Virtual network.
- 1601 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @rew_data, You might want to check if you region is available to use Databricks Apps, please refer to: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/ The error message "This workspace has a data plane configuratio...
- 0 kudos
