- 1097 Views
- 1 replies
- 0 kudos
Is it possible to disable file download in Volumes interface?
Workspace security administration panel offers to disable downloads in notebook folders and workspaces. However, it seems that even if all those downloads are disabled, the "Volumes" panel of Unity Catalog still offers a file download button. Is it p...
- 1097 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @staskh ,Unfortunately, I don't think it is possible to disable it via UI currently. But volumes are governed by UC permission, so maybe you can try to set permission to read/write to approved group of users and take permission of users who should...
- 0 kudos
- 4341 Views
- 1 replies
- 0 kudos
Issues with Delta Sharing API when using Service Principal Token
Hello,I am currently working with the Delta Sharing API and have encountered an issue when using a Service Principal token for authentication. The API call returns the following error:[CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from empty datase...
- 4341 Views
- 1 replies
- 0 kudos
- 0 kudos
Please find the response below: 1) The Delta Sharing API supports both personal access tokens and service principal tokens for authentication. 2) Service principals need to be granted specific roles and permissions to access data. This includes assi...
- 0 kudos
- 4030 Views
- 1 replies
- 0 kudos
- 4030 Views
- 1 replies
- 0 kudos
- 0 kudos
The below documentation shows how to install libraries in a cluster. https://docs.databricks.com/en/libraries/cluster-libraries.html#install-a-library-on-a-cluster
- 0 kudos
- 1110 Views
- 3 replies
- 2 kudos
GCP Databricks GKE cluster with 4 nodes
I am working on setting up GCP Databricks and successfully created first GCP-Databricks workspace, but what I observed is it is incurring additional charges even i am using 14days free trail. It is GKE cluster with 4 nodes which are spin up as part o...
- 1110 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you @Louis_Frolio,Just want to dig more into this as is there any way to reduce this nodes using CLI or creating customer managed network.
- 2 kudos
- 4220 Views
- 2 replies
- 2 kudos
Resolved! Databricks All-purpose compute Pricing
Hello, I am now struggling how to calculate the cost of my job cluster.My configuration is as below:If I have to run the above cluster 18 hours per day, in Standard Tier and East Asia Region, how much will be the pricing of the cluster?Any help provi...
- 4220 Views
- 2 replies
- 2 kudos
- 2 kudos
@karen_c Let me make a small correction.It seems that you have checked the option for Spot Instances, which should make the cost slightly lower. Please refer to the far-right column of the attached pricing table for more details.Additionally, you hav...
- 2 kudos
- 5881 Views
- 5 replies
- 2 kudos
Resolved! Azure Databricks Unity Catalog - Cannot access Managed Volume in notebook
The problemAfter setting up Unity Catalog and a managed Volume, I can upload/download files to/from the volume, on Databricks Workspace UI.However, I cannot access the volume from notebook. I created an All-purpose compute, and run dbutils.fs.ls("/Vo...
- 5881 Views
- 5 replies
- 2 kudos
- 2 kudos
I found the reason and a solution, but I feel this is a bug. And I wonder what is the best practice.When I enable the ADSL Gen2's Public network access from all networks as shown below, I can access the volume from a notebook.However, if I enable the...
- 2 kudos
- 1888 Views
- 2 replies
- 1 kudos
Access to system.billing.usage tables
I have Account, Marketplace, Billing Admin roles. I have visibility to system.billing.list_prices table only.How do I get access to system.billing.usage tables? Databricks instance is on AWS.Thanks
- 1888 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Alberto_Umana, Thanks for your response. I needed Metastore Admin permissions too. In account console, I changed the Metastore Admin to be a group, became a part of the group. With this other tables were visible. With this permission using the gr...
- 1 kudos
- 2746 Views
- 3 replies
- 0 kudos
Best Practices for Daily Source-to-Bronze Data Ingestion in Databricks
How can we effectively manage source-to-bronze data ingestion from a project perspective, particularly when considering daily scheduling strategies using either Auto Loader or Serverless Warehouse COPY INTO commands?
- 2746 Views
- 3 replies
- 0 kudos
- 0 kudos
No, it is not a strict requirement. You can have a single node job cluster run the job if the job is small.
- 0 kudos
- 4859 Views
- 5 replies
- 0 kudos
Any Databricks system tables contain info of the saved/pre-defined queries
How can I find the saved/pre-defined queries in Databricks system tables?system.query.history seems NOT having the info, like query-id or query-name
- 4859 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi Bryan, Databricks system tables do not store saved queries. Query history table captures the query execution details, including: Statement IDExecution statusUser who ran the queryStatement text (if not encrypted)Statement typeExecution durationRes...
- 0 kudos
- 2344 Views
- 2 replies
- 2 kudos
Resolved! Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture
Hi everyone,I’m working on implementing Structured Streaming in Databricks to capture Change Data Capture (CDC) as part of a Medallion Architecture (Bronze, Silver, and Gold layers). While Microsoft’s documentation provides a theoretical approach, I’...
- 2344 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @JissMathew ,Do you have access to databricks academy? I believe in their data engineering track there's pleny of example notebooks.Or you can try dbdemos. For example, here you can find demo notebook for autoloaderDatabricks Autoloader (cloudfile...
- 2 kudos
- 1681 Views
- 1 replies
- 1 kudos
Resolved! Multiple volumes from same external location?
Hey all,Do you know if it's possible to create multiple volumes referencing the same s3 bucket from the same external location?For example, if I have two workspaces (test and prod) testing different versions of pipeline code but with static data I'd ...
- 1681 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, it is a limitation, and it is not possible to create multiple volumes referencing the same S3 bucket. This restriction ensures consistency and prevents conflicts when accessing the same data source.Possible Solution:Use subdirectories within the...
- 1 kudos
- 908 Views
- 3 replies
- 0 kudos
Environment Notification / Message
Is it somehow possible to create a message or alerting for specific Databricks environments to make people more aware that they are using e.g. a PROD environment?It can be reflected in the environment name like "dev" or "prod", yes. But it would be n...
- 908 Views
- 3 replies
- 0 kudos
- 0 kudos
Seems that for Azure the process is a little bit different you might follow steps in https://learn.microsoft.com/en-us/azure/databricks/resources/ideas
- 0 kudos
- 963 Views
- 2 replies
- 0 kudos
getting job_parameters object with sql
Hey,In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.While job_parameters exists in system.workflow.job_run_timeline table, it is not populated ...
- 963 Views
- 2 replies
- 0 kudos
- 0 kudos
@yairofek wrote:Hey,In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.While job_parameters exists in system.workflow.job_run_timeline table, it ...
- 0 kudos
- 9603 Views
- 7 replies
- 6 kudos
Resolved! Secret scope with Azure RBAC
Hello!We have lots of Azure keyvaults that we use in our Azure Databricks workspaces. We have created secret scopes that are backed by the keyvaults. Azure supports two ways of authenticating to keyvaults:- Access policies, which has been marked as l...
- 9603 Views
- 7 replies
- 6 kudos
- 6 kudos
@Chamak You can find 'AzureDatabricks' in User, group or service principal assignment. You dont need to find application id, as it will automatically displayed when you add AzureDatabricks as member. cc: @daniel_sahal
- 6 kudos
- 2266 Views
- 5 replies
- 2 kudos
Resolved! NAT gateway with public IP for SCC disabled Databricks cluster
Hi Team, We need to have single public IP for all outbound traffic flowing through our Databricks cluster. The Secure Cluster Connectivity (SCC) is disabled for our cluster and currently we get dynamic public IPs assigned to the VMs under managed res...
- 2266 Views
- 5 replies
- 2 kudos
