- 1047 Views
- 2 replies
- 0 kudos
If/else task branches
Hi,I have an If/else task, say A and two other tasks B and C.For the false outcome I would like to execute task B. For the true branch I would like to execute task C followed with task B. What is the correct way to express the dependencies of B on th...
- 1047 Views
- 2 replies
- 0 kudos
- 0 kudos
For sure one solution is to package everything in a separate job. Other options?
- 0 kudos
- 1940 Views
- 2 replies
- 0 kudos
Serverless Cluster Issue
Hi,While using Serverless cluster I'm not able to access dbfs files, saying I don't have permission to the file.But while accessing them using All-purpose cluster I'm able to access them.Why am I facing this issue?
- 1940 Views
- 2 replies
- 0 kudos
- 0 kudos
You might be encountering this issue due to permission differences between the serverless and all-purpose clusters. Serverless environments often have restricted access for enhanced security. Make sure the appropriate IAM roles or access controls are...
- 0 kudos
- 1564 Views
- 1 replies
- 0 kudos
RLS and CLS with delta sharing
RLS and CLS is possible to apply in tables that are shared using unity catalog?
- 1564 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Vanessa, RLS and CLS is not directly supported in delta sharing but as an alternative you could add dynamic views to a share to filter rows and columns. Please find the documentation below - https://docs.databricks.com/aws/en/delta-sharing/create-...
- 0 kudos
- 1234 Views
- 1 replies
- 0 kudos
Difference between creating a schema manually vs schema through SQL code externally?
I have created a bronze schema manually using catalog->create schema. I have provided external location. The "details" table look like this:However, when I created silver schema but this time using sql script i.e. %sqlCREATE SCHEMA xyz.silverMANAGED ...
- 1234 Views
- 1 replies
- 0 kudos
- 0 kudos
When a schema is created manually via the Databricks catalog UI, ownership defaults to the user who created the schema, and ownership properties may not be explicitly recorded by default.On the other hand, when creating a schema using SQL commands, D...
- 0 kudos
- 4070 Views
- 1 replies
- 0 kudos
What is the possible cause in calling the dataricks Job permission API
PATCH /api/2.0/permissions/jobs/{job_id} { "error_code": "INVALID_PARAMETER_VALUE", "message": "Owner permissions cannot be modified via an update / PATCH request if the endpoint does not have a valid owner. Please use a set / PUT request ins...
- 4070 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi team,This seems to be an expected behaviour.The PATCH endpoint for updating job permissions is designed for incremental modifications of existing permissions. However, modifying owner permissions is restricted unless a valid owner already exists i...
- 0 kudos
- 19585 Views
- 5 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 19585 Views
- 5 replies
- 0 kudos
- 0 kudos
@ArbMake sure the below is a file with extension .py and not a notebook. That should resolve the issue./Workspace/Repos/Connectors/Dev/utils/schemas/Comptroller.py'
- 0 kudos
- 4207 Views
- 1 replies
- 0 kudos
How to specify init file path
Hi team,I want to create a job and install some libs to job cluster.1. For job cluster, understand we can specify libraries under task, but if I want to install the lib to the whole cluster seems the only way is to use init script right?2. In my env,...
- 4207 Views
- 1 replies
- 0 kudos
- 0 kudos
There is no direct method to dynamically specify the user ID in this path across environments. If dynamic determination of the user ID per environment is essential, you may need additional scripting or automation to set the correct path before execu...
- 0 kudos
- 4147 Views
- 1 replies
- 1 kudos
Catalog Volume Preview for PDF files
Hi,Does anyone know if and when Databricks will enable the Preview option to work for PDF files stored in a Catalog Volume?Thanks!Nick
- 4147 Views
- 1 replies
- 1 kudos
- 1 kudos
this capability has not been explicitly mentioned or confirmed in the available documentation. Further clarification or inquiries with Databricks product teams may be useful to confirm if such functionality is planned for future releases.
- 1 kudos
- 4358 Views
- 1 replies
- 0 kudos
Access control on view
We've created a view containing PII mapping values in Unity Catalog. We need to grant other users the ability to query this view and access the mapping values, but we must prevent them from seeing the view's definition. Is it possible to grant "exec...
- 4358 Views
- 1 replies
- 0 kudos
- 0 kudos
In Unity Catalog, enabling users to query a view while hiding its definition is currently not supported directly. Unity Catalog requires view definitions to be visible during query execution for metadata access purposes. While querying, users typical...
- 0 kudos
- 5039 Views
- 1 replies
- 0 kudos
Making dynamic tasks like in Airflow, but in Databricks?
I've used Airflow which allows us to create a DAG with dynamic tasks, for example we can have a list of items (such as table names), loop through an operator that accepts a table name and create a task for each table without having to create a new n...
- 5039 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, it is possible to create dynamic tasks in Databricks workflows, similar to the approach using Apache Airflow, by leveraging Databricks' job orchestration capabilities. However, the implementation may differ from Airflow's dynamic DAG creation. D...
- 0 kudos
- 2777 Views
- 3 replies
- 1 kudos
Resolved! Unable to import a library before restarting the cluster
Hi everyone,Anyone has problem when import a library in Databricks notebook? I found it failed to import. Then I restarted the cluster, run it again and it successfully imported the library.My concern here is that: I scheduled to run the notebook at ...
- 2777 Views
- 3 replies
- 1 kudos
- 2354 Views
- 5 replies
- 1 kudos
Resolved! Getting an error "You cannot use dbutils within a spark job"
I am using mlflow to register my custom model using a simple code below. The DatabricksParams extracts all the params from dbutils and sets the params dictionary and dbutils is not used anywhere else within the rest of my code base. The code fails wh...
- 2354 Views
- 5 replies
- 1 kudos
- 2011 Views
- 3 replies
- 0 kudos
Resolved! Auto Loader for copying files on s3
Hey community,I have a folder on s3 with around 5 million small files. On a daily basis new files are added. I would like to simply copy those new files to another folder on s3. My approach is to use an Auto Loader of which I attached the code below....
- 2011 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey LRALVAThe first time running your code I got the error: PicklingError: Could not serialize object: Exception: You cannot use dbutils within a spark job You cannot use dbutils within a spark job or otherwise pickle it.So I changed the copy_single...
- 0 kudos
- 1570 Views
- 1 replies
- 0 kudos
Resolved! Is SSL cert needed for Azure Databricks API calls?
Hi, I'm using Azure Databricks and making API calls to the endpoints with verify=False Eg.response = requests.get('https://%s/api/2.0/clusters/list' % (databricks_domain),headers=request_headers,verify=False)Security scanners are flagging the use of ...
- 1570 Views
- 1 replies
- 0 kudos
- 0 kudos
In API communication with Azure Databricks, it is not necessary to manually install or configure an SSL certificate on the client side because Azure Databricks endpoints inherently use TLS (Transport Layer Security) for secure communication. To addre...
- 0 kudos
- 4925 Views
- 2 replies
- 1 kudos
Docker image fails to initalize - Databricks Container Service
I am trying to use a public docker image hosted in Dockerhub for my job. I have not included public_auth field as per this guide since the image type is public and when the cluster is trying to initialize it fails with the error: Invalid argument: C...
- 4925 Views
- 2 replies
- 1 kudos
- 1 kudos
Did you figure this issue out? I am having the exact same problem - I am also having trouble finding thorough documentation/examples on the use of DCS, so if you have stumbled upon any links in that regard, i'd also appreciate it.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
