- 1475 Views
- 1 replies
- 0 kudos
Handling files used more than once in a streaming pipeline
I am implementing Structured Streaming using Delta Live Table. I want to delete the parquet files once they are used. What options should I set so that the files loaded in S3 are not deleted?
- 1475 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, It sounds like your Structured Streaming source is S3, in which case the easiest solution is likely to manage the stream source using an S3 Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)...
- 0 kudos
- 2398 Views
- 0 replies
- 0 kudos
Unable to Install Cluster-Scoped Libraries on Runtime >13.3
Hello team,I'm trying to upgrade our databricks runtime to 13.3 from 9.1, but i've been having issues installing libraries on the compute from our internal artifactoryHowever, when I tried this on a unity-catalog enabled workspace, it works seamless...
- 2398 Views
- 0 replies
- 0 kudos
- 11098 Views
- 7 replies
- 2 kudos
Run Task as Service Principal with Code in Azure DevOps Repo
Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...
- 11098 Views
- 7 replies
- 2 kudos
- 2 kudos
@pgruetter​ :To enable a service principal to access a specific Azure DevOps repository, you need to grant it the necessary permissions at both the organization and repository levels.Here are the steps to grant the service principal the necessary per...
- 2 kudos
- 2919 Views
- 2 replies
- 1 kudos
how to i able to read column mapping metadata for delta tables
i want to read column mapping metadatahttps://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mappingin above link we can able to find the code block with json data. the same data i want to read in pyspark.. is there any option to read that ...
- 2919 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,Information about the delta table such as history information could be found by running a `describe history table_name`. A `rename column` operation could be found in the `operation` column with a value of `RENAME COLUMN`. If you then look at the ...
- 1 kudos
- 5271 Views
- 2 replies
- 0 kudos
Resolved! How to fix (AWS SSO) Test Connection failed
What did I configure incorrectly? About SSO settings in my Databricks account. What troubleshooting should I do? I don't see any error message.I follow step instruction from AWS View step-by-step instructions. link belowView step-by-step instructions...
- 5271 Views
- 2 replies
- 0 kudos
- 0 kudos
I got solution I forgot to create SaaS users with the same subject as the AD users.Preprovisioned usersPreprovisioned users, means users must already exist in the downstream SaaS application. For instance, you may need to create SaaS users with the s...
- 0 kudos
- 3464 Views
- 0 replies
- 0 kudos
Delta Live Tables are dropped when pipeline is deleted
I created simplistic DLT pipeline that create one table. When I delete the pipeline the tables is dropped as well. That's not really desired behavior. Since I remember there was a strong distinction between data (stored in tables) and processing (spa...
- 3464 Views
- 0 replies
- 0 kudos
- 3218 Views
- 1 replies
- 1 kudos
UC Volume access for spark and other config files
Hi All,Wondering if anyone else getting this problem:We trying to host krb5.conf and jaas.conf for our compute to be able to connect to Kerberised JDBC sources, we attempting to store these files in Catalog volumes, but at run time when initiating th...
- 3218 Views
- 1 replies
- 1 kudos
- 1 kudos
Haven't been able to access volume path when using jdbc format.
- 1 kudos
- 1878 Views
- 1 replies
- 0 kudos
Deltalke performance
HiI am new to databricks and i am trying to understand the use case of deta lakehouse. Is it good idea to build datawarehouse using deltalake architecture. Is it going to give same performance as that of RDBMS clouse datawarehous like snowflake? Whic...
- 1878 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sas , One of the benefits of the Data Lakehouse architecture, is that it combines the best of both Data Warehouses and Data Lakes all on one unified platform to help you reduce costs and deliver on your data and AI initiatives faster. It brings t...
- 0 kudos
- 4687 Views
- 1 replies
- 0 kudos
Resolved! Installing a private pypi package from Gitlab on a cluster
I have published a pypi package in a private Gitlab repository and I want to install it in my notebook but I don't know how and the documentation doesn't help me much either. I have created a Gitlab token that I use in the index url and I try to inst...
- 4687 Views
- 1 replies
- 0 kudos
- 0 kudos
This problem was solved by removing the `python>=3.11` requirement.
- 0 kudos
- 6575 Views
- 2 replies
- 0 kudos
Resolved! Remove ZOrdering
Hi, I am trying to demonstrate the effectiveness of ZOrdering but to do this would like to remove the existing ZOrdering first. So my plan is:1. Remove existing ZOrdering2. Run a query and show the explain plan3. Add ZOrdering to column used for Joi...
- 6575 Views
- 2 replies
- 0 kudos
- 0 kudos
@DaveLeach - you can try dropping the table and create it again instead of #1.
- 0 kudos
- 4749 Views
- 3 replies
- 2 kudos
Resolved! Columns tab in Data Explorer doesn't reflect schema changes to table
Hey team, we've noticed that schema changes to a table after creation aren't reflected in the "Columns" tab in the Data Explorer. For example, we added a column called signal_description to a table but its addition isn't reflected in the UI. Is this ...
- 4749 Views
- 3 replies
- 2 kudos
- 2 kudos
@Tharun-Kumar Is this behavior the default behavior in case a schema change happens on a table registered in unity catalog?In that case I would have to run the repair command regularly in order to ensure that the schema displayed is actually the one ...
- 2 kudos
- 3740 Views
- 1 replies
- 0 kudos
User: anonymous is not authorized to perform: sqs:receivemessage on resource
from pyspark.sql import functions as F from pyspark.sql import types as T from pyspark.sql import DataFrame, Column from pyspark.sql.types import Row import dlt S3_PATH = 's3://datalake-lab/xxxx/' S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/' @dl...
- 3740 Views
- 1 replies
- 0 kudos
- 64790 Views
- 7 replies
- 0 kudos
How to check file exists in databricks
I Have a while loop there i have to check a file exists or not if exists read the file in a data frame else go to another file
- 64790 Views
- 7 replies
- 0 kudos
- 0 kudos
How to check if a file exists in DBFS?Let's write a Python function to check if the file exists or not-------------------------------------------------------------def file_exists(path): try: dbutils.fs.ls(path) return True except ...
- 0 kudos
- 15131 Views
- 1 replies
- 2 kudos
Universal Azure Credential Passthrough
At the moment, Azure Databricks has the feature to use AzureAD login for the workspace and create single user clusters with Azure Data Lake Storage credential passthrough. But this can only be used for Data Lake Storage.Is there already a way, or are...
- 15131 Views
- 1 replies
- 2 kudos
- 2 kudos
I have exactly the same issue. I have the need to call a protected API within a notebook but have no access to the current user's access token. I've had to resort to nasty workarounds involving installing and running the Azure CLI from within the not...
- 2 kudos
- 6254 Views
- 1 replies
- 0 kudos
pytesseract.pytesseract.TesseractNotFoundError in databricks notebook
I'm trying to extract the text data from image file in Databricks notebook I have installed below libraries using pip command: %pip install pytesseract tesseract pillow --upgradebut it didn't work and threw below error pytesseract.pytesseract.Tessera...
- 6254 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @neha_ayodhya - can you please try the following via an init script to the Databricks cluster sudo apt-get update -y sudo apt-get install -y tesseract-ocr sudo apt-get install -y libtesseract-dev /databricks/python/bin/pip install pytesseract a...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
