Hello Databricks Community,We are using a Lakeflow Connect managed ingestion pipeline with the query-based connector to ingest Oracle tables via Unity Catalog foreign catalogs (Lakehouse Federation) on Azure Databricks.One of our requirements is to c...
Hi all,We want to ingest from an Oracle Autonomous Database on Exadata Cloud@Customer with the Lakeflow Connect query-based Oracle connector (classic compute). The database enforces mutual TLS: the listener requires a client certificate from the Orac...
Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...
Hi everyone,I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.Here are the symptoms:The Setup: Storage credentials and an external location were successfully created, and...
Hi @Khasim_1 Are you doing direct file access or via unity catalog tables/volumes? if you are reading directly then you need to grant READ FILES to the identity used by your pipeline/notebook to this external location.If you are accessing via uc tabl...
The following error occured when running .count() on a big sparkDF. Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failur...
Hi @sinclair Seems you have so many upstream transformations before doing this count operation on running dataframe??for me your nested json is suspicious , you might be trying to retrieve something nullable there causing this issue, can you check ru...
May someone please help me designing the Migration of SAP S/4 HANA to Databricks. How to design this. what all we need to consider as LLD.1. How Data needs to be extracted and by which tool ? near–real-time replication is required2. Each layer for Da...
@Pratikmsbsvm I will try to be brief but accurate.1. Getting data out of SAPYou need a properly supported connector/extraction method. You can start researching Fivetran, SNP Glue and alike. What you need to understand is that this can be expensive, ...
Hi Databricks Team,I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.EnvironmentDatabricks Edition: Free EditionCloud: AWSWorkspace region: us-east-2Metastore region: us-east-2S3 bucket re...
Hi @Visheshd_21 Based on the validation results, the failure is occurring at the initial AssumeRole step. Since Self Assume Role and External ID checks are skipped, Databricks is not successfully assuming the IAM role, so I would troubleshoot the AWS...
Azure Managed Identity Credential with Access Connector Id /subscriptions/c03723bd-79a8-401a-8392-a6aef2b0d7b9/resourceGroups/RG_databricks_masterclass/providers/Microsoft.Databricks/accessConnectors/ac_meta_new and Managed Identity ID: for Account I...
Resourceid: /subscriptions/c03723bd-79a8-401a-8392-a6aef2b0d7b9/resourceGroups/RG_databricks_masterclass/providers/Microsoft.Databricks/accessConnectors/ac_meta_new Databricks Accountid:https://accounts.azuredatabricks.net/?account_id=79762b8f-7...
We'd like to store MLflow 3 traces in Unity Catalog, using an experiment linked to a UC table-prefix trace location. In our Azure UK South workspace, linking the location fails with:PERMISSION_DENIED: Zerobus failed to open the table. Make sure the t...
Private endpoint storage for traces remains a listed limitation. Store Open Telemetry traces in Unity Catalog lists this as a limitation among other constraints, Traces cannot yet be written to storage protected by Private Link. It is stated as a kno...
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
Hi @capoorhimanshu,Your worry about the closure is fair. A secret that rides inside the serialized task is harder to control than one that never leaves a secret store.As far as I can tell, there's no supported setting on standard access mode that pus...
Workspace ID 7474651448614968 (Free Edition, AWS). Credential databricks-dataengineer-v2 (IAM role arn:aws:iam::278137443935:role/DATABRICKS-DATAENGINEER) fails validation at the Assume Role step; CREATE EXTERNAL LOCATION returns PERMISSION_DENIED .....
Hi @Arisk, I agree with @DoTA that your AWS setup looks right. The master role ARN you're using matches the one in the AWS setup docs (https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual), and I coul...
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
Hi @Islam_hoti, good instinct not to just slap skipChangeCommits on and walk away. Both of your ideas work. Which one I'd pick depends on how often corrections happen.Option A: rare, batched corrections (what you have). The Delta streaming docs descr...
Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE <table_name> SET MANAGEDDatabricks document...
Hi @Davide, this thread never got a reply, but you asked the same thing in SET MANAGED and rollback in relation to VACUUM and posted your test result there, so I'm linking it for anyone who lands here first.Short version: the 14-day rollback window i...
Hi there,Usercase: Fraud Detection Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reportsAsk: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology s...
Hi @abhietc31 I assume this is what you are looking for :https://docs.databricks.com/aws/en/structured-streaming/real-time/conceptsPossible in both pyspark structured streaming pipelines or lakeflow pipelines. Depends how you are curating your pipeli...
Hi,I recently started trying out the Genie App Builder first presented at DAIS. For this I created an app space and I wanted to connect my app to a lakebase project. But I can't seem to find any options to do this like it works with apps which are no...
Hi @JanW Based on the current documentation, this looks like a limitation of the new Genie App Builder + App Spaces model.Genie App Builder apps are created inside an App Space, and App Spaces currently define governance mainly through user authoriz...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
