- 9875 Views
- 6 replies
- 19 kudos
Practice Exams for Databricks Certified Data Engineer Professional exam
Can anyone help with official Practice Exams set for Databricks Certified Data Engineer Professional exam, like we have below for Databricks Certified Data Engineer AssociatePractice exam for the Databricks Certified Data Engineer Associate exam
- 9875 Views
- 6 replies
- 19 kudos
- 5057 Views
- 6 replies
- 1 kudos
How to trigger an Azure Data Factory pipeline through API using parameters
Hello All,I have a use case where I want to trigger an Azure Data Factory pipeline through API. Right now I am calling the API in Databricks and using Service Principal(token based) to connect to ADF from Databricks.The ADF pipeline has some paramete...
- 5057 Views
- 6 replies
- 1 kudos
- 1 kudos
Hello @Andolina1,try to send your payload like:body = {'curr_working_user': f'{parameters}'}response = requests.post(url, headers=headers, json=body)the pipeline's parameter should be named curr_working_user. With these changes your setup should work...
- 1 kudos
- 736 Views
- 3 replies
- 0 kudos
Using Databricks Sql Statement Execution api
TeamWe have internal chatbot service that will send query to data bricks SQL execution API.Number of queries vary from 50 per minute to 100 per minutes. and we are trying to limit response size by applying limit 10. Basically trying hard to use all o...
- 736 Views
- 3 replies
- 0 kudos
- 0 kudos
As a suggestion, you can also think of creating your own API to query directly your tables via JDBC/ODBC connections over a SQL Warehouse. This case, limitations would be only those associated to SQL Warehouses and your API but not the Databricks API...
- 0 kudos
- 574 Views
- 2 replies
- 2 kudos
Resolved! I/F security about using medallion architecture
I’m new to writing requirement definitions, and I’d like to ask a question about interface (I/F) security.My question is:Do I need to define the authentication and security mechanisms (such as OAuth2, Managed Identity, Service Principals, etc.) betwe...
- 574 Views
- 2 replies
- 2 kudos
- 2 kudos
I'll try to summarize and go directly to the key points as I see this:- Client to S3 SAS Token or OAUTH 2.0 with Service to Service authentication (preferred)- Databricks to S3 Use Service Principal or Managed Identities (preferred)- Bronze/Silver/...
- 2 kudos
- 961 Views
- 2 replies
- 2 kudos
Resolved! Can’t save results to target table – out-of-memory error
Hi team, I’m processing ~5,000 EMR notes with a Databricks notebook. The job reads from `crc_lakehouse.bronze.emr_notes`, runs SciSpaCy UMLS entity extraction plus a fine-tuned BERT sentiment model per partition, and builds a DataFrame (`df_entities`...
- 961 Views
- 2 replies
- 2 kudos
- 2 kudos
You’re right that the behaviour is weird at first glance (“5k rows on a 64 GB cluster and I blow up on write”), but your stack trace is actually very revealing: this isn’t a classic Delta write / shuffle OOM – it’s SciSpaCy/UMLS falling over when loa...
- 2 kudos
- 4886 Views
- 3 replies
- 2 kudos
DLT x UC x Auto Loader
Now that the Directory Listing Mode of Auto Loader is officially deprecated, is there a solution for using File Notification Mode in a DLT pipeline writing to a UC-managed table? My understanding is that File Notification Mode is only available on si...
- 4886 Views
- 3 replies
- 2 kudos
- 2 kudos
Databricks introduced Managed File Events which completely bypasses the need for the cluster's identity to provision cloud resources, resolving the conflict with the Shared cluster mode.Steps to Implement in DLTEnable File Events on the External Loca...
- 2 kudos
- 3781 Views
- 3 replies
- 0 kudos
Error during build process for serving model caused by detectron2
Hi All,Introduction: I am trying to register my model on Databricks so that I can serve it as an endpoint. The packages that I need are "torch", "mlflow", "torchvision", "numpy" and "git+https://github.com/facebookresearch/detectron2.git". For this, ...
- 3781 Views
- 3 replies
- 0 kudos
- 0 kudos
Found an answer!Basically pip was somehow installed the dependencies from the git repo first and was not following the given order so in order to solve this, I added the libraries for conda to install.``` conda_env = { "channels": [ "defa...
- 0 kudos
- 834 Views
- 5 replies
- 0 kudos
Lakebridge analyzer not able to determine DDL.
Databricks analyzer does not shows any DDL statement count, I've also tested with just a simple SELECT * query (SELECT * FROM SCHEMA_NAME.TABLE_NAME;) . Is there any solution for this ?My target was to get a detailed analysis on SnowSQL code. Any h...
- 834 Views
- 5 replies
- 0 kudos
- 0 kudos
The Lakebridge analyzer counts DDL statements, not regular queries. A simple SELECT * is DML, not DDL, so it won’t show up in the DDL count.To get meaningful results for SnowSQL code analysis:Include actual DDL statements like CREATE TABLE, ALTER TAB...
- 0 kudos
- 5094 Views
- 1 replies
- 1 kudos
Schema Registry certificate auth with Unity Catalog volumes.
Greetings.We currently have a Spark structured streaming job (Scala) retrieving avro data from an Azure Eventhub with a confluent schema registry endpoint (using an Azure Api Management gateway with certificate authentication).Until now the .jks file...
- 5094 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the detailed context—here’s a concise, actionable troubleshooting plan tailored to Databricks with Unity Catalog volumes and Avro + Confluent Schema Registry over APIM with mTLS. What’s likely going wrong Based on your description, the ini...
- 1 kudos
- 6070 Views
- 2 replies
- 0 kudos
Adding a message to azure service bus
I am trying to send a message to a service bus in azure. But I get following error:ServiceBusError: Handler failed: DefaultAzureCredential failed to retrieve a token from the included credentials.This is the line that fails: credential = DefaultAzure...
- 6070 Views
- 2 replies
- 0 kudos
- 0 kudos
It looks like the issue is with the Azure credential chain rather than Service Bus itself; in Databricks notebooks, DefaultAzureCredential won’t succeed unless there’s a valid identity available (env vars, CLI login, managed identity, or a Databricks...
- 0 kudos
- 6742 Views
- 2 replies
- 0 kudos
How Install Pyrfc into AWS Databrick using Volumes
I'm trying to install Pyrfc in a Databricks Cluster (already tried in r5.xlarge, m5.xlarge, and c6gd.xlarge). I'm following these link.https://community.databricks.com/t5/data-engineering/how-can-i-cluster-install-a-c-python-library-pyrfc/td-p/8118Bu...
- 6742 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for the details. The PyRFC package is a Python binding around the SAP NetWeaver RFC SDK and requires the SAP NW RFC SDK to be present at build/run time; it does not work as a pure Python wheel on Linux without the SDK. The project is archived ...
- 0 kudos
- 3185 Views
- 2 replies
- 1 kudos
how to add Microsoft Entra ID managed service principal to aws databricks
Hi,I would like to add a Microsoft Entra ID managed service principal to AWS Databricks, but I have noticed that this option does not appear to be available-I am only able to create managed service principals directly within Databricks.For comparison...
- 3185 Views
- 2 replies
- 1 kudos
- 1 kudos
You cannot add a Microsoft Entra ID–managed service principal to Databricks on AWS today; AWS workspaces only support Databricks‑managed service principals that you create in the Databricks account/workspace, not service principals federated from Ent...
- 1 kudos
- 4227 Views
- 3 replies
- 0 kudos
What is the best way to connect Oracle CRM cloud from databricks?
What is the best way to connect Oracle CRM cloud from databricks?
- 4227 Views
- 3 replies
- 0 kudos
- 0 kudos
Oracle CRM on Demand is a Cloud platform not a relational database.Is there any proven solution for this requirement?
- 0 kudos
- 10369 Views
- 5 replies
- 6 kudos
Resolved! Limit on number of result rows displayed on databricks SQL UI
Databricks SQL UI currently limits the query results display to 64000 rows. When will this limit go away? Using SSMS I get 40MM rows results in the UI and my users won't switch to databricks SQL for this reason
- 10369 Views
- 5 replies
- 6 kudos
- 6 kudos
want to understand the reason behind it. retrieving 500k records is not a problem, is it rendering over the http thats the problematic?
- 6 kudos
- 891 Views
- 4 replies
- 5 kudos
Lakeflow jobs
Hi I am currently working on migrating all ADF jobs to LakeFlow jobs. I have a few questions:Pipeline cost: What is the cost model for running LakeFlow pipelines? Any documentation available? ADF vs Lakeflow Job?Job reuse: Do LakeFlow jobs reuse the...
- 891 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Nidhig ,1. Regarding pipeline cost - here you're mostly paying for compute usage. So the exact price depends on which plan you are at and which cloud provider you are using. For instance, for Azure premium plan and US East region you have followi...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
