Does enabling Change Data Feed on a Delta table affect OPTIMIZE and ZORDER performance?After enabling CDF on several large Delta tables, our OPTIMIZE jobs are taking noticeably longer. Is this expected, and are there any tuning parameters to minimize...
Hi everyone,I'm facing two specific issues in my Databricks Premium workspace (AWS - sa-east-1).Serverless Connectivity Issue: When using Serverless compute, I can successfully call APIs ending in .com, but calls to .com.br domains fail with connecti...
Hi there,Great breakdown of the symptoms — these are actually two distinct issues likely sharing a common root cause in your VPC/network configuration. Let me address both:Issue 1: Serverless Compute — .com.br DNS Resolution FailureRoot CauseServerle...
I am fairly new to Databricks and am learning it because a company I am working on is going to use it. One of the things they are going to use is Lakebase postgres so I thought I'd set it up on my personal account. First I don't see app switcher, sec...
If it is available, you would see at Compute->Lakebase and tabs for Provisioned and Autoscaling. This option Lakebase is next to Apps. There is another option dotted grid on the top right corner, the option just before your profile name, if you expan...
We connect Tableau and Power BI to our Databricks SQL warehouse via OAuth tokens. Does Unity Catalog row filters apply at the SQL layer regardless of the BI tool, or do we need additional enforcement at the warehouse level?
Unity Catalog row filters apply at the SQL/query layer, so if Tableau or Power BI is querying a Databricks SQL warehouse, the filters are enforced there — you do not need a separate warehouse-level row-filter feature. Row filters and column masks are...
We’re currently designing our Medallion Architecture pipelines using Lakeflow Jobs, and I wanted to get some opinions on orchestration best practices.Right now, our approach is essentially 1 job per target table (for example, each Bronze/Silver/Gold ...
We typically organize our workloads with one job per catalog, and then use one or more pipelines to load tables into the appropriate schemas. As our data engineers ingest raw data, this structure is primarily applied in the Silver and Gold layers of ...
Hi all,Lately I've been experimenting with the newish feature of scheduling jobs on a table update trigger. There's one thing thats blokcing me from implementing it however and I was hoping someone found a solution to it.We occasionally perform a vac...
Hi @Garybary, Quick clarification on how table update triggers actually behave, because this changes the answer significantly. Table update triggers fire on data-changing operations only (writes, merges, updates, deletes). A standalone VACUUM does NO...
Hello guys, i'm using databricks free edition (serverless) and i am trying to connect to a azure data lake storage.The problem I'm having is that in the free edition we can't configure the cluster so I tried to make the connection via notebook using ...
If you want to read from your Azure storage account using Databricks Free Edition, you can add a specific option when reading:spark.read.option("fs.azure.account.key.<storage-account-name>.dfs.core.windows.net", "your_storage_account...
Hello Community!I am writing to you with my idea about data ingestion job which we have to implement in our project.The data which we have are in CSV file format and depending on the case it differs a little bit. Before uploading we pivoting csv file...
Hello Community,We have a case in our project that we would like to solve in an elegant and scalable manner. As always, I would really appreciate your suggestions and experience.In short:We have a multi-step job consisting of 4 stages. In one of the ...
’m trying to build a UC usage/refresh tracking table for every workspace. For each workspace, I want to know how many times a UC table was refreshed or accessed each month. To do this, I’m reading the Databricks audit logs and I need to extract only ...
Hi @Danish11052000, You are on the right track with the COALESCE approach. The reason for the inconsistency is that different Unity Catalog action types populate different keys in request_params. Here is a breakdown of the key fields and which action...
Hey everyone,I recently worked on building a modern financial data lakehouse using Spark Declarative Pipeline OSS (SDP OSS), Apache Iceberg, and AWS Glue Catalog.The blog covers:- Building declarative data pipelines with Spark- Using Apache Iceberg a...
I’m currently developing a Databricks App, and the app’s service principal needs access to Unity Catalog tables. From what I can tell, it doesn’t seem possible to grant Unity Catalog permissions through DABs yet — only through the UI, based on the cu...
Hi @mnissen1337 ,But there is a way to do this in DABs. Look at following section in documentation:Manage Databricks apps using Declarative Automation Bundles | Databricks on AWSIf my answer was helpful, please consider marking it as accepted solutio...
I've just created an Azure Databricks workspace, tier (Premium) and when trying to create a cluster, when I click on compute, the UI opens automatically the menu SQL Warehouse, not sure if it's a glitch as shown below. Someone said "Ask the admin to ...
Hi @sminamioka This does not look like a UI glitch. In newer Azure Databricks workspaces, access to classic compute / clusters depends on workspace entitlements and compute policy permissions.If clicking Compute takes you directly to SQL Warehouses, ...
Hi,I have the following problem: on a medallion architecture on a bronze volume I get files every month containing the data for each sensor reading during the period 1 of month 00:00 to last day 23:00. I have a manual job that calls the python files ...
Hi @Guillermo-HR Yes — batch is usually the right fix here. What’s happening is that your query is using event-time window aggregation in Structured Streaming with append output mode. In that mode, Spark only emits a window after it is sure the wind...
We are testing Delta writer in our environment to create bronze tables and recently, I just needed to add one table to the notebook code and rerun the whole notebook that failed because of this error : [DELTA_CREATE_EXTERNAL_TABLE_WITHOUT_TXN_LOG] Y...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
