Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
1311 Posts
Activity in Databricks Platform Discussions
- 28 Views
- 2 replies
- 1 kudos
Databricks Python stored procedures
Hi,I am using databricks runtime 17.3.x-scala2.13 , But I am unable to create python stored procedures, (functions are possible but they dont support a spark session like below) , any thoughts/help is much appreciated ? [INVALID_STATEMENT_OR_CLAUSE]...
- 28 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @RGSLCA ,This video is a bit misleading - look at the comments section. This feature was not released and as of now you can only create stored procedure using an SQL language.
- 1 kudos
- 106 Views
- 2 replies
- 0 kudos
Enable CDC in Lakeflow Connect Tables
Hello everyone, I'm implementing a project that requires AutoCDC workflows using Lakeflow.The architecture is as follows: Data is ingested from a database using Lakeflow Connect, and then a declarative pipeline performs some transformations on this d...
- 106 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
The short answer is no, you don't need to enable it, because the tables generated by Lakeflow Connect already capture and stream CDC metadata natively. In fact, streaming tables in Delta Live Tables (DLT) and Lakeflow pipelines are built on top of an...
- 0 kudos
- 45 Views
- 1 replies
- 0 kudos
Account Prices for Azure
For AWS and GCP, the discounts provided by Databricks can be found in system.billing.account_prices table. How can we find the same in Azure?
- 45 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @nara_shikamaru Azure Databricks is a native service similar to Synapse in Azure unlike others. Databricks generally never sees the final aligned price and hence its unavailable in system tables.You can follow belowAzure Daily Export - You can exp...
- 0 kudos
- 1253 Views
- 2 replies
- 3 kudos
Databricks Genie Code One Month In — What It Means for Us as Data Engineers
As a data engineer, I've been watching Genie Code closely since it went GA last month, and I wanted to share some honest thoughts here today not a feature dump, but how it's starting to change the way I think about my own role.Databricks recently sha...
- 1253 Views
- 2 replies
- 3 kudos
Latest Reply
- 3 kudos
Very well said. Thanks for sharing your honest experiences!
- 3 kudos
- 71 Views
- 2 replies
- 0 kudos
Error in Genie Mobile
Hello,We are using Azure Databricks Premium and we are trying to access Databricks Genie through the mobile app.When logging in, we receive the following error:OAuth application with client_id: 'databricks-mobile' not available in Databricks account ...
- 71 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
The error specifically says the mobile OAuth app databricks-mobile isn’t available for that Databricks account, which usually means the account isn’t provisioned for mobile access, rather than an issue with your login itself.I think you need to enabl...
- 0 kudos
- 163 Views
- 2 replies
- 0 kudos
DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.
Hi In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.I created an instance pool with Instance type: Standard_DS3_v2 ...
- 163 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
I tried to reproduce and it worked as expected for me. Check your pipeline JSON to make sure your "clusters" spec there is explicitly mentioning the pool details. Example: "pipeline_type": "WORKSPACE", "name": "mojama-dlt-classic-demo", "cluste...
- 0 kudos
- 60 Views
- 0 replies
- 0 kudos
network security perimeter post-setup questions
I've set up the network security perimeter with Terraform using Configure an Azure network security perimeter for Azure resources - Azure Databricks | Microsoft Learn as my guide. All of the resources associated to the NSP, and the Status of each is...
- 60 Views
- 0 replies
- 0 kudos
- 94 Views
- 2 replies
- 0 kudos
PostgreSQL ingestion source not supported in workspace when deploying Databricks Asset Bundle
I'm trying to deploy a Databricks Asset Bundle that creates a DLT/Lakeflow pipeline with a PostgreSQL ingestion source. The bundle builds successfully, uploads artifacts, and starts deploying resources, but the pipeline creation fails with the follow...
- 94 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @muaaz ,Yep, that feature is in public preview. Usually, when something is in public preview you have access to that feature, but in this case they require some steps to enroll it for a given workspace. So, as @balajij8 and docs suggests -> conta...
- 0 kudos
- 223 Views
- 6 replies
- 1 kudos
Resolved! Automate Lakeflow connect to ingest 300 tables not manually
I have data in PostgreSQL and I’m using Lakeflow Connect via UI to ingest it into Databricks streaming tables.Currently, each Lakeflow Connect pipeline only allows connecting one PostgreSQL table. I have around 300 tables, and creating pipelines manu...
- 223 Views
- 6 replies
- 1 kudos
- 61 Views
- 0 replies
- 0 kudos
Apache Spark Masterclass (In-Person, Bengaluru) | 6 June
Hi everyone,We're hosting the next session of our Data Engineering Masterclass Series focused on Apache Spark.This is an in-person, hands-on session for engineers interested in modern data engineering, distributed data processing, and real-world Spar...
- 61 Views
- 0 replies
- 0 kudos
- 1081 Views
- 3 replies
- 2 kudos
Resolved! Not able to find lab for Data Engineering Learning Path
I am not able to find the data engineering learning path , i did open partner databricks academy lab which redirected to uplimit where i also enrolled myself to instructor led course but not able to see any labs.
- 1081 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
You are right the self paced e-learning courses does not include any labs. However, they are available on instructor led courses available on Uplimit. I recently enrolled for one and here is how it worked for me:1. On Uplimit portal enroll for an upc...
- 2 kudos
- 244 Views
- 3 replies
- 0 kudos
Resolved! Databricks SQL connection becomes stale in long-running app
I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...
- 244 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
SQLAlchemy dialect is a wrapper for the native databricks sql connector. You can try to pass the various authentication configuration supported by the underlying SQL connector directly into the connect_args dictionary parameter of the alchemy engine....
- 0 kudos
- 3369 Views
- 8 replies
- 1 kudos
Resolved! Delta Live Tables - skipChangeCommits in SQL
Hi,Could anyone tell me if the skipChangeCommits option is supported in SQL mode? I can use it successfully using Python, but it doesn't look like it is supported by SQL.It seems to be a glaring omission from the SQL support, or support for this will...
- 3369 Views
- 8 replies
- 1 kudos
Latest Reply
- 1 kudos
This is now supported:CREATE OR REFRESH STREAMING TABLE basic_stAS SELECT * FROM STREAM samples.nyctaxi.trips WITH (SKIPCHANGECOMMITS);Supported in runtime 17.3 and later.Documentation: https://docs.databricks.com/aws/en/ldp/developer/sql-dev#create-...
- 1 kudos
- 82 Views
- 0 replies
- 0 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Post removed — reposting with corrections
- 82 Views
- 0 replies
- 0 kudos
- 204 Views
- 3 replies
- 0 kudos
Liquid Clustering VS Z-ordering
I want to understand difference b/w Liquid Clustering VS Z-ordering and also how both works?
- 204 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Rupa0503, In simple terms... both Liquid Clustering and Z-ordering are ways to improve data layout so Databricks can skip more irrelevant files during reads, but they are not the same thing. If I had to summarise it simply... Z-ordering is the o...
- 0 kudos
| User | Count |
|---|---|
| 1837 | |
| 885 | |
| 783 | |
| 471 | |
| 312 |
