Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
16766 Posts
Activity in Discussions
- 3 Views
- 0 replies
- 0 kudos
Databricks Data Ingestion
Hi all,I am currently exploring the data ingestion feature offered by databricks, specifically connecting to a SQL server. I have gone through the UI and configured an ingestion pipeline that connects to one specific server and can read tables into d...
- 3 Views
- 0 replies
- 0 kudos
- 15 Views
- 2 replies
- 0 kudos
Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...
- 15 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@gowri_databrick Mlflow version is 3.8.1 and I want to control how and where my experiments are stored!! Do you know how to do that?
- 0 kudos
- 42 Views
- 3 replies
- 1 kudos
Resolved! How can i rename a column in a delta table?
I have a delta table and i want to rename one of its columns.what is the recommended way to rename a column in databricks?is there any difference between renaming a column using sql and using pyspark?
- 42 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
@gowri_databrick Common Approaches are:SQL migration through PySpark: This is only a Python wrapper around the same SQL DDL. It is not a different renaming mechanism. It needs the columnMapping.mode = 'name' to be enabled on the table first.spark.sql...
- 1 kudos
- 90 Views
- 3 replies
- 1 kudos
Partitioning vs Liquid Clustering (per-table):
Can PARTITION BY and CLUSTER BY (Liquid Clustering) be used simultaneously on the same table? If we use only PARTITION BY, is there a negative performance impact on materialized-view refreshes in Silver/Gold? Since materialized views read only increm...
- 90 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hey @AshokB, good question. To add on to what @balajij8 and @Satyasai already said:On combining PARTITION BY and CLUSTER BY: you can't do it, as they are mutually exclusive. Delta will throw an error if you try to define both on the same table. Have...
- 1 kudos
- 45 Views
- 1 replies
- 0 kudos
Lakebase Postgre updating Delta Table.
I am using Postgre for OLTP processing for POS application.Lag is reduced a lot, however when there is updation on Postgre table, I need to sync back to delta table. There is one way from delta table sync table (read only). Any design pattern sas to ...
- 45 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@rkhbo3003 You can use Lakebase Change Data Feed (CDF) for it. It is the pattern for this Postgres to Delta ingestion path. It continuously streams every INSERT, UPDATE and DELETE from the Lakebase Postgres write-ahead log (WAL) directly into Unity C...
- 0 kudos
- 51 Views
- 0 replies
- 0 kudos
From Semantic Similarity to Business Authority: Why Genie Ontology and OntoRank Matter
Enterprise AI does not usually fail because the model lacks intelligence.It fails because the model does not understand what the organization means.Consider a simple question:“What is our current exposure to active customer?”Behind this question are ...
- 51 Views
- 0 replies
- 0 kudos
- 133 Views
- 2 replies
- 0 kudos
Unable to change password on Webassesor
The form on Databricks Webassesor doesn't permit me to change my password, meaning I am unable to log in. Is anyone else having this issue?
- 133 Views
- 2 replies
- 0 kudos
- 160 Views
- 4 replies
- 0 kudos
Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a
Environment:Databricks Runtime: Current channel, Photon enabledEdition: ProPlatform: AzureDescription:My setup consists of:A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting ...
- 160 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
The error confirms that the silver Delta stream encountered a schema transition it considers read-incompatible. It does not identify the original SQL Server DDL, so I would verify the sequence before assigning a root cause. Lakeflow Connect normally ...
- 0 kudos
- 60 Views
- 0 replies
- 0 kudos
Webassesor last name change
Hi, anyone knows how to contact databricks support? Or how long it takes to get any response from them?I need to change my last name in webassesor, can't do this my self. Already had to move my exam 4 times becasue support is not responsive. I've tri...
- 60 Views
- 0 replies
- 0 kudos
- 49 Views
- 0 replies
- 0 kudos
Como aprender correctamente en el entorno de DataBricks y aprovecharlo al 100%
Hola a todos, soy ingeneriero en sistemas y deseo aprender a utilizar esta grandiosa herramienta para poder sacarle el mejor provecho posible.
- 49 Views
- 0 replies
- 0 kudos
- 56 Views
- 0 replies
- 0 kudos
Who pays cross-cloud egress for Delta Sharing from GCP to Azure?
Hi DBX community!! Are cross-cloud egress charges for a Databricks-to-Databricks #DeltaShare billed to the provider account, and do they apply when the shared objects are views backed by Cloudflare?I'm setting up a Delta Share using Databricks on GCP...
- 56 Views
- 0 replies
- 0 kudos
- 131 Views
- 3 replies
- 0 kudos
Certification Not Issued
Hi Everyone,I know this may not be the right place to raise a support-related issue, but I’m reaching out to see if anyone from the community can offer guidance or additional assistance.I passed my Databricks Data Engineer Associate certification sev...
- 131 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Psangi, Thank you for filing a ticket with our support team, support team will respond shortly. It can take up to 48 hours for badges/ certificates to be issued. Also please check your spam just in case it went there. Please note that we canno...
- 0 kudos
- 116 Views
- 4 replies
- 2 kudos
How to calculate cost of each table for the specific Databricks Run ID
Hi Databricks Community,I need help calculating table-level cost for each specific Job Run ID in Databricks.I have multiple pipelines/jobs, and the same table can run multiple times with different Run IDs.For example:Job A│├── Run ID 1001│ ├── Table ...
- 116 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
To calculate table-level cost for each specific Job Run ID in Databricks, you must join system billing logs with operational metadata.Databricks billing metrics (system.billing.usage) track cost down to the Task Run ID level, but Databricks does not ...
- 2 kudos
- 95 Views
- 3 replies
- 0 kudos
Private listing of App Asset in Databricks Marketplace by 3rd Party Provider
Hello, Could you help confirm whether a third-party provider is able to publish an App asset type as a private listing within a Private Exchange?If this is supported, is it a self-service capability, or does it need to be enabled per account? And if ...
- 95 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
@AndreiaC Databricks Apps are currently not supported as an asset type in Databricks Marketplace. While Marketplace supports data assets, tables, views, volumes, notebooks, machine learning models and MCP servers (Preview), Apps operate under a diffe...
- 0 kudos
- 127 Views
- 4 replies
- 0 kudos
System generated Service Principal -App
I created an App and creating agent to generate ETL pipeline through prompts. User interaction is with app, however when I do schema check on ADLS it says user does not have access. I cannot see SP in UI as it is system generated, even though I have ...
- 127 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Step-by-Step SolutionStep 1: Identify the System Service PrincipalBecause system-generated SPs are hidden from the standard Workspace Users UI, you must query it using the Databricks CLI or REST API.Run this command in the Databricks CLI:Bash Command...
- 0 kudos
| User | Count |
|---|---|
| 1900 | |
| 1045 | |
| 931 | |
| 480 | |
| 377 |
