cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

11781 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

504 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

329 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

992 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

658 Posts

Activity in Databricks Platform Discussions

pdiamond
by > • Contributor
  • 78 Views
  • 1 replies
  • 0 kudos

Lakebase error logs

Anyone know where to see any logs related to Lakebase/Postgres? I have a Tableau Prep flow that is failing but the error is not clear and I'm trying to find out what the database is capturing.

Data Engineering
Reply
  • 78 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @pdiamond ,You can try to use Lakebase monitoring tools to capture query generated by Tableau Prep.Monitor | Databricks on AWSAlternatively, it seems that you can also use external monitoring tools. So you can connect to your lakebase instance usi...

  • 0 kudos
michealjones
by > • Visitor
  • 37 Views
  • 0 replies
  • 0 kudos

LinkedIn Fined for GDPR Violations – Key Takeaways

The Irish Data Protection Commission has fined LinkedIn for breaches of GDPR related to how user data was processed. This case highlights the ongoing importance of transparent data practices and strict compliance, even for large platforms. It’s a use...

Data Engineering
Reply
  • 37 Views
  • 0 replies
  • 0 kudos
Reply
Malthe
by > • Contributor III
  • 37 Views
  • 0 replies
  • 0 kudos

Intermittent connectivity issues between Power BI and Databricks

We're experiencing connectivity issues between Power BI (refreshing a dataset) and Databricks.On the Databricks side, we're using a SQL Warehouse (serverless), which reports queries timing out:Query has been timed out due to inactivity.On the Power B...

Warehousing & Analytics
Reply
  • 37 Views
  • 0 replies
  • 0 kudos
Reply
Leif
by > • New Contributor
  • 85 Views
  • 2 replies
  • 0 kudos

RLS

Hi,I have created a function that I have applied as a row filter function to multiple tables.The function takes one input parameter (a column value from the table). It then uses session_user() to look up a user in our users table. If the user is foun...

Warehousing & Analytics
Reply
  • 85 Views
  • 2 replies
  • 0 kudos
Reply
Latest Reply
Leif
New Contributor
  • 0 kudos

Hi,Thanks for the reply! Yeah, no the raw data has not changed at all. My thought is, can the row filter function handle reading from another table than the raw data table? I mean, I do a read in the users table to find a value, and want to compare t...

  • 0 kudos
1 More Replies
jmartin1
by > • New Contributor
  • 71 Views
  • 0 replies
  • 0 kudos

Claude Code-Execution Call Format

Is it possible to use the code execution tool with instances of Claude hosted through Databricks? If I try to format the payload like in Anthropic's documentation, I get an error that the function isn't defined properly:  tools=[{ "type": ...

Generative AI
Reply
  • 71 Views
  • 0 replies
  • 0 kudos
Reply
dbxsteve
by > • New Contributor II
  • 184 Views
  • 5 replies
  • 0 kudos

Proxy configuration - while bootstraping

I am trying to start a cluster in Az databricks,our policy is to use proxy for outbound traffic. I  have configured http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY, no_proxy and NO_PROXY List in env variables and global . Made sure proxy is bypassin...

Administration & Architecture
Reply
  • 184 Views
  • 5 replies
  • 0 kudos
Reply
Latest Reply
dbxsteve
New Contributor II
  • 0 kudos

@Debayan​ 

  • 0 kudos
4 More Replies
alxsbn
by > • Contributor
  • 104 Views
  • 1 replies
  • 0 kudos

Resolved! Metric View measure on joined table

On the Databricks documentation they take the TPCH classical tables to create some example https://docs.databricks.com/aws/en/metric-views/create/It's pretty easy to create a metric view using orders and do a measure like SUM(totalprice). Now let's s...

Warehousing & Analytics
Reply
  • 104 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @alxsbn ,  The Metric View joins are designed for "Many-to-One" relationships. Because orders and lineitem have a One-to-Many relationship (one order has multiple line items), you cannot join lineitem onto an orders-based Metric View and aggregate...

  • 0 kudos
Giuseppe_C
by > • New Contributor
  • 107 Views
  • 1 replies
  • 0 kudos

Databricks import directory false positive import

Hello evryone,I'm using databricks CLI to moving several directories from Azure Repos to Databricks Workspace.The problem is that files are not updating properly, with no error to display.The self-hosted agent in pipeline i'm using has installed the ...

Administration & Architecture
Reply
  • 107 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
Abeshek
New Contributor
  • 0 kudos

Hi @Giuseppe_C,Databricks CLI is not syncing updates during your pipeline runs. Several teams we work with have faced the same issue with legacy CLI versions and workspace import behavior. We’ve helped them stabilize CI/CD pipelines for Databricks, i...

  • 0 kudos
Jackryan360
by > • New Contributor
  • 97 Views
  • 2 replies
  • 0 kudos

How do Agentic AI services differ from traditional AI automation tools?

Hi everyone,I’m looking to understand the real difference between Agentic AI services and the traditional AI automation tools many businesses already use.In your experience, what makes Agentic AI services more advanced or effective?Are the advantages...

Generative AI
Agentic AI
AI
erpsoftware
Microsoft
Reply
  • 97 Views
  • 2 replies
  • 0 kudos
Reply
Latest Reply
Abeshek
New Contributor
  • 0 kudos

Hi @Jackryan360, I saw your question on Agentic AI vs traditional automation. Many teams are exploring the same distinction as they shift from rule-based workflows to autonomous, multi-step agent systems. At Kanerika, we’ve been helping companies eva...

  • 0 kudos
1 More Replies
dvd_lg_bricks
by > • New Contributor
  • 315 Views
  • 10 replies
  • 3 kudos

Questions About Workers and Executors Configuration in Databricks

Hi everyone, sorry, I’m new here. I’m considering migrating to Databricks, but I need to clarify a few things first.When I define and launch an application, I see that I can specify the number of workers, and then later configure the number of execut...

Data Engineering
Reply
  • 315 Views
  • 10 replies
  • 3 kudos
Reply
Latest Reply
Abeshek
New Contributor
  • 3 kudos

Your Databricks question about workers versus executors. Many teams encounter the same sizing and configuration issues when evaluating a migration. At Kanerika, we help companies plan cluster architecture, optimize Spark workloads, and avoid overspen...

  • 3 kudos
9 More Replies
michal1228
by > • New Contributor
  • 160 Views
  • 4 replies
  • 0 kudos

Import Python Modules with Git Folder Error

Dear Databricks Community, We encountered Bug in behaviour of import method explained in documentation https://learn.microsoft.com/en-us/azure/databricks/files/workspace-modules#autoreload-for-python-modules. Couple months ago we migrated our pipelin...

Data Engineering
Reply
  • 160 Views
  • 4 replies
  • 0 kudos
Reply
Latest Reply
michal1228
New Contributor
  • 0 kudos

We're using DBR version 16.4

  • 0 kudos
3 More Replies
Fatimah-Tariq
by > • New Contributor III
  • 204 Views
  • 7 replies
  • 4 kudos

Resolved! Writing to Foreign catalog

I have a running notebook job where I am doing some processing and writing the tables in a foreign catalog. It has been running successfully for about an year. The job is scheduled and runs on job cluster with DBR 16.2Recently, I had to add new noteb...

Data Engineering
Reply
  • 204 Views
  • 7 replies
  • 4 kudos
Reply
Latest Reply
Fatimah-Tariq
New Contributor III
  • 4 kudos

Thank you @Louis_Frolio! your suggestions really helped me understanding the scenario.

  • 4 kudos
6 More Replies
kumarV
by > • New Contributor
  • 102 Views
  • 1 replies
  • 0 kudos

Databricks Job : Unable to read Databricks job run parameter in scala code and sql query.

we are created data bricks job with Jar (scala code) and provided parameters/jar parameters and able to read those as arguments in main method.  we are running job with parameters (run parameters / job parameters) , those parametes are not able to re...

Administration & Architecture
Reply
  • 102 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Hey @kumarV , I did some digging and here are some hints/tips to help you further troubleshoot. Yep — this really comes down to how parameters flow through Lakeflow Jobs depending on the task type. JAR tasks are the odd duck: they don’t get the same ...

  • 0 kudos
skuvisk
by > • New Contributor
  • 108 Views
  • 2 replies
  • 1 kudos

CLS function with lookup fails on dates

Hello,I'm conducting research on utilizing CLS in a project. We are implementing a lookup table to determine what tags a user can see. The CLS function looks like this:CREATE OR REPLACE FUNCTION {catalog}.{schema}.mask_column(value VARIANT, tag STRIN...

Data Engineering
Reply
  • 108 Views
  • 2 replies
  • 1 kudos
Reply
Latest Reply
skuvisk
New Contributor
  • 1 kudos

Thank you for an insightful answer @Poorva21. I conclude from your reasoning that this is the result of an optimization/engine error. It seems like I will need to resort to a workaround for the date columns then...

  • 1 kudos
1 More Replies
Jarno
by > • New Contributor
  • 146 Views
  • 4 replies
  • 0 kudos

Dangerous implicit type conversions on 17.3 LTS.

Starting with DBR 17 running Spark 4.0, spark.sql.ansi.enabled is set to true by default. With the flag enabled, strings are implicitly converted to numbers in a very dangerous manner. ConsiderSELECT 123='123';SELECT 123='123X';The first one is succe...

Data Engineering
Reply
  • 146 Views
  • 4 replies
  • 0 kudos
Reply
Latest Reply
Jarno
New Contributor
  • 0 kudos

FYI, it seems I was mistaken about the behaviour of '::' on Spark 4.0.1. It does indeed work like CAST on both DBR 17.3 and Spark 4.0.1 and raises an exception on '123X'::int. The '?::' operator seems to be a Databricks only extension at the moment (...

  • 0 kudos
3 More Replies
Load more

Featured Posts

View all