Databricks Community

Hi @Phani1, Implementing Databricks cell-level code parallel execution through the Python threading library can be beneficial for performance, but there are some considerations to keep in mind. Let’s break it down: Resource Consumption and Alloca...

Hi @Fresher, It sounds like you’re experiencing an issue with user synchronization between Azure AD and Databricks. Let’s troubleshoot this together! Here are some steps you can take to resolve the issue: Check SCIM Provisioning Configuration: En...

Hi @chloeh, You’re working with a Spark SQL data transformation pipeline involving chained window aggregations. Let’s look at your code snippet and see if we can identify the issue. First, let’s break down the steps you’ve implemented: You’re read...

Honestly you shouldn't have any kind of limitation executing diferent workflows.I did a test case in my Databricks and if you have your workflows with a job cluster your shouldn't have limitation. But I did all my test in Azure and just for you to kn...

Hi @namankhamesara, Thank you for reaching out! It appears there might be an issue with accessing the data for your course. To expedite your request and resolve this issue promptly, please list your concerns on our ticketing portal. Our support staff...

Hi @Shazam, Great questions! Let’s break down each scenario: Initial Data Migration: When migrating data from an existing platform to Databricks, you might have a large initial load of records. In this case, ingestion time clustering can still be...

Hi @halox6000, To stop the progress bar output from tqdm, you can use the disable argument. Set it to True to silence any tqdm output. In fact, it will not only hide the display but also skip the progress bar calculations entirely1. Here’s an examp...

Hey MrD,I faced this issue while running Azure VMs. A restart and re atatching the cluster helped me. Please let me know if that works for you.

As advised, I double confirmed that no code or cluster configuration was changed (even got a second set of eyes on it that confirmed the same).I was able to find a "fix" which puts a bandaid on the issue:I was able to pinpoint that the issue seems to...

I am not super expert but I have been using databricks for a while and I can say that - when you use any Python library like asyncio, ThredPool and so one - this is good only to some maintenance things, small api calls etc.When you want to leverage s...

Hi @Menegat, It seems you’re encountering an issue with your Delta tables during incremental updates. Let’s dive into this and explore potential solutions. Delta Live Tables and Incremental Updates: Delta Live Tables allow for incremental updates...

Hi @georgef, It appears that you’re encountering issues with importing modules within a Databricks Asset Bundles (DABs) project. Let’s explore some potential solutions to address this problem. Bundle Deployment and Import Paths: When deploying a ...

Hi @lindsey, It seems you’re encountering an issue with Terraform and Databricks when trying to destroy resources. Let’s explore some potential solutions to address this problem: Resource Order in Terraform Configuration: Ensure that the databric...