The Databricks MVP Program is our way of thanking and recognizing the community members, data scientists, data engineers, developers and open source enthusiasts who go above and beyond to uplift the data and AI community. Whether they’re speaking at ...
- 150 Views
- 0 replies
- 4 kudos
20 hours ago
The Data + AI Skills Gap The “skills gap” has been a concern for CEOs and leaders for many years, and the gap is only widening. According to McKinsey, as many as 375 million workers globally might have to change occupations soon to meet company needs...
- 219 Views
- 0 replies
- 1 kudos
Monday
Apply Now! Are you passionate about data and want to make a difference for thousands of data practitioners? Databricks is looking for a dedicated and knowledgeable Community Technical Moderator to guide our thriving online community and empower users...
- 727 Views
- 1 replies
- 3 kudos
2 weeks ago
How to unlock enterprise AI: “You can have all the AI in the world, but if it’s on a shaky data foundation, then it’s not going to bring you any value.” — Carol Clements, Chief Digital and Technology Officer, JetBlue Companies everywhere have been qu...
- 214 Views
- 1 replies
- 1 kudos
Thursday
Are you solving real-world problems with data and AI? Your peers want to be inspired by your work! At Data + AI Summit 2025, we’re looking for data engineers, ML engineers, data scientists and analysts who are pushing the limits of AI, analytics and ...
- 327 Views
- 0 replies
- 2 kudos
a week ago
Databricks Performance Index is derived statistically from repeating workloads, accounting for changes irrelevant to the engine, and computed against billions of production queries. Lower is better. As the season of giving approaches, we at Databri...
- 379 Views
- 0 replies
- 2 kudos
a week ago
Community Activity
- 0 Views
- 0 replies
- 0 kudos
Bookmark in pdf
I am creating a pdf using pyspark and trying to make bookmarks for each table in the pages. All the bookmarks end up pointing to the first table in the first page. Please help me out here.
- 0 Views
- 0 replies
- 0 kudos
- 11 Views
- 1 replies
- 0 kudos
Cloning of Workflow from One env to different env using Job API
Hi Team,One of my team members recently shared one requirement: he wants to migrate the 10 Workflows from the sandbox to the dev environment to run his model in dev env.I wanted to move all these workflows in an automated way and one of the solutions...
- 11 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi AmetDatabricks Asset Bundles are designed precisely for the requirements you have.https://docs.databricks.com/en/dev-tools/bundles/index.htmlYou can also transfer existing jobs that were created manually into a bundle. This works like this, for ex...
- 0 kudos
- 846 Views
- 2 replies
- 7 kudos
Unity Catalog Functions: Tools for Users and Agents
Everyone wants to use Generative AI to solve business problems, but developers have realized that simply deploying a large language model isn’t going to be enough to cross the finish line. Business problems are complex and require a solution capable ...
- 846 Views
- 2 replies
- 7 kudos
- 33294 Views
- 31 replies
- 14 kudos
Not received my certificate after passing Data Engineer Associate exam.
I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate.-----------------------------------------------Hello Rupesh,You have completed the assessment, Databricks Certified Data Engineer A...
- 33294 Views
- 31 replies
- 14 kudos
Latest Reply
- 14 kudos
I've successfully passed Databricks Data Engineer Associate Certified exam on 20th Nov 2024 9am IST but still have not received the certificate.whom do I need to connect to receive the certificate ?
- 14 kudos
- 4607 Views
- 6 replies
- 1 kudos
Resolved! [Databricks Assets Bundles] no deployment state
Good morning, I'm trying to run: databricks bundle run --debug -t dev integration_tests_job My bundle looks: bundle: name: x include: - ./resources/*.yml targets: dev: mode: development default: true workspace: host: x r...
- 4607 Views
- 6 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello, Reopening this ticket in hopes that either of you had some luck in resolving your bug. I am currently facing the same issue where I can deploy an asset bundle via the local CLI without issue (by deploy I mean the bundle code is written to my ...
- 1 kudos
- 20 Views
- 0 replies
- 0 kudos
PAT needed but not allowed in "Advanced Data Engineering - 6.5L Deploy pipeline with the CLI Lab"
It is stated in the lab notebook that:Run the setupRun the setup script for this lesson by running the cell below. This will ensure that:The Databricks CLI is installedAuthentication is configuredA pipeline is createdHowever, when I tried to run the ...
- 20 Views
- 0 replies
- 0 kudos
- 1841 Views
- 1 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 1841 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
I am also looking for an answer to this question. Did you manage to find a solution @Andrewcon ?
- 1 kudos
- 40 Views
- 2 replies
- 0 kudos
Networking configuration of Azure Databricks managed storage account
Hi all,I created an Azure Databricks Workspace, and the workspace creates an Azure Databricks managed storage account.The networking configuration of the storage account is "Enabled from all networks".Shall I change it to "Enabled from selected virtu...
- 40 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Changing the networking configuration of your Azure Databricks managed storage account to "Enabled from selected virtual networks and IP addresses" is a good step for enhancing security. However, the "Insufficient permissions" status you are seeing f...
- 0 kudos
- 13 Views
- 2 replies
- 1 kudos
Is there any option in databricks to estimate cost of a query before execution
Hi Team,I want to check if there is any option in data bricks which can help to estimate cost of a query before execution?I mean calculate DBU before actual query execution based on the logical plan? Regards
- 13 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Is there any way to track the progress or ETA?Do we have access to ideas portal? Where can we search this reference number DB-I-5730?
- 1 kudos
- 8674 Views
- 5 replies
- 0 kudos
Error: Folder xxxx@xxx.com is protected
Hello, On Azure Databricks i'm trying to remove a folder on the Repos folder using the following command : databricks workspace delete "/Repos/xxx@xx.com"I got the following error message:databricks workspace delete "/Repos/xxxx@xx.com"Error: Folder ...
- 8674 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
You have deleted the user from the workspace and the folder is still showing?
- 0 kudos
- 25 Views
- 1 replies
- 0 kudos
Table Trigger - Too many logfiles
Hi, we have implemented a job that runs on a trigger of a table update. The job worked perfectly, until the source table now have accumulated too many log files and the job isn't running anymore. Only the error message below:Storage location /abcd/_d...
- 25 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @184754 Interesting topic, as the docs says:"Log files are deleted automatically and asynchronously after checkpoint operations and are not governed by VACUUM. While the default retention period of log files is 30 days, running VACUUM on a table r...
- 0 kudos
- 1032 Views
- 2 replies
- 0 kudos
Is it possible to use Liquid Clustering on Delta Live Tables / Materialised Views?
Is it possible to use Liquid Clustering on Delta Live Tables? If it is available what is the Python syntax for adding liquid clustering to a Delta Live Table / Materialised view please?
- 1032 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @amr, materialised views are not tables, they are views. Liquid clustering is not supported on views so it will throw [EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE] error. Unfortunately it will be the same case for the "optimize" command as well.
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
Voucher for Data Engineer Associate certification
Hi, I am looking for a voucher to complete the Data Engineer Associate certification. Could anyone help me please?
- 23 Views
- 0 replies
- 0 kudos
- 18 Views
- 0 replies
- 0 kudos
Mark Your Calendar: Data Day Texas + AI 2025 on January 25!
Hey Austin Databricks Group,If you're passionate about data and AI, here's an event you won't want to miss: Data Day Texas + AI 2025, happening on Saturday, January 25. This event brings together experts and enthusiasts to share insights, trends, and...
- 18 Views
- 0 replies
- 0 kudos
- 1280 Views
- 2 replies
- 0 kudos
Foreign table to delta streaming table
I want to copy a table from a foreign catalog as my streaming table. This is the code I used but I am getting error: Table table_name does not support either micro-batch or continuous scan.; spark.readStream .table(table_name) ...
- 1280 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
I also want to bump this! This is my exact problem right now as well.
- 0 kudos
Announcements
Latest from our Blog
Understanding Unity Catalog
Throughout the dozens of engagements I’ve had since joining Databricks, I’ve found that customers often struggle to understand the scope and concept of Unity Catalog. Questions like “Does it store my ...
Seamlessly Integrate Databricks on AWS with Power BI SSO Using Microsoft Entra ID
Introduction This article is a must-read if you manage data and analytics with Databricks on AWS, Power BI, and Microsoft Entra ID. Integrating Databricks on AWS with Power BI through Single Sign-On ...
Building an Inpainting App Using Hugging Face, Databricks Model Serving, and Databricks Apps
This is a summary of the blog: https://lnkd.in/dArDi-Cf The blog provides a comprehensive guide to building an image inpainting application, focusing on filling or reconstructing missing or undesired ...
Achieving High Data Quality with Databricks SQL: Techniques and Tools
Authors: Malay Panigrahi and Nikhil Chandna Organisations increasingly rely on data to fuel their decisions and gain a competitive edge. However, as the volume and complexity of data grow, so do the c...
