Over the past few months, we’ve been gathering your feedback and focusing on both the quality of Databricks Assistant’s responses and the overall user experience. Today, we're excited to showcase a more advanced Databricks Assistant, packed with powe...
- 473 Views
- 0 replies
- 2 kudos
2 weeks ago
Meet Filip Niziol, a Senior Data Engineer at Digiterre and an active contributor to the Databricks Community. With a rich background in data architecture and a passion for building scalable data solutions, Filip shares insights on his career, Databri...
- 324 Views
- 3 replies
- 5 kudos
Monday
Each month, we recognize one outstanding member of our Databricks Community who has made a real impact. Whether you share solutions, spark discussions, or help others, this is your chance to get featured! What’s a Monthly Community Champion? Our ...
- 202 Views
- 0 replies
- 4 kudos
Monday
Over the years, organizations have amassed a vast amount of unstructured text data—documents, reports, and emails—but extracting meaningful insights has remained a challenge. Large Language Models (LLMs) now offer a scalable way to analyze this data,...
- 490 Views
- 0 replies
- 1 kudos
2 weeks ago
Migrating your data warehouse workloads is one of the most challenging yet essential tasks for any organization. Whether the motivation is the growth of your business and scalability requirements or reducing the high license and hardware cost of your...
- 1218 Views
- 0 replies
- 2 kudos
2 weeks ago
Community Activity
- 107 Views
- 3 replies
- 0 kudos
why latestOffset and getBatch takes so long time
Hi team,Kinesis -> delta table raw -> job with trigger=availableNow -> delta table target. The Kinesis->delta table raw is running continuously. The job is daily with trigger=availableNow. The job reads from raw, do some transformation, and run a MER...
- 107 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
When delta table tries to restore a version, it can find the closest checkpoint and apply versions up to the target. However, delta table needs to read and apply many versions to get changes between 2 versions. There are no compact file can be used. ...
- 0 kudos
- 9 Views
- 0 replies
- 0 kudos
Databricks Python function achieving Parallelism
Hello everyone,I have a very basic question wrt Databricks spark parallelism.I have a python function within a for loop, so I believe this is running sequentially.Databricks cluster is enabled with Photon and with Spark 15x, does that mean the driver...
- 9 Views
- 0 replies
- 0 kudos
- 10 Views
- 0 replies
- 0 kudos
Just Passed Databricks-Machine-Learning-Professional exam
Hi guys,I have the ML Professional exam scehduled for later this month and while I can find many resources, practice exams, and posts related to the ML Associate exam, I'm having trouble finding the same for the Professional exam.Anyone happen to hav...
- 10 Views
- 0 replies
- 0 kudos
- 41 Views
- 2 replies
- 1 kudos
Unverified Commits via Databricks Repos: Seeking Solution for GitHub Verification
The team is integrating Databricks Repos with Personal Access Tokens (PAT) to commit code directly to GitHub. Our organization requires signed commits for verification purposes.Issue: When committing via Databricks Repos, the commits appear as unveri...
- 41 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Can you please share the link to this doc DB-I-3082. I couldn't find it.
- 1 kudos
- 39 Views
- 1 replies
- 0 kudos
Is it possible to configure Databricks to leverage an AI PC’s NVMe storage for faster I/O?
Hello Databricks Community,I’m working on improving the performance of my Databricks workflows and exploring different storage options. I have an AI PC with NVMe storage, and I want to know if it’s possible to set up Databricks to use this storage fo...
- 39 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This is not how you should be using Databricks. Databricks is SaaS/Paas and it's main operating model is operate on data stored in Cloud Object Storage (GCS, S3, ADLS Gen 2). It is optimized for cloud storage, morever it is designed to be very fast...
- 0 kudos
- 44 Views
- 1 replies
- 0 kudos
Help needed with automated file ingestion process for full refresh mode using autoloader preferable
Description:As a data engineer, I need to implement an automated process to ingest data from multiple files in a subdirectory and create corresponding bronze tables. This process should handle full file refreshes and consider strategies to limit the ...
- 44 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
What do you mean by "full file refreshes"? Does this refer to the fact that file names will be reused?
- 0 kudos
- 104 Views
- 3 replies
- 1 kudos
cloud_infra_costs
I was looking at the system catalog and realized that there is an empty table called cloud_infra_costs. Could you tell me what is this for and why it is empty?
- 104 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks for replying. This makes sense. Any idea why it is empty and what to do to populate this?
- 1 kudos
- 1973 Views
- 2 replies
- 2 kudos
The New Data Analyst: Transforming BI in the Age of AI with Databricks Genie
Introduction Data and AI media will try to convince you of two terribly misguided ideas about AI’s impact on analytics. First of all, there’s the idea that prompt engineers who know just the right phrase to unlock the secret powers of generative AI w...
- 1973 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
I missed this question, sorry - Genie API is in private preview, reach out to your account team for access
- 2 kudos
- 70 Views
- 2 replies
- 0 kudos
UC, pyodbc, Shared Cluster, and :Can't open lib 'ODBC Driver 17 for SQL Server' : file not found
Hey Databricks! Trying to use the pyodbc init script in a Volume in UC on a shared compute cluster but receive error: "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"). I fo...
- 70 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Benni ,Ensure that the ODBC Driver 17 for SQL Server is installed on the Databricks cluster, you can verify this by running the following command in a notebook cell on the cluster to check if the driver is available in the system:!ls /opt/microso...
- 0 kudos
- 32 Views
- 1 replies
- 0 kudos
Cannot downgrade workspace object permissions using API
Hi!I'd like to restrict some users' permissions using REST API and got an issue while trying to update a permission on 'directories'.I'm trying to set a user's permission on their default username folder in the workspace to 'can edit' so that they ca...
- 32 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @takak, Greetings from Databricks! What is the REST API you are making the call to? Looks like this might not be supported programmatically, but will try to test it internally. it appears that the CAN_MANAGE permission is a higher-level permission...
- 0 kudos
- 82 Views
- 2 replies
- 0 kudos
I want to play from my phone
Maybe someone can advise something so that you can play on the road or on vacation
- 82 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey there! If you're looking to use your phone for some fun, it's totally possible and actually pretty convenient! For ios users, there are plenty of options to enjoy a variety of activities with ease. One platform you might want to check out is Jeet...
- 0 kudos
- 30 Views
- 1 replies
- 0 kudos
Error Creating Table from delta location dbfs
[UC_FILE_SCHEME_FOR_TABLE_CREATION_NOT_SUPPORTED] Creating table in Unity Catalog with file scheme dbfs is not supported. Instead, please create a federated data source connection using the CREATE CONNECTION command for the same table provider, then ...
- 30 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @vyasakhilesh,The error you're seeing, [UC_FILE_SCHEME_FOR_TABLE_CREATION_NOT_SUPPORTED], occurs because Unity Catalog in Databricks does not support creating tables directly from DBFS (Databricks File System) locations.In this case, you're trying...
- 0 kudos
- 101 Views
- 6 replies
- 0 kudos
Resolved! SQL View Formatting in Catalog - Can you turn it off?
It appears as though Databricks now formats SQL View definitions when showing them in the Catalog. Our solution is based on views, and we have comment tags in those views. We format these views so that it is easy for us to find and update parts of ...
- 101 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
Thank you! I will submit that request.In case anyone else stumbles upon this post, here is a function you can add that will return the view definition from information_schema, unformatted with an ALTER statement at the top:DROP FUNCTION IF EXISTS <c...
- 0 kudos
- 34 Views
- 1 replies
- 0 kudos
Cannot able to attend the quiz after databricks lakehouse fundamentals
Hi, I have already finished my databricks fundamentals quiz in my partner academy account. I got the certificate (There was no badge logo in the certificate), and also not yet received my badge in my mail..its been 2 days since i completed my exam.Ki...
- 34 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Kaushi, Please file a ticket with our support team and allow the support team 24-48 hours for a resolution.
- 0 kudos
- 54 Views
- 1 replies
- 0 kudos
Cleared the exam , however didnt receive the certificate yet after 48 hours
Cleared the exam for databricks data engg associate , however I didnt receive the certificate yet even now 2 days are already passed
- 54 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@priti18 Please file a ticket with our support team and allow the support team 24-48 hours for a resolution.
- 0 kudos
Announcements
Latest from our Blog
Getting started with fine-tuning on Databricks
Authors: Ellen Hirt, Narjes Majdoub, Giran Moodley Contents ContentsIntroductionSection 1: Preparing the EnvironmentSection 2: Step-by-Step Instructions1. Data preparation2. Model initialisation and t...
Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs
Databricks Unity Catalog (UC) is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Unity Catalog provides a single source ...
Databricks observability using Grafana and Prometheus
Databricks Observability using Grafana and Prometheus As software systems scale, the amount of data they process and the work they do grows with them. This is not limited only to cloud-native services...
Best practices for safe data experimentation with Databricks
In today’s data-centric world, experimentation is essential for developers and data scientists to create cutting-edge models, test hypotheses, and build robust data pipelines. However, giving these te...
