- 20 Views
- 1 replies
- 0 kudos
Unavailable GPU compute
Hello,I would like to create a ML compute with GPU. I am on GCP europe-west1 and the only available options for me are the G2 family and one instance of the A3 family (a3-highgpu-8g [H100]). I have been trying multiple times at different times but I ...
- 20 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @oye ,You’re hitting a cloud capacity issue, not a Databricks configuration problem. The Databricks GCP GPU docs list A2 and G2 as the supported GPU instance families. A3/H100 is not in the supported list: https://docs.databricks.com/gcp/en/comput...
- 0 kudos
- 13 Views
- 0 replies
- 0 kudos
pkgutils walk_packages stopped working in DBR 17.2
Hi,After moving from Databricks runtime 17.1 to 17.2 suddenly my pkgutils walk_packages doesn't identify any packages within my repository anymore.This is my example code:import pkgutil import os packages = pkgutil.walk_packages([os.getcwd()]) print...
- 13 Views
- 0 replies
- 0 kudos
- 14 Views
- 0 replies
- 0 kudos
Unable to read from or write to Snowflake Open Catalog via Databricks
I have Snowflake Iceberg tables whose metadata is stored in Snowflake Open Catalog. I am trying to read these tables from the Open Catalog and write back to the Open Catalog using Databricks.I have explored the available documentation but haven’t bee...
- 14 Views
- 0 replies
- 0 kudos
- 39 Views
- 0 replies
- 0 kudos
spark conf for serveless jobs
Hello Guys, I use serveless on databricks Azure, so i have build a decorator which instanciate a SparkSession. My job use autolaoder / kafka using mode availableNow. Someone Knows which spark conf is required beacause i want to add it ? Thanx import...
- 39 Views
- 0 replies
- 0 kudos
- 193 Views
- 2 replies
- 2 kudos
Resolved! setup databricks connect on VsCode and PyCharm
Hello Guyz,Someone Know what's is the best pratices to setup databricks connect for Pycharm and VsCode using Docker, Justfile and .env file Cordially, Seefoods
- 193 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @seefoods!I’ve worked with Databricks Connect and VSCode in different projects, and although your question mentions Docker, Justfile and .env, the “best practices” really depend on what you’re trying to do. Here’s what has worked best for me:1.- D...
- 2 kudos
- 91 Views
- 3 replies
- 0 kudos
Read Array of Arrays of Objects JSON file using Spark
Hi Databricks Community! This is my first post in this forum, so I hope you can forgive me if it's not according to the forum best practices After lots of searching, I decided to share the peculiar issue I'm running into in this community.I try to lo...
- 91 Views
- 3 replies
- 0 kudos
- 0 kudos
I guess I was a bit over enthusiastic by accepting the answer.When I run the following on the single object array of arrays (as shown in the original post) I get a single row with column "value" and value null. from pyspark.sql import functions as F,...
- 0 kudos
- 75 Views
- 2 replies
- 3 kudos
Resolved! Data Bricks Engineer - DEA Exam vs Training
Hi, I love the Databricks resources but I'm a little confused on what training to take. My focus is studying and practicing for the Databricks Engineer Associate exam, but when I hear of the 'training', I'm not sure which training people are referrin...
- 75 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @rc10000!+1 to what @Louis_Frolio mentioned above.The Learning Plan is designed for users preparing for the Databricks Certified Data Engineer Associate and Professional exams. Also below are a few paths, depending on what you’re looking for: ...
- 3 kudos
- 77 Views
- 1 replies
- 1 kudos
Resolved! Lakeflow Connect - Databricks Data Engineer Associate Exam Post-July 2025
Hi, I'm asking another Databricks Data Engineer Associate Exam Dec 2025 question. For those who have taken the DEA exam, is Lakeflow Connect a relevant topic for the test? Been a little confused on what resource to rely on besides the official study ...
- 77 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @rc10000,Lakeflow Connect is mentioned in the exam guide under training, but it’s more about the ingestion concepts. These topics come under the Development & Ingestion section. I’d suggest following the official exam guide first and Databricks Ac...
- 1 kudos
- 354 Views
- 6 replies
- 5 kudos
IDENTIFIER in SQL Views not supported?
Dear community,We are phasing out the dollar param `${catalog_name}` because it has been deprecated since runtime 15.2.We use this parameter in many queries and should now be replaced by the IDENTIFIER clause.In the query below where we retrieve data...
- 354 Views
- 6 replies
- 5 kudos
- 5 kudos
I have good news: in runtime 18, IDENTIFIER and parameter markers are supported everywhere! We need to wait a month or two as the SQL warehouse and serverless are still on runtime 17.
- 5 kudos
- 52 Views
- 0 replies
- 0 kudos
Queries from Serverless compute referring to older/deleted/vacuumed version of the delta tables.
Hi Team,I have a unity catalog based managed delta table which I am able to successfully query using the regular compute/cluster options.But when I try to query the same table using a Serverless/SQL Warehouse, they are referring to an older version /...
- 52 Views
- 0 replies
- 0 kudos
- 165 Views
- 2 replies
- 0 kudos
Databricks pipeline fails expectation on execute python script, throws error: Update FAILES
Hi Community,I'm new to Databricks and am trying to make and implement pipeline expectations, The pipelines work without errors and my job works. I've tried multiple ways to implement expectations, sql and python. I keep resolving the errors but end ...
- 165 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, @RobFer1985 The primary cause of your error is a circular reference in your logic: you are defining a table named orders_2 while simultaneously trying to readStream from that same table. In Delta Live Tables (DLT), the function acts as the "wr...
- 0 kudos
- 2672 Views
- 1 replies
- 1 kudos
"Error: cannot read mws credentials: invalid Databricks Account configuration" on TF Destroy
I have a terraform project that creates a workspace in Databricks, assigns it to an existing metastore, then creates external location/storage credential/catalog. The apply works and all expected resources are created. However, without touching any r...
- 2672 Views
- 1 replies
- 1 kudos
- 1 kudos
I have the same problem, were you able to solve it?
- 1 kudos
- 108 Views
- 4 replies
- 0 kudos
Power BI refresh job task
I have tried Databricks job task to refresh power bi dataset and I have found 2 issues.1. I set up tables in Power BI Desktop using Import mode. After deploying the model to Power BI Service, I was able to download it as an Import mode model. However...
- 108 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you send a screenshot of the refresh power BI task in the jobs UI within Databricks please?
- 0 kudos
- 134 Views
- 5 replies
- 0 kudos
Extract Snowflake data based on environment
Hi all, In the development workspace, I need to extract data from a table/view in Snowflake development environment. Example table is called as VD_DWH.SALES.SALES_DETAILWhen we deploy the code into production, it needs to extract data from a table/vi...
- 134 Views
- 5 replies
- 0 kudos
- 0 kudos
Create a single job that runs your migration notebook.In the job settings, under Parameters, add a key like env with a default value (e.g., dev).When you create different job runs (or schedule them), override the parameter:For development runs, set e...
- 0 kudos
- 2238 Views
- 3 replies
- 0 kudos
unity catalog system.access.audit lag
Hello,We have unity catalog enabled workspace. To get the completion time of a pipeline that runs multiple times a day, I am checking system.access.audit table. Comparing the completion time of the pipeline compared to other pipeline time I am creat...
- 2238 Views
- 3 replies
- 0 kudos
- 0 kudos
@angel_ba - This is expected/designed behaviour.Audit logs are ingested into the system tables asynchronously. Databricks batches these events befor surfacing them in UC system tables. Alternate (prhaps) the best way is to use Job API for start/compl...
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 791 | |
| 512 | |
| 349 | |
| 287 |
