- 3998 Views
- 2 replies
- 2 kudos
tqdm progressbar in Databricks jobs
Hi,I'm using Databricks workflows to run a train job using `pytorch` + `lightning`. `lightning` has a built in progressbar built on `tqdm` that tracks the progress. It works OK in when I run the notebook outside of a workflow. But when I try to run n...
- 3998 Views
- 2 replies
- 2 kudos
- 2 kudos
I have found that only progressbar2 can work in both interactive notebooks and workflow notebooks. It's limited, but better than nothing. Tqdm is broken in workflows.
- 2 kudos
- 1046 Views
- 3 replies
- 0 kudos
GCP Serverless SQL Warehouse Tag Propagation?
I have a serverless SQL Warehouse with a tag on it that is not making it to GCP.We have various job and AP clusters with tags that I can see in GCP- trying to have everything tagged for the purpose of monitoring billing/usage centrally.Do serverless ...
- 1046 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey @Kayla ,This is because serverless infrastructure is fully managed by Databricks, and you do not have direct control over the underlying resources as you do with standard clusters (non-serverless SQL warehouse)You can track SQL Warehouse usage wi...
- 0 kudos
- 7421 Views
- 6 replies
- 8 kudos
Reading an Iceberg table with AWS Glue Data Catalog as metastore
I have created an Iceberg table using AWS Glue, however whenever I try to read it using a Databricks cluster, I get `java.lang.InstantiationException`. I have tried every combination of Spark configs for my Databricks compute cluster that I can think...
- 7421 Views
- 6 replies
- 8 kudos
- 8 kudos
In case someone happens upon this in the future, I ended up using Unity Catalog with Hive metastore federation for Glue. The Iceberg support is currently "coming soon in Public Preview."
- 8 kudos
- 3182 Views
- 10 replies
- 2 kudos
Unity Credential Scope id not found in thread locals
i am facing issue :- [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Unity Credential Scope id not found in thread locals.Issue occurs:-when we try to list files using dbutils.fs.lsand also this occurs at times when we try to write o...
- 3182 Views
- 10 replies
- 2 kudos
- 2 kudos
Thanks for the reply.Its working in dbr 15.4 but I want to use it with 13.3, is there a workaround?
- 2 kudos
- 4431 Views
- 4 replies
- 0 kudos
Best practices for ensuring data quality in batch pipelines
Hello everyone,I couldn't find a topic on this - what are your best practices to ensuring data quality in batch pipelines?I've got a big pipeline processing data once per day. We though about either going with DBT or DLT but DLT seems more directed f...
- 4431 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey Greg_cI use DBT daily for batch data ingestion, and I believe it’s a great option. However, it’s important to consider that adopting DBT introduces additional complexity, and the team should carefully evaluate the impact of adding a new tool to t...
- 0 kudos
- 3299 Views
- 5 replies
- 1 kudos
Cluster idle time and usage details
How can we find out the usage details of the Databricks cluster? Specifically, we need to know how many nodes are in use, how long the cluster is idle, the time it takes to start up, and the jobs it is running along with their durations. Is there a q...
- 3299 Views
- 5 replies
- 1 kudos
- 1 kudos
Hey @hboleto It’s difficult to accurately estimate the final cost of a Serverless cluster, as it is fully managed by Databricks. In contrast, Classic clusters allow for finer resource tuning since you can define spot instances and other instance type...
- 1 kudos
- 1826 Views
- 1 replies
- 0 kudos
ETL or Transformations Testing Framework for Databricks
Hi! I'm looking for any ETL or Transformations Testing Framework for Databricks -need to support automation of the following steps:1) create/store test datasets (mock inputs and a golden copy of the output),2) run ETL (notebook) being tested3) compar...
- 1826 Views
- 1 replies
- 0 kudos
- 0 kudos
You can do all of this yourself with a testing workflow. You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests. This would be the first step of the workflow. Then call your notebooks. Your c...
- 0 kudos
- 639 Views
- 1 replies
- 0 kudos
Unable to pass the task variables from Python Wheel to ForEach task
I understand that task variables are supported in Databricks notebook , but there is a requirement from client to use python wheel package in Databricks workflow . We are not able to set the task variables using dbutils in python wheel file. Kindly s...
- 639 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Divya_sreeE you can pass dynamic variables between tasks using Databricks' job parameters.1) In your first Python wheel task, generate the dynamic variables and use the Databricks REST API to update the job parameters.2) In the For Each loop, ret...
- 0 kudos
- 10190 Views
- 5 replies
- 3 kudos
Resolved! Add Identity Column to Existing Table
Hello everyone. I am working with tables that need an identity column. I currently have a view in which I cast the different columns to the data type that I want. Now I want the result of this view to be inserted or merged into a table. The schema of...
- 10190 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @Jasper Puts​ how did you solve this issue of creating a identity column to existing table.I'm also getting the same error as you got.
- 3 kudos
- 2458 Views
- 3 replies
- 1 kudos
Export data from databricks to prem
Hello everyoneI need to export some data to sql server management studio on premise.I need to verify that the new data on databricks is aligned with the older data that we have on premise.Is it possible to export data as an Excel sheet or .csv file?R...
- 2458 Views
- 3 replies
- 1 kudos
- 1 kudos
You can compare your databricks data with on-prem sql server data in two ways:Firstly, you have to make connection between sql-server and databricks using volumes. Using volumes we can mount sql server data into databricks unity catalog1.we can comp...
- 1 kudos
- 5250 Views
- 2 replies
- 1 kudos
Resolved! Databricks 13.3LTS to 15.4 LTS Migration - Spark job with source DB2 database not working
I'm trying to migrate a spark job from Databricks 13.3 LTS to 15.4 LTS. The spark job uses db2jcc4.jar for DB2 database connection. Below is the spark code:%scala // Import Spark SQL import org.apache.spark.sql.SparkSession // Create a Spark session...
- 5250 Views
- 2 replies
- 1 kudos
- 2635 Views
- 1 replies
- 0 kudos
Handling Unknown Fields in DLT Pipeline
HiI'm working on a DLT pipeline where I read JSON files stored in S3.I'm using the auto loader to identify the file schema and adding schema hints for some fields to specify their type.When running it against a single data file that contains addition...
- 2635 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi community and @mikeagicman i saw this error when trying to load a json file. I discovered the problem was that the schemaLocation i was using was pointing to a different table schema, so it was trying to match columns that did not exist. When i se...
- 0 kudos
- 4572 Views
- 4 replies
- 4 kudos
Using ipywidgets latest versions
Hello everyone,I upgraded my cluster to DBR 13.0, which comes with ipywidgets version 7.7.2 installed.However, I want to use the TagsInput widget, which is new since version 8.0.4.If i upgrade the ipywidgets package to version 8.0.4, none of the widg...
- 4572 Views
- 4 replies
- 4 kudos
- 4 kudos
I can confirm that installing a newer ipywidgets library version at a cluster level does not resolve these issues. The arcgis library relies on ipywidgets v8 to render maps. Even when I install ipywidgets > 8 at the cluster level, the widgets still d...
- 4 kudos
- 750 Views
- 3 replies
- 1 kudos
- 750 Views
- 3 replies
- 1 kudos
- 1 kudos
Unfortunately as this is from the Azure side I do not have specific steps or example i can share
- 1 kudos
- 30563 Views
- 19 replies
- 15 kudos
How to enable/verify cloud fetch from PowerBI
I tried to benchmark the Powerbi Databricks connector vs the powerbi Delta Lake reader on a dataset of 2.15million rows. I found that the delta lake reader used 20 seconds, while importing through the SQL compute endpoint took ~75 seconds. When I loo...
- 30563 Views
- 19 replies
- 15 kudos
- 15 kudos
I'm troubleshooting slow speeds (~6Mbps) from Azure Databricks to the PowerBI Service (Fabric) via dataflows.Drivers are up to date. PowerBI is using Microsoft's Spark ODBC driver Version 2.7.6.1014, confirmed via log4j.HybridCloudStoreResultHandler...
- 15 kudos
| User | Count |
|---|---|
| 1623 | |
| 790 | |
| 501 | |
| 349 | |
| 287 |
