- 2921 Views
- 2 replies
- 3 kudos
Resolved! ModuleNotFound error when using transformWithStateInPandas via a class defined outside the notebook
As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...
- 2921 Views
- 2 replies
- 3 kudos
- 3 kudos
This is no longer an issue; it must be some patch version of DBX Runtime 16.4 fixed it and it works now without doing any changes to original code.Thanks.
- 3 kudos
- 3213 Views
- 5 replies
- 4 kudos
Resolved! when automatic liquid clustering is enabled, how to know which columns are used for clustering?
Let's say a table is configured to have automatic liquid clustering:ALTER TABLE table1 CLUSTER BY AUTO; How to know which columns were chosen by Databricks?
- 3213 Views
- 5 replies
- 4 kudos
- 4 kudos
From the documentation, it seems that in Python, there is such an option, only when creating or replacing a table.# To set clustering columns and auto, which serves as a way to give a hint # for the initial selection. df.writeTo(...).using("delta") ...
- 4 kudos
- 3652 Views
- 7 replies
- 3 kudos
Lakebridge questions
We have a few questions before we propose Lakebridge as the migration tooling for one of our customers, where the requirement is to migrate from Redshift to Databricks. We need help with your quick response so we can proceed with the next steps:Our u...
- 3652 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi community,We’re currently planning a migration from an on-premise SQL Server data warehouse (with associated SSIS packages) to Databricks Unity Catalog. As part of this effort, we’re evaluating the use of Lakebridge for assessment, conversion, and...
- 3 kudos
- 7927 Views
- 7 replies
- 7 kudos
Slow PySpark operations after long DAG that contains many joins and transformations
We are using PySpark and notice that when we are doing many transformations/aggregations/joins of the data then at some point the execution time of simple task (count, display, union of 2 tables, ...) become very slow even if we have a small data (ex...
- 7927 Views
- 7 replies
- 7 kudos
- 7 kudos
It's a cluster with 128 GO of memory, when looking in Spark UI there is 54 GO for storage memory. Honestly I don't think it's memory issue like I said it's a small data and if we do checkpoint at same point then continu we don't have the problem afte...
- 7 kudos
- 1829 Views
- 1 replies
- 1 kudos
Resolved! Cannot export embedded dashboard widget as CSV or other formats except PNG
I’ve integrated a Databricks dashboard into my web application for all my users, following the guidelines in this article:Embedding Databricks Dashboards.This integration worked perfectly initially. However, I’m now encountering an issue with exporti...
- 1829 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @rajg , Based on the link you shared, it looks to me like you have an external embedding situation? If so, this is a feature that is not currently available, but it is a commonly requested feature. External dashboard embedding is currently in Publ...
- 1 kudos
- 18247 Views
- 10 replies
- 2 kudos
What is the best way to read in a ms access .accdb database into Databricks from a mounted drive?
I am currently trying to read in .accdb files from a mounted drive. Based on my research it looks like I would have to use a package like JayDeBeApi with ucanaccess drivers or pyodbc with ms access drivers.Will this work?Thanks for any help.
- 18247 Views
- 10 replies
- 2 kudos
- 2 kudos
Hi @Leland Adams Hope you are doing well. Thank you for posting your question and giving us additional information. Do you think you were able to solve the query?We'd love to hear from you.
- 2 kudos
- 1399 Views
- 2 replies
- 1 kudos
URGENT: Delta writes to S3 fail after workspace migrated to Premium
Delta writes to S3 fail after workspace migrated to Premium (401 “Credential was not sent or unsupported type”)SummaryAfter our Databricks workspace migrated from Standard to Premium, all Delta writes to S3 started failing with:com.databricks.s3commi...
- 1399 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @DBU100725 - Are you using a No isolation shared cluster? Can you check if this was turned ON in your account?
- 1 kudos
- 1356 Views
- 1 replies
- 1 kudos
Lakebridge conversion tool: Incorrect Databricks SQL script generated
Hi Team,I was able to successfully install and use the Lakebridge code conversion tool to convert my SQL Server script into a Databricks SQL script. However, the generated script contains several syntax errors. Could you please let me know if I might...
- 1356 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi there!Known lakebase issues are listed here:https://github.com/databrickslabs/lakebridge/issuesDoes any of this apply to your use case?1. Variable scope errors in WHERE clauses or subqueries 2. DELETE/UPDATE FROM statements incorrectly converted ...
- 1 kudos
- 4850 Views
- 1 replies
- 1 kudos
Resolved! Issue with Root Folder Configuration in Databricks Asset Bundles for DLT Pipelines
I'm currently working with Databricks Asset Bundles to deploy my DLT pipelines, but I’ve encountered an issue I can't resolve.The problem is that I’m unable to configure the root folder within the Asset Bundle in a way that lets me define a custom pa...
- 4850 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Davila , I did some digging and have come with some things you can think about as you work through your issue. Here’s a clear way to think about what you’re seeing and how to proceed. What’s going on That “Root folder” field in the DLT UI is in...
- 1 kudos
- 5407 Views
- 6 replies
- 0 kudos
Notebook in path workspace/repos/.internal/**_commits/** was unable to be accessed
I have a workflow job (source is git) to access a notebook and execute it. From the job, it failed with error:Py4JJavaError: An error occurred while calling o466.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAI...
- 5407 Views
- 6 replies
- 0 kudos
- 0 kudos
Just some clarification: the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE. However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...
- 0 kudos
- 4891 Views
- 2 replies
- 0 kudos
Integration options for Databricks Jobs and DataDog?
I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and Spark Jobs and Tasks. I'm looking for somethin...
- 4891 Views
- 2 replies
- 0 kudos
- 0 kudos
Personally, I'm using their Data Jobs Monitoring product https://docs.datadoghq.com/data_jobs/databricks/ that works great and gives the right insights both for my high level job execution stats and Spark deeper metrics
- 0 kudos
- 1579 Views
- 4 replies
- 2 kudos
Resolved! [Databricks Asset Bundles] Bug: driver_node_type_id not updated
Working with databricks asset bundles (using the new python-based definition), if you have a job_cluster defined using driver_node_type_id, and then update it to no longer have it defined, but only node_type_id, the driver node_type never gets update...
- 1579 Views
- 4 replies
- 2 kudos
- 2 kudos
There is no built-in way in Databricks Asset bundles or terraform to automatically inherit the value of driver_node_type_id for node_type_id."You must set both explicitly in your configuration"You can always see your updated detail resource from the ...
- 2 kudos
- 3271 Views
- 2 replies
- 0 kudos
Resolved! Understanding least common type in databricks
I was reading the data type rules and found about least common type.I have a doubt. What is the least common type of STRING and INT? The referred link gives the following example saying the least common type is BIGINT.-- The least common type between...
- 3271 Views
- 2 replies
- 0 kudos
- 2568 Views
- 4 replies
- 4 kudos
Resolved! Least Common Type is different in Serverless and All Purpose Cluster.
The following statement gives different outputs in different computes.In Databricks, 15.4 LTS%sqlSELECT typeof(coalesce(5, '6'));-- OutputstringIn Serverless, environment version 4%sqlSELECT typeof(coalesce(5, '6'));-- OutputbigintThere are other cas...
- 2568 Views
- 4 replies
- 4 kudos
- 4 kudos
@Dhruv-22 Regarding your 1st question, I'm not sureYou can refer to https://docs.databricks.com/aws/en/sql/language-manual/parameters/ansi_mode#system-default to understand what happens when ansi mode is disabled
- 4 kudos
- 2622 Views
- 2 replies
- 3 kudos
Resolved! Regarding : How to use Row_number() in dlt pipelines
We have two streaming tables : customer_info and customer_info_history and we joined them using full join to create temp table in pyspark and now we want to eliminate the de-duped records from this temp table. Tried using row_number() but facing bel...
- 2622 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @anusha98 , You’re hitting a real limitation of Structured Streaming: non-time window functions (like row_number() over (...)) aren’t allowed on streaming DFs. You need to use agg().max() to get the “latest value per key” @dlt.table(name="temp_...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
