- 1187 Views
- 7 replies
- 3 kudos
Lakebridge questions
We have a few questions before we propose Lakebridge as the migration tooling for one of our customers, where the requirement is to migrate from Redshift to Databricks. We need help with your quick response so we can proceed with the next steps:Our u...
- 1187 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi community,We’re currently planning a migration from an on-premise SQL Server data warehouse (with associated SSIS packages) to Databricks Unity Catalog. As part of this effort, we’re evaluating the use of Lakebridge for assessment, conversion, and...
- 3 kudos
- 550 Views
- 7 replies
- 7 kudos
Slow PySpark operations after long DAG that contains many joins and transformations
We are using PySpark and notice that when we are doing many transformations/aggregations/joins of the data then at some point the execution time of simple task (count, display, union of 2 tables, ...) become very slow even if we have a small data (ex...
- 550 Views
- 7 replies
- 7 kudos
- 7 kudos
It's a cluster with 128 GO of memory, when looking in Spark UI there is 54 GO for storage memory. Honestly I don't think it's memory issue like I said it's a small data and if we do checkpoint at same point then continu we don't have the problem afte...
- 7 kudos
- 256 Views
- 1 replies
- 1 kudos
Resolved! Cannot export embedded dashboard widget as CSV or other formats except PNG
I’ve integrated a Databricks dashboard into my web application for all my users, following the guidelines in this article:Embedding Databricks Dashboards.This integration worked perfectly initially. However, I’m now encountering an issue with exporti...
- 256 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @rajg , Based on the link you shared, it looks to me like you have an external embedding situation? If so, this is a feature that is not currently available, but it is a commonly requested feature. External dashboard embedding is currently in Publ...
- 1 kudos
- 14145 Views
- 10 replies
- 2 kudos
What is the best way to read in a ms access .accdb database into Databricks from a mounted drive?
I am currently trying to read in .accdb files from a mounted drive. Based on my research it looks like I would have to use a package like JayDeBeApi with ucanaccess drivers or pyodbc with ms access drivers.Will this work?Thanks for any help.
- 14145 Views
- 10 replies
- 2 kudos
- 2 kudos
Hi @Leland Adams Hope you are doing well. Thank you for posting your question and giving us additional information. Do you think you were able to solve the query?We'd love to hear from you.
- 2 kudos
- 348 Views
- 2 replies
- 1 kudos
URGENT: Delta writes to S3 fail after workspace migrated to Premium
Delta writes to S3 fail after workspace migrated to Premium (401 “Credential was not sent or unsupported type”)SummaryAfter our Databricks workspace migrated from Standard to Premium, all Delta writes to S3 started failing with:com.databricks.s3commi...
- 348 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @DBU100725 - Are you using a No isolation shared cluster? Can you check if this was turned ON in your account?
- 1 kudos
- 474 Views
- 1 replies
- 1 kudos
Lakebridge conversion tool: Incorrect Databricks SQL script generated
Hi Team,I was able to successfully install and use the Lakebridge code conversion tool to convert my SQL Server script into a Databricks SQL script. However, the generated script contains several syntax errors. Could you please let me know if I might...
- 474 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi there!Known lakebase issues are listed here:https://github.com/databrickslabs/lakebridge/issuesDoes any of this apply to your use case?1. Variable scope errors in WHERE clauses or subqueries 2. DELETE/UPDATE FROM statements incorrectly converted ...
- 1 kudos
- 1600 Views
- 1 replies
- 1 kudos
Resolved! Issue with Root Folder Configuration in Databricks Asset Bundles for DLT Pipelines
I'm currently working with Databricks Asset Bundles to deploy my DLT pipelines, but I’ve encountered an issue I can't resolve.The problem is that I’m unable to configure the root folder within the Asset Bundle in a way that lets me define a custom pa...
- 1600 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Davila , I did some digging and have come with some things you can think about as you work through your issue. Here’s a clear way to think about what you’re seeing and how to proceed. What’s going on That “Root folder” field in the DLT UI is in...
- 1 kudos
- 2081 Views
- 6 replies
- 0 kudos
Notebook in path workspace/repos/.internal/**_commits/** was unable to be accessed
I have a workflow job (source is git) to access a notebook and execute it. From the job, it failed with error:Py4JJavaError: An error occurred while calling o466.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAI...
- 2081 Views
- 6 replies
- 0 kudos
- 0 kudos
Just some clarification: the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE. However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...
- 0 kudos
- 2384 Views
- 2 replies
- 0 kudos
Integration options for Databricks Jobs and DataDog?
I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and Spark Jobs and Tasks. I'm looking for somethin...
- 2384 Views
- 2 replies
- 0 kudos
- 0 kudos
Personally, I'm using their Data Jobs Monitoring product https://docs.datadoghq.com/data_jobs/databricks/ that works great and gives the right insights both for my high level job execution stats and Spark deeper metrics
- 0 kudos
- 272 Views
- 2 replies
- 1 kudos
Resolved! Can't mergeSchema handle int and bigint?
I have a table which has a column of data type 'bigint'. While overwriting it with new data, given that I do full loads, I used 'mergeSchema' to handle schema changes. The new data's datatype was int. I thought mergeSchema can easily handle that, but...
- 272 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Dhruv,Delta won't automatically upcast unless you explicitly handle it. Cast the column Lob_Pk to LongType (which maps to BIGINT in SQL/Delta). Try below snippetfrom pyspark.sql.functions import colfrom pyspark.sql.types import LongTypecrm_retail_...
- 1 kudos
- 209 Views
- 4 replies
- 2 kudos
Resolved! [Databricks Asset Bundles] Bug: driver_node_type_id not updated
Working with databricks asset bundles (using the new python-based definition), if you have a job_cluster defined using driver_node_type_id, and then update it to no longer have it defined, but only node_type_id, the driver node_type never gets update...
- 209 Views
- 4 replies
- 2 kudos
- 2 kudos
There is no built-in way in Databricks Asset bundles or terraform to automatically inherit the value of driver_node_type_id for node_type_id."You must set both explicitly in your configuration"You can always see your updated detail resource from the ...
- 2 kudos
- 1868 Views
- 2 replies
- 0 kudos
Resolved! Understanding least common type in databricks
I was reading the data type rules and found about least common type.I have a doubt. What is the least common type of STRING and INT? The referred link gives the following example saying the least common type is BIGINT.-- The least common type between...
- 1868 Views
- 2 replies
- 0 kudos
- 269 Views
- 4 replies
- 4 kudos
Resolved! Least Common Type is different in Serverless and All Purpose Cluster.
The following statement gives different outputs in different computes.In Databricks, 15.4 LTS%sqlSELECT typeof(coalesce(5, '6'));-- OutputstringIn Serverless, environment version 4%sqlSELECT typeof(coalesce(5, '6'));-- OutputbigintThere are other cas...
- 269 Views
- 4 replies
- 4 kudos
- 4 kudos
@Dhruv-22 Regarding your 1st question, I'm not sureYou can refer to https://docs.databricks.com/aws/en/sql/language-manual/parameters/ansi_mode#system-default to understand what happens when ansi mode is disabled
- 4 kudos
- 196 Views
- 2 replies
- 3 kudos
Resolved! Regarding : How to use Row_number() in dlt pipelines
We have two streaming tables : customer_info and customer_info_history and we joined them using full join to create temp table in pyspark and now we want to eliminate the de-duped records from this temp table. Tried using row_number() but facing bel...
- 196 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @anusha98 , You’re hitting a real limitation of Structured Streaming: non-time window functions (like row_number() over (...)) aren’t allowed on streaming DFs. You need to use agg().max() to get the “latest value per key” @dlt.table(name="temp_...
- 3 kudos
- 127 Views
- 1 replies
- 1 kudos
Lakeflow Pipelines Trying to Read accented file with spark.readStream but failure
Trying to read a accented file(French characters) but the spark.readStream function is not working and special characters turn into something strange(ex. �) spark.readStream .format("cloudfiles") .option("cloudFiles....
- 127 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @AmarKap , When Spark decodes CP1252 bytes as UTF-8/ISO-8859-1, you’ll see the replacement char like � Can you read the file as : df = (spark.readStream.format("cloudFiles").option("cloudFiles.format", "text").option("encoding", "windows-1252")...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
