I am getting an error when the entity I need to store in a ValueState is a large array (over 15k-20k items). No error (and works correctly) if I trim the array to under 10k samples. The same error is raised when using it as a value for MapState or as...
As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...
This is no longer an issue; it must be some patch version of DBX Runtime 16.4 fixed it and it works now without doing any changes to original code.Thanks.
Let's say a table is configured to have automatic liquid clustering:ALTER TABLE table1 CLUSTER BY AUTO; How to know which columns were chosen by Databricks?
From the documentation, it seems that in Python, there is such an option, only when creating or replacing a table.# To set clustering columns and auto, which serves as a way to give a hint # for the initial selection. df.writeTo(...).using("delta") ...
We have a few questions before we propose Lakebridge as the migration tooling for one of our customers, where the requirement is to migrate from Redshift to Databricks. We need help with your quick response so we can proceed with the next steps:Our u...
Hi community,We’re currently planning a migration from an on-premise SQL Server data warehouse (with associated SSIS packages) to Databricks Unity Catalog. As part of this effort, we’re evaluating the use of Lakebridge for assessment, conversion, and...
We are using PySpark and notice that when we are doing many transformations/aggregations/joins of the data then at some point the execution time of simple task (count, display, union of 2 tables, ...) become very slow even if we have a small data (ex...
It's a cluster with 128 GO of memory, when looking in Spark UI there is 54 GO for storage memory. Honestly I don't think it's memory issue like I said it's a small data and if we do checkpoint at same point then continu we don't have the problem afte...
I’ve integrated a Databricks dashboard into my web application for all my users, following the guidelines in this article:Embedding Databricks Dashboards.This integration worked perfectly initially. However, I’m now encountering an issue with exporti...
Hi @rajg , Based on the link you shared, it looks to me like you have an external embedding situation? If so, this is a feature that is not currently available, but it is a commonly requested feature. External dashboard embedding is currently in Publ...
I am currently trying to read in .accdb files from a mounted drive. Based on my research it looks like I would have to use a package like JayDeBeApi with ucanaccess drivers or pyodbc with ms access drivers.Will this work?Thanks for any help.
Hi @Leland Adams Hope you are doing well. Thank you for posting your question and giving us additional information. Do you think you were able to solve the query?We'd love to hear from you.
Delta writes to S3 fail after workspace migrated to Premium (401 “Credential was not sent or unsupported type”)SummaryAfter our Databricks workspace migrated from Standard to Premium, all Delta writes to S3 started failing with:com.databricks.s3commi...
Hi @DBU100725 - Are you using a No isolation shared cluster? Can you check if this was turned ON in your account?
Hi Team,I was able to successfully install and use the Lakebridge code conversion tool to convert my SQL Server script into a Databricks SQL script. However, the generated script contains several syntax errors. Could you please let me know if I might...
Hi there!Known lakebase issues are listed here:https://github.com/databrickslabs/lakebridge/issuesDoes any of this apply to your use case?1. Variable scope errors in WHERE clauses or subqueries 2. DELETE/UPDATE FROM statements incorrectly converted ...
I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...
Hi @Oumeima , One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file. For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/....
I'm currently working with Databricks Asset Bundles to deploy my DLT pipelines, but I’ve encountered an issue I can't resolve.The problem is that I’m unable to configure the root folder within the Asset Bundle in a way that lets me define a custom pa...
Hey @Davila , I did some digging and have come with some things you can think about as you work through your issue. Here’s a clear way to think about what you’re seeing and how to proceed. What’s going on That “Root folder” field in the DLT UI is in...
I have a workflow job (source is git) to access a notebook and execute it. From the job, it failed with error:Py4JJavaError: An error occurred while calling o466.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAI...
Just some clarification: the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE. However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...
I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and Spark Jobs and Tasks. I'm looking for somethin...
Personally, I'm using their Data Jobs Monitoring product https://docs.datadoghq.com/data_jobs/databricks/ that works great and gives the right insights both for my high level job execution stats and Spark deeper metrics
I have a table which has a column of data type 'bigint'. While overwriting it with new data, given that I do full loads, I used 'mergeSchema' to handle schema changes. The new data's datatype was int. I thought mergeSchema can easily handle that, but...
Hi Dhruv,Delta won't automatically upcast unless you explicitly handle it. Cast the column Lob_Pk to LongType (which maps to BIGINT in SQL/Delta). Try below snippetfrom pyspark.sql.functions import colfrom pyspark.sql.types import LongTypecrm_retail_...
Working with databricks asset bundles (using the new python-based definition), if you have a job_cluster defined using driver_node_type_id, and then update it to no longer have it defined, but only node_type_id, the driver node_type never gets update...
There is no built-in way in Databricks Asset bundles or terraform to automatically inherit the value of driver_node_type_id for node_type_id."You must set both explicitly in your configuration"You can always see your updated detail resource from the ...
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 468 | |
| 349 | |
| 287 |
