- 4483 Views
- 1 replies
- 1 kudos
AWS Databricks and Fabric OneLake integration
Bit of a weird scenario and I wanted to hear from the experts in this communityLets say I have a Fabric Lakehouse (OneLake) and I want to read that data into Databricks (AWS) Unity Catalog to play with that data. What is the recommended mechanism to ...
- 4483 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @de2298, Currently, Microsoft Fabric does not offer a built-in connector that allows direct querying or exposure of Delta Share tables from AWS Databricks into a Fabric Warehouse. The Unity Catalog mirroring feature is supported only with Azure Da...
- 1 kudos
- 4933 Views
- 4 replies
- 0 kudos
Resolved! Declarative Pipelines with datacontracts
I'm wondering if anyone has successfully integrated data contracts with declarative pipelines in Databricks. Specifically, I want to reuse the quality checks and schema definitions from the contract directly within the pipeline's stages. I haven't fo...
- 4933 Views
- 4 replies
- 0 kudos
- 0 kudos
Suggested Steps:Define the data contractCreate a YAML/JSON file containing:Schema (column names, data types, required fields)Data quality rules (null checks, ranges, regex patterns, allowed value lists)Governance metadata (e.g., data sensitivity, LGP...
- 0 kudos
- 5120 Views
- 7 replies
- 2 kudos
Issue: NoSuchMethodError in Spark Job While Upgrading to Databricks 15.5 LTS
Problem DescriptionI am attempting to upgrade my application from Databricks runtime version 12.2 LTS to 15.5 LTS. During this upgrade, my Spark job fails with the following error:java.lang.NoSuchMethodError: org.apache.spark.scheduler.SparkListenerA...
- 5120 Views
- 7 replies
- 2 kudos
- 2 kudos
@DBonomo @sahil_s_jain We can write separate getSchema method inside the BulkCopyUtils.scala file and call that method instead of referring it from spark. You can add the below function in the BulkCopyUtils.scala file and build it locally. You can th...
- 2 kudos
- 2462 Views
- 3 replies
- 3 kudos
Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster
Use Case Description:Manually uploading orders data into databricks Then moving it into a bronze layer using the below code technical Code used python Getting error:The Delta Live Tables (DLT) module is not supported on this cluster. You should eith...
- 2462 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @ManojkMohan ,If you try to run a DLT pipeline on an (e.g.) all-purpose compute cluster, it will fail. DLT pipelines require a DLT job computer cluster.To run it, create a new pipeline in the Databricks UI, assign your DLT notebook to it, and s...
- 3 kudos
- 4162 Views
- 4 replies
- 2 kudos
How to search an array of words in a text field
Example:TABLE 1FIELD_TEXTI like salty food and Italian foodI have Italian foodbread, rice and beansmexican foodscoke, spritearray['italia', 'mex','coke']match TABLE1 X ARRAYResults:I like salty food and Italian foodI have Italian foodmexican foodsis ...
- 4162 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes, it’s possible to search an array of words in a text field using SQL with LIKE clauses or regex functions, while PySpark provides higher scalability with functions like rlike and array_contains (Wikipedia explains that SQL is a domain-specific la...
- 2 kudos
- 1365 Views
- 1 replies
- 1 kudos
Resolved! Enable Classic (Non-Serverless) Clusters on Free Trial
Hi Databricks community,I’m using a Free Trial cloud account. Currently and I need to create classic clusters for Spark exercises.Is there a way to enable Standard/Classic Clusters in a trial workspace, or any workaround for Free Trial users?Any guid...
- 1365 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @SCPablo ,If you are referring to the 14 days free trial account (link: https://docs.databricks.com/aws/en/getting-started/free-trial) , you can create compute clusters and experiment with them. But if you are referring to the Databricks Free E...
- 1 kudos
- 5423 Views
- 2 replies
- 3 kudos
Resolved! Performance Issues with Writing Large DataFrames to Managed Tables in Databricks (3.5B+ Rows)
Hi Community,I'm working on a large-scale data processing job in Databricks and facing performance and stability issues during the write operations. Here's a detailed breakdown of my use case and environment:Use Case Overview:Primary Data Frames:Firs...
- 5423 Views
- 2 replies
- 3 kudos
- 3 kudos
I found the solution, Please refer to the below links for the solutionLinkedIn Post: https://www.linkedin.com/posts/activity-7363497408925745154-LaaL?utm_source=share&utm_medium=member_desktop&rcm=ACoAACTtno0BU78QJcWz-X3GHtKRvhXxf5fod90Medium Blob: h...
- 3 kudos
- 5078 Views
- 7 replies
- 1 kudos
Resolved! Lakebridge Transpile paths
Hi there,What kind of source and target paths can I use in Transpile command?I'm trying to run command:databricks labs lakebridge transpile --source-dialect tsql --input-source and I get error:ERROR [src/databricks/labs/lakebridge.transpile] ValueErr...
- 5078 Views
- 7 replies
- 1 kudos
- 1 kudos
Also, what cloud source locations are available for Transpile- dbfs, Unity Catalog volumes, etc?
- 1 kudos
- 1094 Views
- 1 replies
- 0 kudos
notebook_params deprecation
In Databricks documentation for workflow / Jobs /REST API 2.1 / Trigger a new job run - some fields in the request body are stated as depreciated, for example notebook_params, jar_params. In Jobs REST API 2.2 those fields do not appear at all.Ho...
- 1094 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Keren! Fields like notebook_params and jar_params are deprecated in Jobs REST API 2.1 and no longer appear in 2.2. While an end-of-life date has not been announced yet, it’s recommended to transition to the newer parameter fields rather than r...
- 0 kudos
- 935 Views
- 1 replies
- 0 kudos
Two developer working on same same Genie space
My teammate and I are working on the same Genie space at the same time. We’re both adding instructions, and it seems like Genie is getting confused. In this situation, how does Genie decide which instructions to follow? Will it overwrite one set of i...
- 935 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @flashmav! For now, when multiple users work on the same Genie space at the same time, changes made by one person may overwrite another’s updates if both are editing the same instructions or example queries. To avoid conflicts and ensure your u...
- 0 kudos
- 1080 Views
- 1 replies
- 2 kudos
Resolved! Notebooks - Pasting images from clipboard
Hello, I am new to this community. I really miss the Jypyter-style feature to copy paste images from the clipboard into the markdown cells as I am work on a notebook. As you might know, when an image is pasted from clipboard, first that gets converte...
- 1080 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @rs-tcs! Thanks for sharing your feedback.Yes, direct pasting from the clipboard isn’t supported at the moment. Right now, Databricks Notebooks support adding images in markdown by linking to a file/URL or by dragging and dropping an image from...
- 2 kudos
- 4789 Views
- 8 replies
- 4 kudos
Resolved! databricks sql create function - input table name as parameter and returns complete table.
hi , I am trying to create a databricks sql unity catalog function which will take table_name as input parameter and returns the full table as output. I am getting error, kindly help CREATE OR REPLACE FUNCTION catalog.schema.get_table( table_name STR...
- 4789 Views
- 8 replies
- 4 kudos
- 4 kudos
@szymon_dybczak I admire that you always find the appropriate information in the documentation. I will try my best to emulate this behaviour with other posts .@saurabh_aher great workaround suggestion with a stored procedure. Lots of lessons learned ...
- 4 kudos
- 3494 Views
- 6 replies
- 2 kudos
Resolved! Databricks Asset Bundles: using loops
Hey,I am using DAB's to deploy the job below.This code works but I would like to use it for other suppliers as well.Is there a way to loop over a loop of suppliers: ['nike', 'adidas',...] and fill those variables so that config_nike_gsheet_to_databri...
- 3494 Views
- 6 replies
- 2 kudos
- 2 kudos
@Daan ,You can maintain a default template that holds the common configuration. While creating a job-specific configuration, you can safely merge your job-specific dictionary with the base template using the | operator.
- 2 kudos
- 4006 Views
- 3 replies
- 4 kudos
Access to For Each run ids from jobs rest API
Hello CommunityI'm using the for_each tasks in workflows, but I'm struggling to access to the job information through the job APIs.In short, using the runs api (Get a single job run | Jobs API | REST API reference | Databricks on AWS), I'm able to ac...
- 4006 Views
- 3 replies
- 4 kudos
- 4 kudos
This feature would be extremely helpful in monitoring each task in the `foreachtask` task.
- 4 kudos
- 2949 Views
- 6 replies
- 4 kudos
Resolved! DLT pipelines - sporadic ModuleNotFoundError
When we run DLT pipelines (which we deploy via DABs), we get a sporadic issue when attempting to install our bundle's wheel file.First, in every DLT pipeline, we as a first step a script that looks like the followingimport subprocess as sp from impor...
- 2949 Views
- 6 replies
- 4 kudos
- 4 kudos
If you're encountering intermittent ModuleNotFoundError when your DLT pipeline tries to install your asset bundle’s wheel file, this typically points to inconsistencies in how your dependencies are packaged or where they’re being deployed. Common cul...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
