- 3397 Views
- 1 replies
- 1 kudos
Unable to create databricks group and add permission via terraform
I have the following terraform code to create a databricks group and add permission to a workflow: resource "databricks_group" "dbx_group" { display_name = "ENV_MONITORING_TEAM" } resource "databricks_permissions" "workflow_permission" { job_id ...
- 3397 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @soumiknow , I did some digging internally and found something that may help: Based on the information gathered, I can now draft a comprehensive response to this Databricks Community question about the Terraform authentication issue. ## Dra...
- 1 kudos
- 2174 Views
- 2 replies
- 2 kudos
Resolved! How to make FOR cycle and dynamic SQL and variables work together
I am working on a testing notebook where the table that is tested can be given as a widget. I wanted to write it in SQL. The notebook does the following steps in a cycle that should run 10 times:1. Store the starting version of a delta table in a var...
- 2174 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you! I realised that the example I gave was bad. However, what I was missing is that I did not know how to set a variable in SQL scripting. Including the SET command within the sql string does not work, you have to use the EXECUTE IMMEDIATE ......
- 2 kudos
- 2655 Views
- 4 replies
- 1 kudos
Resolved! Problem in VS Code Extention
Until a few days ago, I was working with Databricks Connect using the VS Code extension, and everything worked perfectly.In my .databrickscfg file, I had authentication configured like this: [name]host:token: When I ran my code, everything worked fi...
- 2655 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @DatabricksEngi1 - Please ensure you have a Python Venv set up for each Python version that you use with Databricks Connect. Also, I have given step-by-step ways to debug the issue, clear the cache, etc [Read the files and instructions carefully b...
- 1 kudos
- 1100 Views
- 1 replies
- 1 kudos
Resolved! TransformWithState is not emitting for live streams
Hi Team, For one of my custom logics i went with transformwithState processor. However it is not working for live stream inputs., I have a start date filter on my df_base and when I give start date that is not current, the processor computes df_loss ...
- 1100 Views
- 1 replies
- 1 kudos
- 1 kudos
I managed to solve this. The issue was with how I handled the value state in the def init method. It was handled as a dataframe which caused the state to never materialize nor update therefore emitting nulls.I changed them to a tuple of values and th...
- 1 kudos
- 3545 Views
- 1 replies
- 0 kudos
Delta sharing view and cached data in DSFF
I've created a view with row level access based on CURRENT_RECIPIENT() function in the where clause. And I have 100s of clients as recipients that query this view.The problem is, when I modify this view CREATE OR REPLACE with a new sql code, and reci...
- 3545 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you tried something like this already? Force Cache Invalidation (Recommended) -- After CREATE OR REPLACE VIEW, execute: ALTER SHARE <share_name> REMOVE TABLE <schema>.<view_name>; ALTER SHARE <share_name> ADD TABLE <schema>.<view_name>; Thi...
- 0 kudos
- 3155 Views
- 1 replies
- 2 kudos
Resolved! Lakebridge code conversion | Permission issue
Hi,I’ve successfully installed the transpile module from Lakebridge and tried the tool to convert Informatica mappings into PySpark code. However, I’m encountering a PermissionError during execution. I’ve provided the relevant environment details and...
- 3155 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @abhijit007 - I see that this has been resolved in the 0.10.5 release. Can you please retest and confirm?
- 2 kudos
- 1213 Views
- 2 replies
- 3 kudos
Resolved! Results from the spark application to driver
I tried to read many articles but still not clear on this:The executors complete the execution of tasks and have the results with them.1. The results(output data) from all executors is transported to driver in all cases or executors persist it if tha...
- 1213 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @raghvendrarm1 , Below are the answers to your questions: Do executors always send “results” to the driver? No. Only actions that return values (e.g., collect, take, first, count) bring data back to the driver. collect explicitly “returns al...
- 3 kudos
- 2749 Views
- 3 replies
- 2 kudos
Resolved! Cannot import pyspark.pipelines module
Question: What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error? I'm using latest Free Edition of Databricks that has runtime version 17.2 and PySpark version 4.0.0.Error:ImportError: cannot im...
- 2749 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Saf4Databricks - Are you trying to use it from a standalone Databricks notebook? You should only use it from with Lakeflow Declarative Pipeline (LDP). The link you shared is about LDP. Here is an example where I used it.
- 2 kudos
- 2172 Views
- 4 replies
- 1 kudos
Resolved! Can't enable "variantType-preview" using DLTs
Using create_streaming_table and passing table properties as follows, I get an error running the pipeline for the first time:> Your table schema requires manually enablement of the following table feature(s): variantType-preview.I'm using this code:c...
- 2172 Views
- 4 replies
- 1 kudos
- 1 kudos
There's a workaround available in most situations which is to first create the table without the VARIANT column, run the pipeline at least once, and then add the column in a subsequent refresh.
- 1 kudos
- 2796 Views
- 3 replies
- 2 kudos
Resolved! Liquid Clustering With Merge
Hello I’m facing severe performance issues with a merge into databricksmerge_condition = """ source.data_hierarchy = target.data_hierarchy AND source.sensor_id = target.sensor_id AND source.timestamp = target.timestamp """The target Delt...
- 2796 Views
- 3 replies
- 2 kudos
- 5562 Views
- 8 replies
- 4 kudos
DLT pipeline fails with “can not infer schema from empty dataset” — works fine when run manually
Hi everyone,I’m running into an issue with a Delta Live Tables (DLT) pipeline that processes a few transformation layers (raw → intermediate → primary → feature).When I trigger the entire pipeline, it fails with the following error:can not infer sche...
- 5562 Views
- 8 replies
- 4 kudos
- 4 kudos
@databricksero Explicit Schema Definition: When calling spark.createDataFrame(pdf_cleaned), explicitly provide the schema even if the DataFrame is empty. This helps Spark infer the types and prevents the “cannot infer schema from empty dataset” erro...
- 4 kudos
- 12447 Views
- 1 replies
- 0 kudos
py4j.protocol.Py4JJavaError: An error occurred while calling o359.sql. : java.util.NoSuchElementExce
Hi!We are creating table in streaming job every micro-batch using spark.sql('create or replace table ... using delta as ...') command. This query includes combining data from multiple tables.Sometimes our job fails with error:py4j.Py4JException: An e...
- 12447 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @deng_dev ,Did you discover any way to raise this error gracefully? I'm facing the same error when running the kinesis stream. Although I'm aware of what the error is but my intent is to raise and log the error gracefully
- 0 kudos
- 1332 Views
- 1 replies
- 1 kudos
Resolved! Learning Path for Spark Developer Associate
Hello Everyone,Happy for being a part of Virtual Journey !!Enrolled in Associate Spark Developer and completed learning path in Databricks Academy. Can anyone please confirm is completing learning path enough for obtaining 50% off voucher for certifi...
- 1332 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Bhavana_Y! To be eligible for the incentives, you’ll need to complete one of the pathways mentioned in the Learning Festival post. Based on your screenshot, it looks like you’ve completed all four modules of LEARNING PATHWAY 7: APACHE SPARK DE...
- 1 kudos
- 1968 Views
- 4 replies
- 3 kudos
Resolved! deadlock occurs with use statement
When issuing a query from Informatica using a Delta connection, the statement use catalog_name.schema_name is executed first. At that time, the following error appeared in the query history:Query could not be scheduled: (conn=5073499)Deadlock found w...
- 1968 Views
- 4 replies
- 3 kudos
- 3 kudos
I’ll try making adjustments on the Informatica side.Thank you for your help.
- 3 kudos
- 3912 Views
- 8 replies
- 7 kudos
Resolved! DAB + DLT destroy fails due to ownership/permissions mismatch
Hi all,We are running into an issue with Databricks Asset Bundles (DAB) when trying to destroy a DLT pipeline. Setup is as follows:Two separate service principals:Deployment SP: used by Azure DevOps for deploying bundles.Run_as SP: used for running t...
- 3912 Views
- 8 replies
- 7 kudos
- 7 kudos
We just released https://github.com/databricks/cli/releases/tag/v0.273.0 with a mitigation for this, the error should disappear if you upgrade. Please try and let us know how it goes. Terraform fix is in https://github.com/databricks/terraform-provid...
- 7 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
