- 2939 Views
- 3 replies
- 0 kudos
DLT Apply Changes problem
Hi All, I am working on DLT pipeline, to create SCD2 for my bronze layer, my architecture has 4 layers, namely Raw, Bronze, Silver, Gold. I am ingesting data directly into raw, and then I am creating history(SCD2) into bronze. My code: @Dlt.view(n...
- 2939 Views
- 3 replies
- 0 kudos
- 1621 Views
- 1 replies
- 1 kudos
Resolved! Automating Purging of All Notebook Revision
Hi everyone,We work with sensitive data in Databricks, so it's crucial from both security and regulatory perspectives to purge all data saved in notebook revisions.Currently, there are two manual methods:Delete all history from each notebook individu...
- 1621 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are some things to consider: Automating the purging of notebook revision history in Databricks is not currently a directly supported feature, and there are some challenges in achieving this: Available Methods:Currently, Databricks provides ma...
- 1 kudos
- 27906 Views
- 14 replies
- 5 kudos
UDF importing from other modules
Hi community,I am using a pyspark udf. The function is being imported from a repo (in the repos section) and registered as a UDF in a the notebook. I am getting a PythonException error when the transformation is run. This is comming from the databric...
- 27906 Views
- 14 replies
- 5 kudos
- 5 kudos
I just ran into and solved this issue. My problem was because in the python script that I loaded in as a module I defined the function that I planned to use as a udf separately from the function that I actually called in my script. I believe that bec...
- 5 kudos
- 87975 Views
- 6 replies
- 9 kudos
How/where can I see a list of my dbfs files?
When using the Community Edition, I'm trying to find a place in the UI where I can browse the files that I've uploaded to dbfs. How/where can I do that? When I try to view them from the Data sidebar I see nothing, yet I know they're there, as if I us...
- 87975 Views
- 6 replies
- 9 kudos
- 17040 Views
- 7 replies
- 8 kudos
Resolved! How can I trigger the execution of a specific step within a Databricks Workflow job?
I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?
- 17040 Views
- 7 replies
- 8 kudos
- 4710 Views
- 3 replies
- 0 kudos
DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] Error in Streaming Table
I am encountering a recurring issue while working with Delta streaming tables in my system. The error message is as follows: com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] gs://cimb-prod-lakehouse/b...
- 4710 Views
- 3 replies
- 0 kudos
- 0 kudos
The issue you're encountering with the error DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] is related to Delta Lake's retention policy for logs and checkpoints, which manages the lifecycle of transaction log files and checkpoint files...
- 0 kudos
- 4478 Views
- 2 replies
- 0 kudos
Databricks SQL Error outputting sesntive data to logs
Hi - I am using `from_json` with FAILFAST to correctly format some data using databricks SQL. However, this function can return the error "[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION] Malformed records are detected in record parsing" with the res...
- 4478 Views
- 2 replies
- 0 kudos
- 0 kudos
You could use mode (default PERMISSIVE allows a mode for dealing with corrupt records during parsing. PERMISSIVE: when it meets a corrupted record, puts the malformed string into a field configured by columnNameOfCorruptRecord, and sets malformed fie...
- 0 kudos
- 5552 Views
- 2 replies
- 0 kudos
Auto loader from tables in Delta Share
Hello,I am trying to read delta table in delta shares shared from other environments.The pipeline runs okay; however, as the delta table is update in the source (delta share in GCP), the code below gets error, unless if I reset the checkpoint. I wond...
- 5552 Views
- 2 replies
- 0 kudos
- 0 kudos
The error you are encountering—DeltaUnsupportedOperationException: [DELTA_SOURCE_TABLE_IGNORE_CHANGES]—occurs because your streaming job detects updates in the source Delta table, which is not supported for they type of source you have. Streaming tab...
- 0 kudos
- 6150 Views
- 2 replies
- 1 kudos
Performance issue when using structured streaming
Hi databricks community! Let me first apology for the long post.I'm implementing a system in databricks to read from a kafka stream into the bronze layer of a delta table. The idea is to do some operations on the data that is coming from kafka, mainl...
- 6150 Views
- 2 replies
- 1 kudos
- 1 kudos
Have you tried using minPartitions Minimum number of partitions to read from Kafka. You can configure Spark to use an arbitrary minimum of partitions to read from Kafka using the minPartitions option. Normally Spark has a 1-1 mapping of Kafka topicPa...
- 1 kudos
- 17563 Views
- 2 replies
- 5 kudos
- 17563 Views
- 2 replies
- 5 kudos
- 5 kudos
Instead of copy paste if someone can explain in practical terms with example, that would be upto the standard of DataBricks community
- 5 kudos
- 12169 Views
- 6 replies
- 5 kudos
Resolved! Unity Catalog - Azure account console - how to access?
I'm trying to access account console in Azure but I only can see the list of workspaces and access them. I didn't find documentation about account console for Azure. Do you know how to access account console?
- 12169 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello @Leszek​ . Please tell me is it works for you ?Did you find the root cause ?I still don't understand why I should grant to myself some extra permissions if I already global administrator, owner of subscription, owner of databricks workspace but...
- 5 kudos
- 2480 Views
- 3 replies
- 0 kudos
Databricks App with DAB
Hi All,I am trying to deploy a DBX APP via DAB, however source_code_path seems not to be parsed correctly to the app configuration.- dbx_dash/-- resources/---- app.yml-- src/---- app.yaml---- app.py-- databricks.ymlresources/app.yml:resources:apps: m...
- 2480 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Adriana, Have you adjusted the root_path in your databricks.yml Kindly add /Workspace and entire path to the root_path Thanks
- 0 kudos
- 4477 Views
- 1 replies
- 0 kudos
INVALID_HANDLE.SESSION_NOT_FOUND
We run several workflows and tasks parallel using serverless compute. In many different places of code we started to get errors as below. It looks like that when one task fails, every other that run at the same moment fails as well. After retry on on...
- 4477 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, The error INVALID_HANDLE.SESSION_NOT_FOUND https://docs.databricks.com/aws/en/error-messages/invalid-handle-error-class#session_not_foundis a handled error but the grpc errors are something where more improvements are being pushed in eve...
- 0 kudos
- 3863 Views
- 1 replies
- 0 kudos
spark.sql makes debugger freeze
I have just created a simple bundle with databricks, and is using Databricks connect to debug locally. This is my script:from pyspark.sql import SparkSession, DataFrame def get_taxis(spark: SparkSession) -> DataFrame: return spark.read.table("samp...
- 3863 Views
- 1 replies
- 0 kudos
- 0 kudos
Ensure that your Databricks Connect is properly set up and is using the correct version compatible with your cluster’s runtime. For VS Code, any mismatches between the installed databricks-connect Python package version and the cluster runtime could ...
- 0 kudos
- 5015 Views
- 1 replies
- 0 kudos
ODI 12C to Databricks equvivalent
Hello All,We are planning to convert from ODI12C to data bricks equivalentsWhat are the steps involved, what are the limitations in this case Thanks,Vasu Kumar T
- 5015 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, These blogs can help you get an idea on the migration planning and insights. https://www.databricks.com/blog/how-migrate-your-oracle-plsql-code-databricks-lakehouse-platform https://www.databricks.com/blog/databricks-migration-strategy-lessons-le...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
