- 758 Views
- 2 replies
- 1 kudos
Dynamic Parameter
I have a query I need to run with two parameters: Workflow and workflow steps. The dropdown list supplied by "Steps" should change based on the input of the "Workflow" dropdown.When I use the following code, it creates the "Steps" dropdown list based...
- 758 Views
- 2 replies
- 1 kudos
- 1 kudos
That's a shame it has to be re-instantiated each time! Thank you for letting me know that functionality is not currently supported.
- 1 kudos
- 1042 Views
- 2 replies
- 0 kudos
Loguru doesn't save logs to Databricks volume
I've added an external volume named "logs" to my Databricks Unity Catalog. Within a Databricks notebook I can verify that it exists (os.path.exists(path='/Volumes/my_catalog/schema_name/logs') and even write a file to it that I can see within the Dat...
- 1042 Views
- 2 replies
- 0 kudos
- 0 kudos
I am having the same problem. I am using a work-around currently but definitely would love to see a solution. FYI: here is my work-around:logger.add(f"{output_folder_path}/../logging/workflow_job1_{datetime_str}.log",rotation='10 days',retention="10 ...
- 0 kudos
- 2336 Views
- 4 replies
- 2 kudos
Resolved! Databricks JDBC Error while connecting from Datastage JDBC connector
I am reading data from Databricks in datatstage 11.7 on prem using datastage JDBC connector and getting the below error. I tried to limit the select queries to one row , it was able to read data form the source, JDBC_Connector_0: The connector encoun...
- 2336 Views
- 4 replies
- 2 kudos
- 2 kudos
Here are some suggestions, not sure if it fits with what you are doing but they are worth mentioning. The Databricks JDBC driver currently does not support batch updates, which is why your updates appear to process row by row with a batch size of 1...
- 2 kudos
- 2841 Views
- 4 replies
- 1 kudos
Resolved! Incremental Load on Materialized Views
Is incremental load possible on Materialized views. I would like to get some tutorials or videos on how to perform incremental refresh on MVs in gold layers. Also is it mandatory to have PKs for performing incremental loads in MVs.
- 2841 Views
- 4 replies
- 1 kudos
- 1252 Views
- 2 replies
- 1 kudos
Resolved! Working of @DLT.table
I am using @Dlt.table decorator to create a table which will store history for my tables.My code works like this:@Dlt.table(name="table name")def target: Custom spark code to create history Even though the spark code creates and prints history ...
- 1252 Views
- 2 replies
- 1 kudos
- 1 kudos
@_singh_vish DLT assumes the result of each @dlt.table decorator is the current state of the table at that point in time. So, when you define a DLT table using @dlt.table, whatever DataFrame is returned by that function will replace the previous data...
- 1 kudos
- 615 Views
- 1 replies
- 0 kudos
DLT job failed to parse timestamp string with T and Z.
Hi I am struggling with converting a timestamp string with T and Z to a timestamp column in my DLT job. here is the relevent code snippet:trans_rules={'timestamp_value', '''to_timestamp(timstamp_str, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")'''}In my DLT func...
- 615 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Thomas_Zhang , Good Day! Can you give it a try with the below code format and check if it helps you to set the timezone with Z : ======================= date_format( to_timestamp(`createdon`, 'yyyy-MM-dd\'T\'HH:mm:ss.SSSSSSSX'), 'yyyy-MM-dd\'T...
- 0 kudos
- 4085 Views
- 5 replies
- 2 kudos
MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS
Im facing an error when updating DBR from 13.3 LTS to 14.3LTSIm using the spark:mongo-spark-connector:10.2.1 and running the following script connectionString = ****** database = ***** collection = ***** spark = SparkSession \ .builder \ ...
- 4085 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @pmaferreira ,can you please help with which version you are using and is it supports ignoreNullValues option?
- 2 kudos
- 746 Views
- 2 replies
- 0 kudos
Resolved! Cluster Issues while assigning it to Notebook
Hi Team Databricks,I have tried to assign cluster to note book in 2 different community edition and its throwing error like:Notebook Detached: Exception when creating execution context: java.netSocketTimeOutException: Connect Timeout.The above error ...
- 746 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Nishant_Kumar25 & @kaleabgirma! There was a recent issue affecting Community Edition clusters, but it has now been mitigated. New cluster creation has been tested and is working as expected. If you're still encountering the error, please try r...
- 0 kudos
- 1743 Views
- 3 replies
- 0 kudos
DLT Apply Changes problem
Hi All, I am working on DLT pipeline, to create SCD2 for my bronze layer, my architecture has 4 layers, namely Raw, Bronze, Silver, Gold. I am ingesting data directly into raw, and then I am creating history(SCD2) into bronze. My code: @Dlt.view(n...
- 1743 Views
- 3 replies
- 0 kudos
- 916 Views
- 1 replies
- 1 kudos
Resolved! Automating Purging of All Notebook Revision
Hi everyone,We work with sensitive data in Databricks, so it's crucial from both security and regulatory perspectives to purge all data saved in notebook revisions.Currently, there are two manual methods:Delete all history from each notebook individu...
- 916 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are some things to consider: Automating the purging of notebook revision history in Databricks is not currently a directly supported feature, and there are some challenges in achieving this: Available Methods:Currently, Databricks provides ma...
- 1 kudos
- 22269 Views
- 14 replies
- 5 kudos
UDF importing from other modules
Hi community,I am using a pyspark udf. The function is being imported from a repo (in the repos section) and registered as a UDF in a the notebook. I am getting a PythonException error when the transformation is run. This is comming from the databric...
- 22269 Views
- 14 replies
- 5 kudos
- 5 kudos
I just ran into and solved this issue. My problem was because in the python script that I loaded in as a module I defined the function that I planned to use as a udf separately from the function that I actually called in my script. I believe that bec...
- 5 kudos
- 78519 Views
- 6 replies
- 9 kudos
How/where can I see a list of my dbfs files?
When using the Community Edition, I'm trying to find a place in the UI where I can browse the files that I've uploaded to dbfs. How/where can I do that? When I try to view them from the Data sidebar I see nothing, yet I know they're there, as if I us...
- 78519 Views
- 6 replies
- 9 kudos
- 14841 Views
- 7 replies
- 8 kudos
Resolved! How can I trigger the execution of a specific step within a Databricks Workflow job?
I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?
- 14841 Views
- 7 replies
- 8 kudos
- 3853 Views
- 3 replies
- 0 kudos
DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] Error in Streaming Table
I am encountering a recurring issue while working with Delta streaming tables in my system. The error message is as follows: com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] gs://cimb-prod-lakehouse/b...
- 3853 Views
- 3 replies
- 0 kudos
- 0 kudos
The issue you're encountering with the error DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] is related to Delta Lake's retention policy for logs and checkpoints, which manages the lifecycle of transaction log files and checkpoint files...
- 0 kudos
- 3741 Views
- 2 replies
- 0 kudos
Databricks SQL Error outputting sesntive data to logs
Hi - I am using `from_json` with FAILFAST to correctly format some data using databricks SQL. However, this function can return the error "[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION] Malformed records are detected in record parsing" with the res...
- 3741 Views
- 2 replies
- 0 kudos
- 0 kudos
You could use mode (default PERMISSIVE allows a mode for dealing with corrupt records during parsing. PERMISSIVE: when it meets a corrupted record, puts the malformed string into a field configured by columnNameOfCorruptRecord, and sets malformed fie...
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
