- 3429 Views
- 2 replies
- 0 kudos
Obtain the source table version number from checkpoint file when using Structured Streaming
Hello!I'm using Structured Streaming to write to a delta table. The source is another delta table written with Structured Streaming as well. In order to datacheck the results I'm attempting to obtain from the checkpoint files of the target table the ...
- 3429 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Retired_mod, thank you for your answer.I'm a bit confused here because you seem to be describing the opposite behavior of what I've seen in our checkpoint files.Here I repost my examples to try to understand better.First checkpoint file:{"sour...
- 0 kudos
- 3961 Views
- 5 replies
- 9 kudos
Databricks Cluster Web terminal different permissions with tmux and xterm.
I am launching web terminal on my databricks cluster and when I am using the ephemeral xterm instance I am easily able to navigate to desired directory in `Workspace` and run anything... for example `ls ./` When I switch to tmux so that I can preserv...
- 3961 Views
- 5 replies
- 9 kudos
- 9 kudos
Hey there, fellow data explorer pals! I totally get your excitement when launching that web terminal on your Databricks cluster and feeling the power of running commands like 'ls ./' in the ephemeral xterm instance. It's like traversing the vast univ...
- 9 kudos
- 759 Views
- 2 replies
- 2 kudos
Resolved! alter DLT Materialized View alter column set MASK
I am trying to mask a column on a DLT materialized view - this is created using DLT syntax. I am not able set the column masking after creation. Appreciate any workaround.alter DLT Materialized View alter column set MASK
- 759 Views
- 2 replies
- 2 kudos
- 21949 Views
- 15 replies
- 12 kudos
Resolved! How to query delta lake using SQL desktop tools like SSMS or DBVisualizer
Is there a way to use sql desktop tools? because delta OSS or databricks does not provide desktop client (similar to azure data studio) to browse and query delta lake objects.I currently use databricks SQL , a webUI in the databricks workspace but se...
- 21949 Views
- 15 replies
- 12 kudos
- 12 kudos
DSR is Delta Standalone Reader. see more here - https://docs.delta.io/latest/delta-standalone.htmlIts a crate (and also now a py library) that allows you to connect to delta tables without using spark (e.g. directly from python and not using pyspa...
- 12 kudos
- 532 Views
- 2 replies
- 0 kudos
Upload of .bin file >400mb
I try to upload to local workspace folder with .bin extension.It is required to have it locally.I tried load from DBFS, but loading files over 265mb is not allowed with cluster. I tried to upload manually but failed with same error "OSError: [Errno5]...
- 532 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello Slash,Thank you for your response. I'm encountering the same issue as described. I tried running the provided code in my Databricks workspace, but I received an error. My question is how the script is expected to access local files, especially ...
- 0 kudos
- 989 Views
- 4 replies
- 0 kudos
Resolved! Discrepancies in Data Engineering GitHub Repositories and Errors in Databricks Notebooks
Hi team,I am writing to express my concerns regarding the recent Databricks webinar on "Data Intelligence with Databricks". During the session, there was mention of two GitHub repositories associated with the notebooks being used as mentioned below.1...
- 989 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Meghana_Vasavad could you please file a ticket with Databricks SupportThey will help you with this request.
- 0 kudos
- 2987 Views
- 5 replies
- 0 kudos
Resolved! Error deploying a DAB
I followed steps listed in this article.After creating and validation of bundle with default template, during deployment using this command:databricks bundle deploy -t dev --profile zzI get this message:Building mySecPrj...Error: build failed mySecPr...
- 2987 Views
- 5 replies
- 0 kudos
- 0 kudos
@daniel_sahal I am receiving the same error and tried your solution and am still getting the invalid command bdist_wheel. Any other suggestions? Thanks
- 0 kudos
- 14108 Views
- 3 replies
- 0 kudos
- 14108 Views
- 3 replies
- 0 kudos
- 0 kudos
I've also run into the same issue, customised docker image does not give DATABRICKS_RUNTIME_VERSION as env. I believe there are still many issues in how customised docker image is used in databricks cluster.Can anyone from databricks help answer it?
- 0 kudos
- 1029 Views
- 6 replies
- 0 kudos
Databricks UC enabled but Lineage not found for one table
Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view fo...
- 1029 Views
- 6 replies
- 0 kudos
- 0 kudos
It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.Lineage is captu...
- 0 kudos
- 3097 Views
- 2 replies
- 3 kudos
Resolved! How to access the job-Scheduling Date from within the notebook?
I have created a job that contains a notebook that reads a file from Azure Storage. The file-name contains the date of when the file was transferred to the storage. A new file arrives every Monday, and the read-job is scheduled to run every Monday. I...
- 3097 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi, I guess the files are in the same directory structure so that you can use cloud files autoloader. It will incrementally read only new files https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loaderSo it will ...
- 3 kudos
- 622 Views
- 1 replies
- 1 kudos
Resolved! Workflow file watch - capture filename trigger
With respect to the file watch trigger in workflows, how can we capture what files and or path was identified as raising the trigger? I'd like to use this information to set parameters based upon the file name and the file path Thank you! https://...
- 622 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @csmcpherson ,This is currently not supported, but databricks team is working on that idea according to below thread:Solved: File information is not passed to trigger job on f... - Databricks Community - 39266As a workaround, if you use autoloader...
- 1 kudos
- 2179 Views
- 1 replies
- 0 kudos
Interview question for ADF
Hello,I am trying to attending interviews for Data Engineer i have 3 years Experince I am looking realtime interview question ,If any one have could you please share.Thank you,Govardhana
- 2179 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Govardhana ,There are plenty of those question on the internet. Below, is the one that's actually quite good:- Top 40+ Azure Data Factory Interview Questions 2024 (k21academy.com)
- 0 kudos
- 3868 Views
- 3 replies
- 1 kudos
Resolved! DLT Event Logs
Hi,Does anyone know what details:planning_information:technique_information[0]:cost under planning_information event type means in my DLT workflow system event logs? For context, I'm trying to track the cost per run of my DLT workflow and I do not ha...
- 3868 Views
- 3 replies
- 1 kudos
- 1 kudos
you can enable system.billing schema and see the costs of the runs from the usage table.
- 1 kudos
- 1221 Views
- 3 replies
- 0 kudos
Error: Cannot use legacy parameters because the job has job parameters configured.
I created a job which has two Job Parameters. How can I use Databricks CLI to pass different values to those parameters.
- 1221 Views
- 3 replies
- 0 kudos
- 721 Views
- 2 replies
- 0 kudos
Reading single file from Databricks DBFS
I have a Test.csv file in FileStore of DBFS in Databricks Community edition. When I try to read the file using With Open, I get the following error:FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv' import os wi...
- 721 Views
- 2 replies
- 0 kudos
- 0 kudos
@EricRM It should work. Please see the accepted response from this same forum here. So, we still need to find a cause of the error. Following is the detailed error message. Maybe, this will help readers understand the issue better and help it resolve...
- 0 kudos