- 13903 Views
- 3 replies
- 3 kudos
Resolved! How to pass parameters to a "Job as Task" from code?
Hi,I would like to use the new "Job as Task" feature but Im having trouble to pass values.ScenarioI have a workflow job which contains 2 tasks.Task_A (type "Notebook"): Read data from a table and based on the contents decide, whether the workflow in ...
- 13903 Views
- 3 replies
- 3 kudos
- 3 kudos
I found the following information: value is the value for this task value’s key. This command must be able to represent the value internally in JSON format. The size of the JSON representation of the value cannot exceed 48 KiB.You can refer to https...
- 3 kudos
- 3003 Views
- 2 replies
- 0 kudos
Find the size of delta table for each month before partition
We have 38 delta tables. We decided to do partition the delta tables for each month.But we have some small tables as well. So we need find the size of delta tables for each month. So that we can use either partition or Z-orderIs there a way to find t...
- 3003 Views
- 2 replies
- 0 kudos
- 0 kudos
For your tables, I’m curious if you could utilize Liquid Clustering to reduce some of the maintenance issues relating to choosing Z-Order vs. partitioning. Saying this, one potential way is to read the Delta transaction log and read the Add Info st...
- 0 kudos
- 16180 Views
- 2 replies
- 4 kudos
Databricks Auto-Loader vs. Delta Live Tables
What is the difference between Databricks Auto-Loader and Delta Live Tables? Both seem to manage ETL for you but I'm confused on where to use one vs. the other.
- 16180 Views
- 2 replies
- 4 kudos
- 4 kudos
You say "...__would__ be a piece..." and "...DLT __would__ pick up...".Is DLT built upon AL?
- 4 kudos
- 4832 Views
- 1 replies
- 0 kudos
Resolved! Error java.lang.NullPointerException using Autoloader
Hi!I am pulling data from a Blob storage to Databrick using Autoloader. This process is working well for almost 10 resources, but for a specific one I am getting this error java.lang.NullPointerException.Looks like this issue in when I connect to th...
- 4832 Views
- 1 replies
- 0 kudos
- 0 kudos
@Maxi1693 - The value for the schemaEvolutionMode should be a string. could you please try changing the below from .option("cloudFiles.schemaEvolutionMode", None) to .option("cloudFiles.schemaEvolutionMode", "none") and let us know. Refe...
- 0 kudos
- 4803 Views
- 5 replies
- 1 kudos
Logs not coming up in the UI - while being written to DBFS
I have a few spark-submit jobs that are being run via Databricks workflows. I have configured logging in DBFS and specified a location in my GCS bucket.The logs are present in that GCS bucket for the latest run but whenever I try to view them from th...
- 4803 Views
- 5 replies
- 1 kudos
- 1 kudos
Yes, I meant to set it to None. Is the issue specific to any particular cluster? Or do you see the issue with all the clusters in your workspace?
- 1 kudos
- 6421 Views
- 2 replies
- 1 kudos
Error Running Delta Live Pipeline.
Hi Guys, I am new to the Delta pipeline. I have created a pipeline and now when i try to run the pipeline i get the error message "PERMISSION_DENIED: You are not authorized to create clusters. Please contact your administrator" even though I can crea...
- 6421 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for responding @Palash01 . thanks for giving me the direction so to get around it i had to get permission to "unrestricted cluster creation".
- 1 kudos
- 1759 Views
- 1 replies
- 0 kudos
Cluster types pricing
Hy guys,How can I get the pricing of cluster types (standard_D*, standard_E*, standart_F*, etc.) ?Im doing a study to decrease the price of my actual cluster.Have any idea ?Thank you, thank you
- 1759 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey, you can use the pricing calculator here: https://www.databricks.com/product/pricing/product-pricing/instance-types
- 0 kudos
- 10560 Views
- 1 replies
- 0 kudos
Resolved! Move a folder from Workspace to DBFS
So, I didn't quite set up my model training output directory correctly, and it saved all my model files to the workspace in the git repo I was working in. I am trying to move these files to DBFS, but when I try using dbutils.fs.mv, I get this error: ...
- 10560 Views
- 1 replies
- 0 kudos
- 0 kudos
Figured it out, just had to use the !cp command, here is what I did, worked perfectly.!cp -r /Workspace/Repos/$RESTOFPATH /dbfs/folder and it put the entire folder i was trying to move, into that dbfs folder.
- 0 kudos
- 6295 Views
- 4 replies
- 1 kudos
FiscalYear Start Period Is not Correct
Hi, I'm trying to create a calendar dimension including a fiscal year with a fiscal start of April 1. I'm using the fiscalyear library and am setting the start to month 4 but it insists on setting April to month 7.runtime 12.1My code snipet is:start_...
- 6295 Views
- 4 replies
- 1 kudos
- 1 kudos
import fiscalyear import datetime def get_fiscal_date(year,month,day): fiscalyear.setup_fiscal_calendar(start_month=4) v_fiscal_month=fiscalyear.FiscalDateTime(year, month, day).fiscal_month #To get the Fiscal Month v_fiscal_quarter=fiscalyea...
- 1 kudos
- 3394 Views
- 1 replies
- 1 kudos
Default delta log retention interval is different than in documentation?
It notes in the documentation here that the default delta log retention interval is 30 days - however when I create checkpoints in the delta log to trigger the cleanup - historical records from 30 days aren't removed; i.e. current day checkpoint is a...
- 3394 Views
- 1 replies
- 1 kudos
- 1 kudos
you need to set SET TBLPROPERTIES ('delta.checkpointRetentionDuration' = '30 days',)
- 1 kudos
- 17111 Views
- 4 replies
- 4 kudos
Resolved! Insert or merge into a table with GENERATED IDENTITY
Hi,When I create an identity column using the GENERATED ALWAYS AS IDENTITY statement and I try to INSERT or MERGE data into that table I keep getting the following error message:Cannot write to 'table', not enough data columns; target table has x col...
- 17111 Views
- 4 replies
- 4 kudos
- 4 kudos
You can run the INSERT by passing the subset of columns you want to provide values for... for example your insert statement would be something like:INSERT INTO target_table_with_identity_col(<list-of-cols-names-without-the-identity-column>SELECT(<lis...
- 4 kudos
- 4561 Views
- 2 replies
- 0 kudos
Structured Streaming Auto Loader UnknownFieldsException and Workflow Retries
Hi. I am using structured streaming and auto loader to read json files, and it is automated by Workflow. I am having difficulties with the job failing as schema changes are detected, but not retrying. Hopefully someone can point me in the right dir...
- 4561 Views
- 2 replies
- 0 kudos
- 0 kudos
Another point I have realised, is that the task and the parent notebook (which then calls the child notebook that runs the auto loader part) does not fail if the schema-changed failure occurs during the auto loader process. It's the child notebook a...
- 0 kudos
- 49921 Views
- 3 replies
- 3 kudos
Copilot Databricks integration
Given Copilot has now been released as a paid for product. Do we have a timeline when it will be integrated into Databricks?Our team are using VScode alot for Copilot and we think it would be super awesome to have it on our Databricks environment. Ou...
- 49921 Views
- 3 replies
- 3 kudos
- 3 kudos
@Vartika no josephk didn't answer Aidan's question. It's about comparing copilot with databricks assistant and can copilot be used in databricks workspace?
- 3 kudos
- 23433 Views
- 12 replies
- 9 kudos
PyPI library sometimes doesn't install during workflow execution
I have a workflow that is running upon a job cluster and contains a task that requires prophet library from PyPI:{ "task_key": "my_task", "depends_on": [ { "task_key": "<...>...
- 23433 Views
- 12 replies
- 9 kudos
- 9 kudos
Hey @Eugene Bikkinin​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 9 kudos
- 3652 Views
- 1 replies
- 0 kudos
How to add a Workflow File Arrival trigger on a file in a Unity Catalog Volume in Azure Databricks
I have a UC volume wil XLSX files, and would like to run a workflow when a new file arrives in the Volume.I was thinking of a workflow file arrival trigger.But that does not work when I add the physical ADLS location of the root folder:External locat...
- 3652 Views
- 1 replies
- 0 kudos
- 0 kudos
Worked it out with Microsoft.-> only works with external volumes, not managed.https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
