Hi @abueno ,
To replicate a SQL `not like '%var4%'` clause in the Dataframe API, you could use `rlike` with negation using `~` such as:
df.filter(~col('col4').rlike('var4')).display()
Here's a basic reproducible example:
df = (spark.range(10).withCol...
Hi @Sheeraj9191 ,
I believe the table you are looking for is `system.billing.usage` (docs: https://docs.databricks.com/en/admin/system-tables/billing.html#billable-usage-table-schema). This table contains information at the job level in field `usage...
Hi @Syleena23 ,
I believe this Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads provides a lot of answers to these questions and can be a great performance tuning and optimization guide in general. Please take a look.
Thank ...
Hi @acegerace ,
By default, table Tags are not included in the cloned metadata. This can be observed by cloning the table and confirming in the `information_schema.table_tags` system table. If this is not the desired behavior, the tags could be set o...
Hi @NavyaSinghvi ,Thank you for clarifying that. I believe I was able to replicate the issue you're describing and was able to circumvent it by adding `job.trigger.file_arrival.location` as a task-level parameter, and not as a job-level parameter.
C...
