Databricks Community

Hi @Sangeethagk, It looks like you’re encountering a couple of issues related to mlflow.pyfunc.spark_udf() and model dependencies. TypeError: ColSpec.init() got an unexpected keyword argument ‘required’: This error occurs when you’re using mlflo...

Hi @ttamas, Thank you for sharing your approach! It’s true that handling task dependencies and passing values between tasks in Databricks can sometimes be complex. Databricks now supports dynamic value references in notebooks. Instead of using dbutil...

Could you please try using SQL functions:  SELECT SUM(CAST(get_json_object(jsondatavar, '$.Value') AS DOUBLE)) AS total_value FROM table WHERE CAST(get_json_object(jsondatavar, '$.customer') AS INT) = 1000

Hi @Sweta, Your question about optimizing updates to PostgreSQL tables from a Spark structured streaming workflow is quite relevant, and I’m glad you’re exploring different approaches. Running updates directly on the main table in PostgreSQL using ...

In your scenario, if the data loaded on day 2 also includes all the data from day 1, you can still apply a "remove duplicates" logic. For instance, you could compute a hashdiff by hashing all the columns and use this to exclude rows you've already se...

Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command

Hi @Phani1, To access data from an S3 location in Azure Databricks, you have a few options: Using AWS Keys and Secret Scopes: Configure Spark properties to set your AWS keys stored in secret scopes as environment variables.Create a secret scope t...

Hi @Mahesh_Yadav, You can capture lineage data directly from Unity Catalog in Databricks without relying on system tables. Here’s how you can achieve this: Using Unity Catalog: Go to your Databricks landing page.Click “New” in the sidebar and sel...

Hello, if you want, you can deploy specific jobs only in the development environment. Since you have only two environments, a straightforward approach is to modify your jobs YAML definition as follows:resources: jobs: # Define the jobs to be de...