Databricks Community

zed

I need to access the date of a given job running as a non-notebook Python job (spark_python_task). I want to pass a value from the cli when running it and being available to access the value in the scriptI tried the approaches in the attached image w...

zed

Hello Databricks Community,I am currently using the Feature Engineering client and have a few questions about best practices for writing to Feature Store Tables.I would like to know more about not using the write_table method directly from the featur...

zed

So, I think if I change the spark_python_task to a notebook_task but I keep the file as python file instead of notebook is ok. Now I can use the data bricks widgets easily and retrieve those parameters and I also put to version control python files i...

zed

I come with 1 more question. To clarify, my previous questions were focused on jobs that are triggered manually, without scheduling.For scheduled jobs—particularly those using Python script tasks—how can I configure the `job.yml` resource and the Pyt...

zed

Hi, thank you for your response. I have a few follow-up questions to clarify best practices when it comes to passing parameters with Python files:If I want to pass parameters, should I avoid using spark_python_task for Python scripts?In the context o...

zed

Thank you for your reply! I have one more question.Since write_table performs a MERGE when writing, does this mean it scans the entire landing feature table for updates during each write? For large tables, this can become a slow and resource-intensiv...

Databricks Community

User Stats

User Activity

Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Consequences of Not Using write_table with Feature Engineering Client and INSERT OVERWRITE

Re: Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Re: Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Re: Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Re: Consequences of Not Using write_table with Feature Engineering Client and INSERT OVERWRITE