Databricks Community

zed

I am using the Feature Engineering client when writing to a time series feature table. Then I have cried two data bricks jobs with the below code. I am running with different run_dates (e.g. '2016-01-07' and '2016-01-08'). When they run concurrently,...

zed · 11-05-2024

I need to access the date of a given job running as a non-notebook Python job (spark_python_task). I want to pass a value from the cli when running it and being available to access the value in the scriptI tried the approaches in the attached image w...

zed · 10-24-2024

Hello Databricks Community,I am currently using the Feature Engineering client and have a few questions about best practices for writing to Feature Store Tables.I would like to know more about not using the write_table method directly from the featur...

zed

Thank you for your response. Indeed that a retry approach can solve this issueI noticed that if I create the table by clustering with USING DELTA CLUSTER BY the column in the below code snippet the ConcurrentAppendException does not happen. Moreover,...

zed

Still no response on this thread...

zed

Hi Walter, You code snippet just adds the the mode argument to write_table, which is the default value, also used by my code. The error I get is when doing that merge.fe.write_table( name="dbc_mlops_premium.taxi_example.trip_pickup_features", ...

zed · 11-05-2024

So, I think if I change the spark_python_task to a notebook_task but I keep the file as python file instead of notebook is ok. Now I can use the data bricks widgets easily and retrieve those parameters and I also put to version control python files i...

zed · 11-05-2024

I come with 1 more question. To clarify, my previous questions were focused on jobs that are triggered manually, without scheduling.For scheduled jobs—particularly those using Python script tasks—how can I configure the `job.yml` resource and the Pyt...

Databricks Community

User Stats

User Activity

ConcurrentAppendException in Feature Engineering write_table

Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Consequences of Not Using write_table with Feature Engineering Client and INSERT OVERWRITE

Re: ConcurrentAppendException in Feature Engineering write_table

Re: ConcurrentAppendException in Feature Engineering write_table

Re: ConcurrentAppendException in Feature Engineering write_table

Re: Can't pass dynamic parameters to non-notebook Python job (spark_python_task)

Re: Can't pass dynamic parameters to non-notebook Python job (spark_python_task)