the inference table table doesn't get updated

MohsenJ — Tue, 09 Apr 2024 12:59:08 GMT

I setup a model serving endpoint and created a monitoring dashboard to monitor its performance. The problem is my inference table doesn't get updated by model serving endpoints.

To test the endpoint I use the following code

import random import time API_ROOT = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get() API_TOKEN = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get() endpoint_name = config["model_serving_endpoint_name"] headers = {"Context-Type": "text/json", "Authorization": f"Bearer {API_TOKEN}"} all_items = df_full_data.select(col("item_id")).distinct() for user_id in range(100,150): print(user_id) items_not_rated_by_user = df_full_data.where(col("user_id")==user_id).select(col("item_id")).distinct()#collect()[0][0] no_rated_items = [item.item_id for item in all_items.subtract(items_not_rated_by_user).limit(4).collect()] data = { "dataframe_records": [ {"user_id":user_id, "item_id":no_rated_items[0], "rating": random.randint(1, 5)}, {"user_id":user_id, "item_id":no_rated_items[1], "rating": random.randint(1, 5)}, {"user_id":user_id, "item_id":no_rated_items[2], "rating": random.randint(1, 5)}, {"user_id":user_id, "item_id":no_rated_items[2], "rating": random.randint(1, 5)}, ] } response = requests.post( url=f"{API_ROOT}/serving-endpoints/{endpoint_name}/invocations", json=data, headers=headers ) print(response.json()) time.sleep(random.randint(60*1, 60*3))

The code worked fine and I could see my inference table filled in with new rows. I setup a monitoring pipeline in which it reads and unpack the result of this table to a new table with the right format for monitoring. For this I use the inference table as a streaming table. I adapted the code for this from the example here

Now I want to check if my unpacked table can automatically react to changes to my inference table whenever the endpoint receive a new request. For that I used the code above to call the endpoint with some new data. but I can't see any update in my inference table although I see from endpoint UI that it received and processed the requests.

here is how I created the endpoint

data = { "name": endpoint_name, "config": { "served_models": [ { "model_name": model_name, "model_version": int(model_version), "workload_size": workload_size, "scale_to_zero_enabled": scale_to_zero, "workload_type": workload_type, } ], "auto_capture_config":{ "catalog_name": catalog_name, "schema_name": model_schema, } }, } headers = {"Context-Type": "text/json", "Authorization": f"Bearer {API_TOKEN}"} response = requests.post( url=f"{API_ROOT}/api/2.0/serving-endpoints", json=data, headers=headers )

and here is the service log of the serving model

[5954fs284d] [2024-04-09 12:19:21 +0000] [7] [INFO] Starting gunicorn 21.2.0 [5954fs284d] [2024-04-09 12:19:21 +0000] [7] [INFO] Listening at: http://0.0.0.0:8080 (7) [5954fs284d] [2024-04-09 12:19:21 +0000] [7] [INFO] Using worker: sync [5954fs284d] [2024-04-09 12:19:21 +0000] [8] [INFO] Booting worker with pid: 8 [5954fs284d] [2024-04-09 12:19:21 +0000] [9] [INFO] Booting worker with pid: 9 [5954fs284d] [2024-04-09 12:19:21 +0000] [10] [INFO] Booting worker with pid: 10 [5954fs284d] [2024-04-09 12:19:21 +0000] [11] [INFO] Booting worker with pid: 11 [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for '5ffd2b27' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for '5ffd2b27' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for '5ffd2b27' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for '5ffd2b27' at 'null' in 'null' [5954fs284d] ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null' [5954fs284d] Setting default log level to "WARN". [5954fs284d] To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). [5954fs284d] Setting default log level to "WARN". [5954fs284d] To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). [5954fs284d] Setting default log level to "WARN". [5954fs284d] To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). [5954fs284d] Setting default log level to "WARN". [5954fs284d] To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). [5954fs284d] 24/04/09 12:19:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [5954fs284d] 24/04/09 12:19:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [5954fs284d] 24/04/09 12:19:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [5954fs284d] 24/04/09 12:19:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. [5954fs284d] 24/04/09 12:19:30 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043. [5954fs284d] 2024/04/09 12:19:33 INFO mlflow.spark: File '/model/sparkml' is already on DFS, copy is not necessary. [5954fs284d] 2024/04/09 12:19:33 INFO mlflow.spark: File '/model/sparkml' is already on DFS, copy is not necessary. [5954fs284d] 2024/04/09 12:19:33 INFO mlflow.spark: File '/model/sparkml' is already on DFS, copy is not necessary. [5954fs284d] 2024/04/09 12:19:33 INFO mlflow.spark: File '/model/sparkml' is already on DFS, copy is not necessary.

attached you see the history of the inference table.

any idea what could be the problem or how can I debugg the issue further?

Re: the inference table table doesn't get updated

Amruth_Ashok — Fri, 29 Aug 2025 09:36:05 GMT

If the inference table is renamed, schema changed, deleted, or if the creator loses data access/the permissions are missing or have recently changed (for the endpoint creator), inference tables may silently stop updating.

topic the inference table table doesn't get updated in Get Started Discussions

the inference table table doesn't get updated

Re: the inference table table doesn't get updated