Hello @AChang,

This is a common issue when the memory requirements of your model exceed the available memory on your current compute resources.

• Moving to a larger compute instance with more memory can help accommodate the memory requirements of your model. This is often the simplest solution if you have the resources available.

• As indicated in the logs, setting the environment variable TF_ENABLE_ONEDNN_OPTS=0 can disable oneDNN custom operations, which might help in some cases

• Ensure that there are no memory leaks in your code. This can be done by monitoring memory usage over time and ensuring that it does not continuously increase