In my situation we cannot split this notebook as ADF pipeline is already in PROD ,I have tried to use the option %%capture .It helps to ran the notebook within size limits but somehow it is corrupting the output. Also checked in the Databricks AI and...
I have faced the issue that notebook is throwing error memory output exceeds 30 MB and its getting failed(Error:run failed with error message The size of the run output (31467502 bytes) exceeds the limit (31457280 bytes).. I have then use this option...
