@holunder42 , I did some digging and here is what I found. Hopefully, it will help you further troubleshoot the issue.
Here’s what’s going on and how to solve it.
Why display behaves differently in a module
On Databricks, display is a notebook-scoped helper injected by the runtime. It lives in the notebook’s IPython namespace, not in your module’s global namespace.
When you write this inside a module:
# foo.py
def df_show(df):
display(df)
and then do:
from foo import df_show
df_show(df)
Python tries to resolve display in this order:
-
Local scope inside df_show
-
The module’s global scope (foo’s globals)
-
Builtins
It does not look in the notebook’s globals, where Databricks has put display. So from the module’s point of view, display just doesn’t exist. The “plain repr output” you’re seeing is effectively the fallback behavior (for example print or df.show() in your wrapper), not the Databricks visual display.
To get notebook-style visuals from code in a module, you need to inject the display function into the module code, or otherwise abstract it.
Recommended pattern: pass display as a dependency
Treat display as a UI dependency that you pass in from the notebook when you have it, and fall back to a simple print/show when you don’t.
foo.py (module):
# foo.py
def df_show(df, visualizer=None):
"""
visualizer: a callable like Databricks `display` or IPython.display.display.
If None, fall back to a simple text representation.
"""
if visualizer is not None:
visualizer(df)
else:
# Fallback for non-Databricks environments
try:
# Nice tabular output in many console contexts
df.show()
except AttributeError:
# Generic fallback
print(df)
On Databricks notebook:
from foo import df_show
df = spark.createDataFrame([{"x": 1}])
# Pass the notebook's `display` into your module
df_show(df, display)
Outside Databricks (plain Python / script):
from foo import df_show
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([{"x": 1}])
# No Databricks display available → falls back to df.show()
df_show(df)
This gives you:
Optional: central “smart display” wrapper
If you want to centralize this logic because you call it many times, define a “smart display” once and pass it around:
# foo.py
def make_smart_display(visualizer=None):
def _smart(obj):
if visualizer is not None:
visualizer(obj)
else:
try:
obj.show()
except AttributeError:
print(obj)
return _smart
def df_show(df, smart_display):
smart_display(df)
Usage on Databricks:
from foo import make_smart_display, df_show
smart_display = make_smart_display(display)
df_show(df, smart_display)
Usage off Databricks:
from foo import make_smart_display, df_show
smart_display = make_smart_display() # no visualizer → fallback
df_show(df, smart_display)
Design takeaway
The core idea is: keep UI concerns (like Databricks display) at the notebook boundary and inject them into reusable modules, rather than hard-coding display inside your modules. That way your code works cleanly both on Databricks and in any plain Python environment.
Cheers, Lou
