Hello @victorNilsson 

I have tried to replicate this issue on my end, but unfortunately was unsuccessful as it worked the first time for me. I have, however, still tried to search for a solution.

I believe the issue you are getting could be linked to the sequencing of events. Databricks may not have finished flushing your csv through to the underlying database before it attempts to load it using Polars. Manually running the commands in two separate cells enforces the sequence, hence you don't get the same issue. To get around this, you can try to force the sync to finish before proceeding by using the following code:

import polars as pl
import os

data = {"a": [1, 2], "b": [3, 4]}

csv_file = "test.csv"

with open(csv_file, "w", encoding="utf-8") as f:
    f.writelines(["a,b\n", "hello,world\n"])
    f.flush()
    os.fsync(f.fileno())

df = pl.read_csv(csv_file)

display(df)

From my understanding, the f.flush() function forces the data to move from python, into the systems OS. After that the os.fsync(f.fileno()) ensures the data is actually written to disk and not just kept in RAM. 

An alternative solution, could be to implement a delay by simply using "import time" and "time.sleep(1)", however this is not as robust.

As I mentioned I was not able to replicate your issue so unfortunately cannot test their effectiveness, however they have both successfully ran in my environment (using a serverless cluster). I hope this helps - Please let me know how you get on.

Regards - Pilsner