As a update: 
Querying with duckdb using tokens in databricks is actually possible.

What was needed:

SET azure_transport_option_type = 'curl';
in duckdb.

Afterwards querying duckdb worked seamlessly. 

I've written about it and also added an example: https://www.codecentric.de/wissens-hub/blog/access-databricks-unitycatalog-from-duckdb