Databricks Community

emorgoch

I've got an ADLS Gen 2 container that has cloud files that I want to ingest with AutoLoader. The files are saved into the root of the container.I created an external location for that container to be able to access it through managed credentials. How...

emorgoch

My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...

emorgoch · 06-16-2026

We're in the process of converting over our Databricks notebooks from .py file to .ipynb. We have disabled storing notebook output in source control at the workspace level.However, what we're discovering is that every cell in our notebooks has 3 time...

emorgoch · 08-01-2024

I've got a notebook that I've written that's going to execute some python code to parse the workspace id to figure out which of my environments that I'm in and set a value for it. I then want to take that value, and pass it through to a code block of...

emorgoch

Thanks @GabFernandes, It wasn't intended to co-habitate the managed resources with the ingestion files, just an error in the setup that lead to it. I think the 7 day timeframe expires tomorrow, so hopefully that will resolve it, else I will get a hol...

emorgoch

Thanks Abhilash,Unfortunately it is an external system that we can't control that's writing the files to the container root and that cannot be changed. As I wrote in my description, this container is not meant to have UC managed resources in it and i...

emorgoch

Thanks comb for the suggestion and the sample code. Looking into the foreachbatch processing, this seems like if could work. My one concern with it is the information here about foreachbatch serializing the processing and removing parallelization, bu...

emorgoch

Thanks Adnan,What I'm seeing with your solution, however, is that I need to maintain a static list of object types that I will be importing. This is what I am attempting to avoid, as system owners may expand the object types exported from the source ...

emorgoch · 07-09-2026

Hey Ashwin,Just wanted to follow up and let you know: it appears that a change went through recently, and for the last week or so, we've now been seeing these time fields getting removed during the commit process within the web GUI. So new files we'v...

Databricks Community

User Stats

User Activity

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

Using autoloader with multiple object types in load path

Managing IPYNB cell timestamps in source control

Passing variables from python to sql in a notebook using serverless compute

Re: INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

Re: INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

Re: Using autoloader with multiple object types in load path

Re: Using autoloader with multiple object types in load path

Re: Managing IPYNB cell timestamps in source control