Hi. Another question, this time about schema inference and column types. I have dabbled with DLT and structured streaming with auto loader (as in, not DLT). My data source use case is json files, which contain nested structures. I noticed that in t...
Hi. I am using structured streaming and auto loader to read json files, and it is automated by Workflow. I am having difficulties with the job failing as schema changes are detected, but not retrying. Hopefully someone can point me in the right dir...
Hi. I have a question, and I've not been able to find an answer. I'm sure there is one...I just haven't found it through searching and browsing the docs. How much does it matter (if it is indeed that simple) if source files read by auto loader are ...
Hi community We are using Unity Catalog, SCIM and Identity Federation, so we have users, groups and service principals at Account level. In what scenarios do users, groups and service principals need explicitly added to a Workspace?
Hi, I'd like to put this out here in case there are some helpful suggestions to be found. What am I trying to achieve?Generate a hash of certain columns in a dataframe (as in a row hash, but not the whole row) where currently one of the columns is an...
Another point I have realised, is that the task and the parent notebook (which then calls the child notebook that runs the auto loader part) does not fail if the schema-changed failure occurs during the auto loader process. It's the child notebook a...
Hi Kaniz, Thank you for your comprehensive response, I appreciate it. I have not resolved the issue in my situation yet, but I am perhaps a little closer. Basically, my Job is a 3-step chain of Tasks::Step 1 is a "set up" Task that queries metadata ...
Hi. Have you looked at using cluster spark configurations? We're using it for default catalog for example and it works quite well. I don't know for certain, but there might be an option to pre-set databases as well.Here's a topic about setting cat...
I'll offer my opinion. I see both of those checks (and treatments, if you're converting types for example) as something for the clean/silver/staging/whatever-you-call-it layer. For us, our bronze layer represents the source data as-is, with SCD typ...
