Databricks

OLAPTrader · 02-04-2024

I first create a catalog and schema and ingest some data into it as follows:catalogName = 'neo'schemaName ='indicators'locationPath = 's3a://databricks-workspace-olaptrader-stack-1-bucket/unity-catalog/99999999xxx'sqlContext.sql(f"CREATE CATALOG IF N...

OLAPTrader · 02-06-2024

My issue was due to the fact that I have over 300 columns and due to datatype mismatches, the rows were actually written to the table, but values were all null. That's why I didnt get any errors. I am doing manual datatype mapping now and I am able t...

OLAPTrader · 02-06-2024

Thanks, I am not dropping the table and I do use append mode. However, the only time the autoloader works if if I do indeed drop the table. I was doing that just for testing. When I comment that line, I never get new files ingested into the table.

OLAPTrader · 02-05-2024

Thank you for your reply. When I uncomment the line which drops the table, this is when it actually works. I get the most recent csv ingested. (plus older ones I assume). This is not what I want to do. so, in short, when I do not drop the table, noth...

Databricks

User Stats

User Activity

autoloader stops working if I do not drop table each time

Re: autoloader stops working if I do not drop table each time

Re: autoloader stops working if I do not drop table each time

Re: autoloader stops working if I do not drop table each time