If it's streaming data, space it out with 10 seconds trigger

.trigger(processingTime="10 seconds")

Do all the JSON files have the same schema? As your table creation is dynamic (df.schema), if all JSON doesn't have the same schema they may be skipped.