https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52746#M29562 <P>Can you set this value to higher number and try</P><P><SPAN>cloudFiles.fetchParallelism its 1 by default</SPAN></P> Fri, 17 Nov 2023 20:14:09 GMT Rdipak 2023-11-17T20:14:09Z https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52722#M29558 <P>I am using DLT with filenotification and DLT job is just fetching 1 notification from SQS queue at a time. My pipeline is expected to process 500K notifications per day but it running hours behind. Any recommendations?<BR /><BR /></P><DIV><DIV><SPAN>spark.readStream.</SPAN><SPAN>format</SPAN><SPAN>(</SPAN><SPAN>"cloudFiles"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>"cloudFiles.schemaLocation"</SPAN><SPAN>,</SPAN><SPAN>"/mnt/abc/"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'cloudFiles.format'</SPAN><SPAN>, </SPAN><SPAN>'json'</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'cloudFiles.inferColumnTypes'</SPAN><SPAN>, </SPAN><SPAN>'true'</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'cloudFiles.useNotifications'</SPAN><SPAN>, </SPAN><SPAN>True</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'skipChangeCommits'</SPAN><SPAN>, </SPAN><SPAN>'true'</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'cloudFiles.backfillInterval'</SPAN><SPAN>, </SPAN><SPAN>'3 hour'</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>option</SPAN><SPAN>(</SPAN><SPAN>'cloudFiles.maxFilesPerTrigger'</SPAN><SPAN>, </SPAN><SPAN>10000</SPAN><SPAN>)<BR /><BR /><BR />Logs:<BR />NotificationFileEventFetcher: [queryId =] Fetched 1 messages from cloud queue storage.<BR />NotificationFileEventFetcher: [queryId =] Fetched 1 messages from cloud queue storage.<BR />NotificationFileEventFetcher: [queryId =] Fetched 1 messages from cloud queue storage.<BR /></SPAN></DIV></DIV> Fri, 17 Nov 2023 17:46:47 GMT https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52722#M29558 kulkpd 2023-11-17T17:46:47Z https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52746#M29562 <P>Can you set this value to higher number and try</P><P><SPAN>cloudFiles.fetchParallelism its 1 by default</SPAN></P> Fri, 17 Nov 2023 20:14:09 GMT https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52746#M29562 Rdipak 2023-11-17T20:14:09Z https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52806#M29604 <P>Thanks<SPAN>cloudFiles.fetchParallelism to</SPAN>&nbsp;100 definitely helped to read more messages from SQS.<BR /><BR />NotificationFileEventFetcher: [queryId = 111] Fetched 100 messages from cloud queue storage</P> Fri, 17 Nov 2023 22:48:16 GMT https://community.databricks.com/t5/data-engineering/autoloader-with-filenotification/m-p/52806#M29604 kulkpd 2023-11-17T22:48:16Z