Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2023 10:38 AM
below option used while performing spark.readStream:::
.option('cloudFiles.format', 'json')
.option('cloudFiles.inferColumnTypes', 'true')
.option('cloudFiles.schemaEvolutionMode', 'rescue')
.option('cloudFiles.useNotifications', True)
.option('skipChangeCommits', 'true')
.option('cloudFiles.fetchParallelism', 100)
.option('cloudFiles.includeExistingFiles', False)
.option('cloudFiles.maxFilesPerTrigger', 10000)
.option('modifiedAfter', '2024-01-01 00:00:00.000000 UTC+0')
.option('cloudFiles.region', 'aws-region')
.option('cloudFiles.queueUrl', 'sqs-name')
.option('cloudFiles.roleArn', 'role-name-optional-if-instance-profile-has-permission')
.option('cloudFiles.roleSessionName','session-name')
.option('cloudFiles.stsEndpoint','required-if-roleArn-being-used')
