Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-07-2024 01:30 AM
@Ayushi_SutharThanks! However, this does not solve the issue; because we have to use shared clusters. In the meantime we found a way of detecting the encoding on shared clusters in the following way:
rawdata = (
spark.read.format("binaryFile")
.load(file_path)
.selectExpr("SUBSTR(content, 0, 500000) AS content")
.collect()[0]
.content
)
encoding = chardet.detect(rawdata)["encoding"]
print(encoding)
