cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vivek_sinha
by Contributor
  • 14937 Views
  • 4 replies
  • 4 kudos

Resolved! PySpark on Jupyterhub K8s || Unable to query data || Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

Pyspark Version: 2.4.5 Hive Version: 1.2 Hadoop Version: 2.7 AWS-SDK Jar: 1.7.4 Hadoop-AWS: 2.7.3When I am trying to show data I am getting Class org.apache.hadoop.fs.s3a.S3AFileSystem not found while I am passing all the information which all are re...

  • 14937 Views
  • 4 replies
  • 4 kudos
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​ Thanks for the response and now I fixed the issue.The image which I was using to launch spark executor didn't have aws jars. After doing necessary changes it started working.But still may thanks for your response.

  • 4 kudos
3 More Replies
yopbibo
by Contributor II
  • 3172 Views
  • 5 replies
  • 4 kudos

Resolved! Column name, starting with a number

Hi,I see it is possible to start a column name with a number, like `123_test`And store in a hive table with a location in delta.On that documentation https://www.stitchdata.com/docs/destinations/databricks-delta/reference#transformations--column-nami...

  • 3172 Views
  • 5 replies
  • 4 kudos
Latest Reply
yopbibo
Contributor II
  • 4 kudos

ha ha, yes, I try to find back the right page in DB documentation. If you have it, please, share.

  • 4 kudos
4 More Replies
herry
by New Contributor III
  • 4172 Views
  • 5 replies
  • 1 kudos

Hive Serde table vs Delta table

This might be stupid question. Does the Hive Serde table have the same features (e.g. transactions) comparing to the Delta table?I tried to find the information in the Databricks documentation but I cannot find a clear answer.I create the Hive Serde ...

  • 4172 Views
  • 5 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

AFAIK Hive SerDe is just Serializer and Deserializer (write and read data to/from storage).Hive uses SerDe (and FileFormat) to read and write table rows. So it is not an actual file format like parquet, orc and also delta lake (which I consider a se...

  • 1 kudos
4 More Replies
SQL
by New Contributor II
  • 1291 Views
  • 5 replies
  • 1 kudos

Presto hive table to delta table conversion

Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with  ex...

  • 1291 Views
  • 5 replies
  • 1 kudos
Latest Reply
SQL
New Contributor II
  • 1 kudos

could you please help me with the alternative function for date_array

  • 1 kudos
4 More Replies
Labels