cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
vivek_sinha
by • Contributor
  • 18842 Views
  • 4 replies
  • 4 kudos

Resolved! PySpark on Jupyterhub K8s || Unable to query data || Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

Pyspark Version: 2.4.5 Hive Version: 1.2 Hadoop Version: 2.7 AWS-SDK Jar: 1.7.4 Hadoop-AWS: 2.7.3When I am trying to show data I am getting Class org.apache.hadoop.fs.s3a.S3AFileSystem not found while I am passing all the information which all are re...

Data Engineering
Reply
  • 18842 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​ Thanks for the response and now I fixed the issue.The image which I was using to launch spark executor didn't have aws jars. After doing necessary changes it started working.But still may thanks for your response.

  • 4 kudos
3 More Replies
yopbibo
by • Contributor II
  • 4589 Views
  • 5 replies
  • 4 kudos

Resolved! Column name, starting with a number

Hi,I see it is possible to start a column name with a number, like `123_test`And store in a hive table with a location in delta.On that documentation https://www.stitchdata.com/docs/destinations/databricks-delta/reference#transformations--column-nami...

Data Engineering
Reply
  • 4589 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
yopbibo
Contributor II
  • 4 kudos

ha ha, yes, I try to find back the right page in DB documentation. If you have it, please, share.

  • 4 kudos
4 More Replies
herry
by • New Contributor III
  • 5244 Views
  • 5 replies
  • 1 kudos

Hive Serde table vs Delta table

This might be stupid question. Does the Hive Serde table have the same features (e.g. transactions) comparing to the Delta table?I tried to find the information in the Databricks documentation but I cannot find a clear answer.I create the Hive Serde ...

Data Engineering
Reply
  • 5244 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

AFAIK Hive SerDe is just Serializer and Deserializer (write and read data to/from storage).Hive uses SerDe (and FileFormat) to read and write table rows. So it is not an actual file format like parquet, orc and also delta lake (which I consider a se...

  • 1 kudos
4 More Replies
SQL
by • New Contributor II
  • 2002 Views
  • 5 replies
  • 1 kudos

Presto hive table to delta table conversion

Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with  ex...

Data Engineering
Reply
  • 2002 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
SQL
New Contributor II
  • 1 kudos

could you please help me with the alternative function for date_array

  • 1 kudos
4 More Replies
Labels