cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mado
by Valued Contributor II
  • 3124 Views
  • 1 replies
  • 0 kudos

Resolved! Error when query a table created by DLT pipeline; "Couldn't find value of a column"

Hi, I create a table using DLT pipeline (triggered once). In the ETL process, I add a new column to the table with Null values by:output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None))Pipeline works and I don't get any error. But, whe...

  • 3124 Views
  • 1 replies
  • 0 kudos
Latest Reply
josruiz22
New Contributor III
  • 0 kudos

Hi,Try converting the None of the output line this :output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None).cast("String"))

  • 0 kudos
Rajkishore
by New Contributor II
  • 6060 Views
  • 6 replies
  • 4 kudos

Need a way to show the non-trimmed data while query a table ?

When querying a json data , the values are getting trimmed. I need to see the full data for that field, is there any way to do so ?

  • 6060 Views
  • 6 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Raj Sethi​ We haven't heard from you since the last response from @Lakshay Goel​  and @Vigneshraja Palaniraj​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, a...

  • 4 kudos
5 More Replies
Dale_Ware
by New Contributor III
  • 1416 Views
  • 2 replies
  • 3 kudos

Resolved! How to query a table with backslashes in the name.

I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...

  • 1416 Views
  • 2 replies
  • 3 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"

  • 3 kudos
1 More Replies
StevenW
by New Contributor III
  • 1709 Views
  • 4 replies
  • 4 kudos

Resolved! Manipulating Data - using Notebooks

I need to read/query table A, manipulate/modify the data and insert the new data into Table A again.I considered using :Cur_Actual = spark.sql("Select * from Table A")currAct_Rows = Cur_Actual.rdd.collect()for row in currAct_Rows: do_somthing(row)...

  • 1709 Views
  • 4 replies
  • 4 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 4 kudos

You can use withColumn() for the transformations and then write data this can be append, overwrite, merge .

  • 4 kudos
3 More Replies
Fred_F
by New Contributor III
  • 4470 Views
  • 7 replies
  • 5 kudos

JDBC connection timeout on workflow cluster

Hi there,​I've a batch process configured in a workflow which fails due to a jdbc timeout on a Postgres DB.​I checked the JDBC connection configuration and it seems to work when I query a table and doing a df.show() in the process and it displays th...

  • 4470 Views
  • 7 replies
  • 5 kudos
Latest Reply
Kaniz
Community Manager
  • 5 kudos

Hi @Fred Foucart​, We haven’t heard from you since the last response from @Rama Krishna N​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to ...

  • 5 kudos
6 More Replies
Labels