cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Mado
by • Valued Contributor II
  • 6494 Views
  • 1 replies
  • 0 kudos

Resolved! Error when query a table created by DLT pipeline; "Couldn't find value of a column"

Hi, I create a table using DLT pipeline (triggered once). In the ETL process, I add a new column to the table with Null values by:output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None))Pipeline works and I don't get any error. But, whe...

Data Engineering
Reply
  • 6494 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
josruiz22
New Contributor III
  • 0 kudos

Hi,Try converting the None of the output line this :output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None).cast("String"))

  • 0 kudos
Rajkishore
by • New Contributor II
  • 11225 Views
  • 6 replies
  • 4 kudos

Need a way to show the non-trimmed data while query a table ?

When querying a json data , the values are getting trimmed. I need to see the full data for that field, is there any way to do so ?

Data Engineering
Reply
  • 11225 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Raj Sethi​ We haven't heard from you since the last response from @Lakshay Goel​  and @Vigneshraja Palaniraj​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, a...

  • 4 kudos
5 More Replies
Dale_Ware
by • New Contributor III
  • 4098 Views
  • 2 replies
  • 3 kudos

Resolved! How to query a table with backslashes in the name.

I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...

Data Engineering
Reply
  • 4098 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"

  • 3 kudos
1 More Replies
StevenW
by • New Contributor III
  • 3739 Views
  • 4 replies
  • 4 kudos

Resolved! Manipulating Data - using Notebooks

I need to read/query table A, manipulate/modify the data and insert the new data into Table A again.I considered using :Cur_Actual = spark.sql("Select * from Table A")currAct_Rows = Cur_Actual.rdd.collect()for row in currAct_Rows: do_somthing(row)...

Data Engineering
Reply
  • 3739 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Manoj12421
Valued Contributor II
  • 4 kudos

You can use withColumn() for the transformations and then write data this can be append, overwrite, merge .

  • 4 kudos
3 More Replies
Fred_F
by • New Contributor III
  • 8040 Views
  • 5 replies
  • 5 kudos

JDBC connection timeout on workflow cluster

Hi there,​I've a batch process configured in a workflow which fails due to a jdbc timeout on a Postgres DB.​I checked the JDBC connection configuration and it seems to work when I query a table and doing a df.show() in the process and it displays th...

Data Engineering
Reply
  • 8040 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
RKNutalapati
Valued Contributor
  • 5 kudos

HI @Fred Foucart​ ,The above code looks good to me. Can you try with below code as well.spark.read\  .format("jdbc") \  .option("url", f"jdbc:postgresql://{host}/{database}") \  .option("driver", "org.postgresql.Driver") \  .option("user", username) ...

  • 5 kudos
4 More Replies
Labels