cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Ericsson
by • New Contributor II
  • 3213 Views
  • 3 replies
  • 1 kudos

SQL week format issue its not showing result as 01(ww)

Hi Folks,I've requirement to show the week number as ww format. Please see the below codeselect weekofyear(date_add(to_date(current_date, 'yyyyMMdd'), +35)). also plz refre the screen shot for result.

result
Data Engineering
Reply
  • 3213 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Feltonrolfson
New Contributor II
  • 1 kudos

It seems you're encountering an issue with SQL week formatting, where the results aren't displaying as expected (01(ww)). This could impact data analysis. monkey mart

  • 1 kudos
2 More Replies
Ram443
by • New Contributor III
  • 35157 Views
  • 9 replies
  • 5 kudos

Resolved! I created a data frame but was not able to see the data

Code to create a data frame:from pyspark.sql import SparkSessionspark=SparkSession.builder.appName("oracle_queries").master("local[4]")\  .config("spark.sql.warehouse.dir", "C:\\softwares\\git\\pyspark\\hive").getOrCreate()from pyspark.sql.functions ...

Data Engineering
Reply
  • 35157 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 5 kudos

@ramanjaneyulu kancharla​  can you please select my answer as best answer

  • 5 kudos
8 More Replies
Jfoxyyc
by • Valued Contributor
  • 2027 Views
  • 2 replies
  • 0 kudos

DLT - deduplication pattern?

Say we have an incremental append happening using autoloader, where filename is being added to the dataframe and that's all. If we want to de-duplicate this data in a rolling window, we can do something like merge into logs using dedupedLogs on ...

Data Engineering
Reply
  • 2027 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jordan Fox​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Neli
by • New Contributor III
  • 4466 Views
  • 2 replies
  • 0 kudos

How to add Current date as one of the column in Databricks

I am trying to create new column "Ingest_date" in table which should contain current date. I am getting error "Current date cannot be used in a generated column". Can you please review and suggest alternative to get the current date in delta table.

image image
Data Engineering
Reply
  • 4466 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

A generation expression can use any SQL functions in Spark that always return the same result when given the same argument valuesSource: https://docs.delta.io/latest/delta-batch.html#use-generated-columnsIt means that it's intended to not work.You ca...

  • 0 kudos
1 More Replies
Kash
by • Contributor III
  • 1443 Views
  • 2 replies
  • 6 kudos

Will Vacuum delete previous folders of data if we z-ordered by as_of_date each day?

Hi there,I've had horrible experiences Vacuuming tables in the past and losing tons of data so I wanted to confirm a few things about Vacuuming and Z-Order.Background:Each day we run an ETL job that appends data in a table and stores the data in S3 b...

Data Engineering
Reply
  • 1443 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Avkash Kana​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 6 kudos
1 More Replies
Sajid1
by • Contributor
  • 29885 Views
  • 3 replies
  • 5 kudos

Resolved! Parse Syntax error ,can anyone guide me what is going wrong here

Select case WHEN {{ Month }} = 0 then add_months(current_date(),-13 ) elseWHEN {{ Month }}> month(add_months(current_date(),-1)) then add_months(to_date(concat(year(current_date())-1,'-',{{Month}},'-',1)),-13)             else add_months(to_date(conc...

Data Engineering
Reply
  • 29885 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 5 kudos

Hi @Sajid Thavalengal Rahiman​ , Have you followed the recommendation given above? Also, could you please paste the whole error with the code?

  • 5 kudos
2 More Replies
Vibhor
by • Contributor
  • 6154 Views
  • 5 replies
  • 2 kudos

Get current date as string in databricks using scala

I want to get current date in scala as a string for example today current date is 3rd jan, want to store it as a new variable dynamically as below, how to get it.val currdate : String = “20220103”when I am using val currdate = Calendar.getInstance.ge...

Data Engineering
Reply
  • 6154 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey @Vibhor Sethi​ Hope you are well!Thank you for posting your question and letting us know that you were able to resolve the issue. Would you be happy to mark it as the best solution? It would be really helpful for the other members too.Cheers!

  • 2 kudos
4 More Replies
philip
by • New Contributor
  • 6963 Views
  • 2 replies
  • 2 kudos

Resolved! current date as default in a widget while scheduling the notebook

I have a scheduled a notebook. can I keep current date as default in widget whenever the notebook run and also i need the flexibility to change the widget value to any other date based on the ad hoc run that I do.

Data Engineering
Reply
  • 6963 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

So building on the answer of Hubert:from datetime import datedate_for_widget = date.today()So if you use date_for_widget as your default value, you are there.And ofc you can fill this date_for_widget variable with anything you want.You can even fetch...

  • 2 kudos
1 More Replies
Labels