cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kolangareth
by New Contributor III
  • 5791 Views
  • 10 replies
  • 3 kudos

Resolved! to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS

I am trying to do a dynamic partition overwrite on delta table using replaceWhere option. This was working fine until I upgraded the DB runtime to 9.1 LTS from 8.3.x. I am concatenating 'year', 'month' and 'day' columns and then using to_date functio...

  • 5791 Views
  • 10 replies
  • 3 kudos
Latest Reply
ltreweek
Visitor
  • 3 kudos

SELECT TO_DATE('20250217','YYYYMMDD'); gives the error: PARSE_SYNTAX_ERROR  syntax error at or near 'select'. sqlstate: 42601.  It datagrip, it works no problem and displays the date.

  • 3 kudos
9 More Replies
RUC
by Visitor
  • 12 Views
  • 0 replies
  • 0 kudos

Update target column with optional source columns

I am trying to load on prem transaction log table and update a databricks table.In Source table,the column reqdetails table hold all the information and expect ProductID remaining columns are dynamic(not all columns exists in the request) all time th...

  • 12 Views
  • 0 replies
  • 0 kudos
analytics_eng
by New Contributor II
  • 599 Views
  • 3 replies
  • 1 kudos

Connection reset by peer logging when importing custom package

Hi! I'm trying to import a custom package I published to Azure Artifacts, but I keep seeing the INFO logging below, which I don't want to display. The package was installed correctly on the cluster, and it imports successfully, but the log still appe...

  • 599 Views
  • 3 replies
  • 1 kudos
Latest Reply
analytics_eng
New Contributor II
  • 1 kudos

Thanks for the suggestions. I investigated all of the above, but they didn't provide a solution. What did work was using another logging package within my custom package: Loguru. Not sure why this helped?

  • 1 kudos
2 More Replies
cmunteanu
by Contributor
  • 36 Views
  • 2 replies
  • 0 kudos

External connection to Azure ADLS Gen2 storage

Hello, I have a problem trying to make an external connection to a blob storage configured as ADLS Gen2 with hierarchical namespace (HNS) enabled. I have setuup the storage account with a container wirh HNS enabled as in the image attached:Next I hav...

cmunteanu_0-1739883785698.png cmunteanu_1-1739883952145.png cmunteanu_2-1739885124675.png
  • 36 Views
  • 2 replies
  • 0 kudos
Latest Reply
hao_hu
New Contributor
  • 0 kudos

Hi, would it work if you try to remove "landing" at the end? Seems the error is complaining that the external location should be a directory.   

  • 0 kudos
1 More Replies
Splush_
by New Contributor II
  • 39 Views
  • 1 replies
  • 0 kudos

Resolved! Hostname not resolving using Spark JDBC

Hey guys,Ive ran into a weird error this morning. Last week I was testing a new Oracle Connector and it ran through smooth the whole last week!This morning at 7 it ran again and following it was showing a "SQLRecoverableException: IO Error: Unknown h...

  • 39 Views
  • 1 replies
  • 0 kudos
Latest Reply
Splush_
New Contributor II
  • 0 kudos

I have even cloned the cluster and it worked on the new one. But after turning the cluster off over night, it started working again the next morning. This is really weird.

  • 0 kudos
kenmyers-8451
by New Contributor II
  • 80 Views
  • 1 replies
  • 0 kudos

Long runtimes on simple copying of data

Hi my team has been trying to identify areas where we can improve our processes. We have some long runtimes on processes that have multiple joins and aggregations. To create a baseline we have been running tests on a simple select and write operation...

kenmyers8451_0-1739400824751.png
  • 80 Views
  • 1 replies
  • 0 kudos
Latest Reply
kenmyers-8451
New Contributor II
  • 0 kudos

I ran another test this week where I changed my source table to not have deletion vectors and no longer believe that step is limiting factor. Without it the compute times seemed to be as follows:reading in data + wscg + exchange = 7.9 hours, ~1/3 wal...

  • 0 kudos
shubham_007
by Contributor II
  • 17 Views
  • 0 replies
  • 0 kudos

Dear experts, need urgent help on logic.

Dear experts,I am facing difficulty while developing pyspark automation logic on “Developing automation logic to delete/remove display() and cache() method used in scripts in multiple databricks notebooks (tasks)”.kindly advise on developing automati...

  • 17 Views
  • 0 replies
  • 0 kudos
ashraf1395
by Valued Contributor II
  • 120 Views
  • 3 replies
  • 1 kudos

How to capture dlt pipeline id / name using dynamic value reference

Hi there,I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters a...

  • 120 Views
  • 3 replies
  • 1 kudos
Latest Reply
ashraf1395
Valued Contributor II
  • 1 kudos

Hi @mourakshit , I tried all the three methods you mentioned . None of them workedmethod_1 returne pipeline_name or pipelin_id as printed value : {{dlt_pipeline.name}} {{dlt_pipeline.id}} not the actual valuesmethond_2 returned not conf like spark.da...

  • 1 kudos
2 More Replies
Dominos
by New Contributor II
  • 133 Views
  • 4 replies
  • 0 kudos

Does DBR 14.3 not support Describe history command?

Hello, We have recently updated DBR version from 9.1 LTS to 14.3 LTS and observed that DESCRIBE HISTORY is not supported in 14.3 LTS. Could you please suggest any alternative to be used for table history? 

  • 133 Views
  • 4 replies
  • 0 kudos
Latest Reply
holly
Databricks Employee
  • 0 kudos

Hi, I'm still not able to recreate this issue with Standard_DS3_v2.  I'm not sure if this is relevant, but do you also have this issue on an old High Concurrency cluster with custom access mode for the Standard_DS3_v2 cluster too? 

  • 0 kudos
3 More Replies
Faizan_khan8171
by New Contributor
  • 25 Views
  • 1 replies
  • 0 kudos

External Location Naming Issue & Impact of Renaming in Unity Catalog

Hey,I created an external location in my test environment using a mount point . Now, when I try to create the same external location in prod, it doesn’t allow me to use the same name. Is there any specific reason for this restriction in Unity Catalog...

  • 25 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @Faizan_khan8171, Thanks for your question. In Unity Catalog, external location names must be unique within the metastore. This restriction prevents naming conflicts and ensures that every external location is easily identifiable and manageable...

  • 0 kudos
RobsonNLPT
by Contributor III
  • 26 Views
  • 1 replies
  • 0 kudos

Databricks Rest API Statement Execution - External Links

Hi.I've tested the adb Rest Api to execute queries on databricks sql serverless. Using INLINE as disposition I have the json array with my correct results but using EXTERNAL_LINKS I have the chunks but the external_link (URL starting with http://stor...

  • 26 Views
  • 1 replies
  • 0 kudos
Latest Reply
RobsonNLPT
Contributor III
  • 0 kudos

By the way I can acess all chunks using the endpoint /api/2.0/sql/statements/{statement_id}/result/chunks/{chunk_index}

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels