cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AlexDavies
by Contributor
  • 9089 Views
  • 7 replies
  • 2 kudos

Resolved! How to upgrade internal hive metadata store version

Is it possible to upgrade the out of the box hive metastore version? running spark.conf.get("spark.sql.hive.metastore.version") indicates that it is running on 0.13.0 However https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7....

  • 9089 Views
  • 7 replies
  • 2 kudos
Latest Reply
pantelis_mare
Contributor III
  • 2 kudos

Hello guys!Atanu's post, although correct does not solve the problem. Is there any official documentation on how to upgrade the internal databricks metastore to a greater version? If this is availble then we can try Atanu's solution (not sure if need...

  • 2 kudos
6 More Replies
159312
by New Contributor III
  • 2571 Views
  • 1 replies
  • 1 kudos

Resolved! How to get autoloader to load files in order

I'm new to spark and Databricks and I'm trying to write a pipeline to take CDC data from a postgres database stored in s3 and ingest it. The file names are numerically ascending unique ids based on datatime (ie20220630-215325970.csv). Right now auto...

  • 2571 Views
  • 1 replies
  • 1 kudos
Latest Reply
Noopur_Nigam
Databricks Employee
  • 1 kudos

Hi @Ben Bogart​ For lexicographically generated files, Auto Loader can leverage the lexical file ordering and optimized listing APIs. For more info on lexical ordering please go through the below link: https://docs.databricks.com/ingestion/auto-loade...

  • 1 kudos
fs
by New Contributor III
  • 9715 Views
  • 12 replies
  • 9 kudos

Resolved! how to access data objects from different languages [R/SQL/Spark/Python]

Hi sorry new to Spark, DataBricks. Please could someone summarise options for moving data between these different languages. Esp. interested in R<=>Python options: can see how to do SQL/Spark. Spent a lot of time googling but no result. Presume can u...

  • 9715 Views
  • 12 replies
  • 9 kudos
Latest Reply
Noopur_Nigam
Databricks Employee
  • 9 kudos

@Fernley Symons​ Thank you for your prompt reply. Apologies, we have just noticed that an answer is already marked as best. Thank you once again.

  • 9 kudos
11 More Replies
Andy_EU
by New Contributor
  • 1278 Views
  • 2 replies
  • 0 kudos

How do you do if/then statements in Delta Line Pipelines?

How do you do if/then statements in Python based Delta Line Pipelines? I'm essentially looking for the Python way of doing CASE statements.

  • 1278 Views
  • 2 replies
  • 0 kudos
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

Hi @Andy Pandy​ I hope that the answer provided by @Jose Gonzalez​ would have helped in resolving your query. Please let us know if you have more doubts or queries still.

  • 0 kudos
1 More Replies
User16783853906
by Contributor III
  • 3443 Views
  • 5 replies
  • 5 kudos

Resolved! Update code for a streaming job in Production

How to update a streaming job in production with minimal/no downtime when there are significant code changes that may not be compatible with the existing checkpoint state to resume the stream processing?

  • 3443 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Thanks for the information, I will try to figure it out for more. Keep sharing such informative post keep suggesting such post.MA Health Connector

  • 5 kudos
4 More Replies
vk217
by Contributor
  • 5224 Views
  • 3 replies
  • 1 kudos

Resolved! ERROR: No matching distribution found for databricks-connect==7.3.34

Previously, our databricks-connect was using 7.3.34 and the builds in pipenv and the builds were successful. As of today the builds are failing with error that the version 7.3.34 no longer exists.Is there a reason this version is no longer supported....

  • 5224 Views
  • 3 replies
  • 1 kudos
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

Hello @Vikas B​ this is the release note -https://docs.databricks.com/release-notes/dbconnect/index.htmlalso,Only the following Databricks Runtime versions are supported:Databricks Runtime 10.4 LTS ML, Databricks Runtime 10.4 LTSDatabricks Runtime 9....

  • 1 kudos
2 More Replies
sohamdhodapkar
by New Contributor
  • 1464 Views
  • 3 replies
  • 3 kudos
  • 1464 Views
  • 3 replies
  • 3 kudos
Latest Reply
Noopur_Nigam
Databricks Employee
  • 3 kudos

Hi @Soham Dhodapkar​ https://docs.databricks.com/lakehouse/index.html This document depicts the component of the lakehouse as described in the image shared by @Hubert Dudek​ .

  • 3 kudos
2 More Replies
codevisionz
by New Contributor
  • 564 Views
  • 0 replies
  • 0 kudos

Our Python Code Examples covers basic concepts, control structures, functions, lists, classes, objects, inheritance, polymorphism, file operations, da...

Our Python Code Examples covers basic concepts, control structures, functions, lists, classes, objects, inheritance, polymorphism, file operations, data structures, sorting algorithms, mathematical functions, mathematical sequences, threads, exceptio...

  • 564 Views
  • 0 replies
  • 0 kudos
Taha_Hussain
by Databricks Employee
  • 1183 Views
  • 0 replies
  • 8 kudos

Databricks Office Hours Register for Office Hours to participate in a LIVE Q&A session and receive technical support directly from Databricks expe...

Databricks Office HoursRegister for Office Hours to participate in a LIVE Q&A session and receive technical support directly from Databricks experts! Our next event is scheduled for July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GMT.Whether you ...

  • 1183 Views
  • 0 replies
  • 8 kudos
EveryDayData
by Contributor
  • 1751 Views
  • 1 replies
  • 1 kudos

MergeSchema on Delta Streaming

Hi Guys,Quick thing can we do MergeSchema on update mode in streaming or it is overwrite schema while using update mode . 

  • 1751 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey @Shikher Singh​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
sriramkumar
by New Contributor II
  • 1379 Views
  • 2 replies
  • 1 kudos

Reasons for new Databricks driver

What are the reasons behind Databricks going for their own driver? What differences are made when switching between the previous Spark driver and the new Databricks driver?Is there any specific document I can look at or just the release notes?Also, w...

  • 1379 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey @Sriramkumar Thamizharasan​ Hope all is well! Just wanted to check in if you were able to resolve your issue would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...

  • 1 kudos
1 More Replies
Krish-685291
by New Contributor III
  • 2074 Views
  • 4 replies
  • 1 kudos

How to perform UPCERTS and DELETE missing records from Target table using MERGE command?

Hi have the following case. I want to perform upcert operation. But along with upcert, I want to delete the records which are missing in source table, but present in the target table. You can think it as a master data update.Source table contains a f...

  • 2074 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey @Krishna Puthran​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 1 kudos
3 More Replies
williamsmith900
by New Contributor
  • 4012 Views
  • 2 replies
  • 1 kudos

How to remove old and unused Docker images

When running Docker for a long time, there are a lot of images in the system. How can I remove all unused Docker images at once safely to free up the storage?In addition, I also want to remove images pulled months ago, So, I'm not asking for removing...

  • 4012 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @william smith​ Hope everything is going great!Does @Prabakar Ammeappin​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you...

  • 1 kudos
1 More Replies
AlexandrePellet
by New Contributor
  • 3694 Views
  • 3 replies
  • 3 kudos

Resolved! Orchestration Preview - Error updating [JobName] - Either new_cluster or existing_cluster_id must be specified.

I've created a new job with the new UI / feature enabled. I managed to create one task with a new job cluster successfully but when adding a second task with a new job cluster and trying to save it I received the following error: Error updating [Job...

  • 3694 Views
  • 3 replies
  • 3 kudos
Latest Reply
eignerfr
New Contributor II
  • 3 kudos

Hi, I have a similar issue.I'm using the 14 day free trial, setting up the default basic-starter cluster.Then I just followed the basic introduction of dbx with python. https://docs.databricks.com/dev-tools/dbx.htmlCommand Execute is working, but whe...

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Latest Photos in Data Engineering