cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

HashStudioz
by New Contributor
  • 735 Views
  • 0 replies
  • 0 kudos

Rs 485 IoT Gateway

RS-485 IoT Gateway is used for transmitting data from one device to another usually far away by using a wired LAN or a Wi-Fi. HashStudioz Technologies Inc. provides Smart IoT Gateway Solutions for Businesses like Pharma industries Etc. Our IoT Gatewa...

  • 735 Views
  • 0 replies
  • 0 kudos
logan0015
by Contributor
  • 2656 Views
  • 3 replies
  • 3 kudos

How do you access a streaming live table's snapshots?

I have read that delta live tables will keep a history of 7 days. However after creating a streaming live table and using the dlt.apply_changes function. With this codedef run_pipeline(table_name,keys,sequence_by): lower_table_name = table_name.l...

  • 2656 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Logan Nicol​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
2 More Replies
Muni
by New Contributor II
  • 9613 Views
  • 5 replies
  • 0 kudos

Resolved! Can we expose REST API using Databricks python notebook?

We are trying to expose a REST API endpoint using python notebook. Does databricks allow this?

  • 9613 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Muniyappan Mani​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
4 More Replies
ABose8
by Contributor
  • 2657 Views
  • 3 replies
  • 2 kudos

I attended "The Best Data Warehouse Is a Lakehouse" on September 20 but have not recieved my voucher although I recieved a mail which mentio...

I attended "The Best Data Warehouse Is a Lakehouse" on September 20 but have not recieved my voucher although I recieved a mail which mentioned below details.@Kaniz Fatma​  Please help me on same.Mail Id: bose.innovator8@gmail.comPlease complete the ...

  • 2657 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Arindam Bose​ That's great to hear! Please mark the answer as best.Thanks and Regards

  • 2 kudos
2 More Replies
Sajid1
by Contributor
  • 5311 Views
  • 3 replies
  • 6 kudos

Resolved! Photon Acceleration not getting enabled for ML runtime in Azure

I tried to enable the photon acceleration in ML runtime 9.1 LTS ML (Scala 2.12,Spark 3.1.2) but getting error "selected runtime version does not support photon".I tried for other versions of ML runtime with single and multinode , access mode being s...

  • 5311 Views
  • 3 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Sajid Thavalengal Rahiman​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 6 kudos
2 More Replies
140015
by New Contributor III
  • 4761 Views
  • 6 replies
  • 1 kudos

Delta Live Tables @expect compare tables count between two stages

Hello,I'm wondering if there is an option to make an expectation on DLT that will compare the number of records between two stages and e.g. fail if there is a difference between those counts?I mean something like this:@dlt.table()def bronze(): Some...

  • 4761 Views
  • 6 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Jacek Dembowiak​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
5 More Replies
SPuchnanda
by New Contributor III
  • 2179 Views
  • 3 replies
  • 1 kudos

I have cleared Databricks Foundational Accreditation today and received the completion certificate. But I didnt receive the badge for the same. By whe...

I have cleared Databricks Foundational Accreditation today and received the completion certificate. But I didnt receive the badge for the same. By when I will receive the badge ?

  • 2179 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Sakshi Puchnanda​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
plynton
by New Contributor II
  • 2187 Views
  • 3 replies
  • 1 kudos

Incorrect results with df.query()

I have tried pulling a single row from a .csv using df.query()However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:df = spark.read.option("header",True).csv(data_fldr + "config...

  • 2187 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Peter Ott​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
bozhu
by Contributor
  • 3056 Views
  • 2 replies
  • 1 kudos

Multiple DLT Pipelines Sharing a Single Cluster

When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.I understand DLT pipelines utilising their own special clusters...

  • 3056 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Bo Zhu​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Lars_J
by New Contributor
  • 2528 Views
  • 2 replies
  • 0 kudos

Databricks-jdbc and vulnerabilities CVE-2022-42004, CVE-2022-42003

The latest version of Databricks-jdbc available through Maven (2.6.29) now has these two vulnerabilities:https://nvd.nist.gov/vuln/detail/CVE-2022-42004https://nvd.nist.gov/vuln/detail/CVE-2022-42003All due to depending on and including in the jar th...

  • 2528 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Lars Joreteg​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
AJDJ
by New Contributor III
  • 3295 Views
  • 3 replies
  • 1 kudos

Pipeline in Community edition

Hi there,I'm learning Databricks using the community edition. I noticed I don't have way to practice the pipeline in community edition. (The icon below compute). Says I need to upgrade. Is there any way to practice pipeline and follow learning lesson...

  • 3295 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @AJ DJ​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
gbradley145
by New Contributor III
  • 12795 Views
  • 4 replies
  • 5 kudos

SQL CTE in Databricks or something similar?

%sqlWITH genCTE AS (SELECT MAX(PredID) + 1 AS PredID, 145 AS SystemIDFROM TableAUNION ALLSELECT PredID + 1FROM genCTE)SELECT * FROM genCTEWhen I attempt this, I get an error that genCTE does not exists.There may be a better way to what I am trying to...

  • 12795 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Greg Bradley​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 5 kudos
3 More Replies
William_Scardua
by Valued Contributor
  • 11991 Views
  • 7 replies
  • 3 kudos

uuid in Merge

Hi guys,I'm trying to use uuid in the merge but I always get an error...import uuid   ( df_events.alias("events").merge( source = df_updates.alias("updates"), condition = "events.cod = updates.cod and events.num = updates.num" ).whenMatch...

  • 11991 Views
  • 7 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @William Scardua​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 3 kudos
6 More Replies
sm1
by New Contributor III
  • 5367 Views
  • 5 replies
  • 3 kudos

New Visualization Tools

How do I add new visualization tool option to my databricks? I don't see a plus sign that will let you choose "Visualization" in my display command results :(. Please help.

Capture
  • 5367 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Suky Muliadikara​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 3 kudos
4 More Replies
ramankr48
by Contributor II
  • 4094 Views
  • 2 replies
  • 3 kudos

Issue with identity key column in databricks?

For the identity key I've used both GENERATED ALWAYS AS IDENTITY(start with 1 increment by 1) andGENERATED BY DEFAULT AS IDENTITY(start with 1 increment by 1)but in both cases, if I'm running my script once then it is fine (identity key is working as...

  • 4094 Views
  • 2 replies
  • 3 kudos
Latest Reply
lizou
Contributor III
  • 3 kudos

yes, by default option allow duplicated values per design.I will avoid this option and use only use GENERATED ALWAYS AS IDENTITY Using BY DEFAULT option is worse than not using it at all in BY Default option, If I forget to set starting value, the ID...

  • 3 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels