cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

TheBeacon
by New Contributor
  • 408 Views
  • 1 replies
  • 0 kudos

Exploring Postman Alternatives for API Testing in VSCode?

Has anyone here explored Postman alternatives within VSCode? I’ve seen mentions of Thunder Client and Apidog. Would love to know if they offer a smoother integration or better functionality.

  • 408 Views
  • 1 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor
  • 0 kudos

HiI'm using ThunderClient as VS Code extension: https://www.thunderclient.com/ The functionality in the Free version is okay. If you want more features, there are plans for some bucks. 

  • 0 kudos
Rutuja_3641
by New Contributor
  • 303 Views
  • 1 replies
  • 0 kudos

Mongo server to Delta Live Tables

I want to fetch data from mongodb server and then show that in delta live table in gcp.

  • 303 Views
  • 1 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor
  • 0 kudos

Hi @Rutuja_3641 Have a look here: https://docs.databricks.com/en/connect/external-systems/mongodb.htmlI think you can easily adapt the code to DLT

  • 0 kudos
RyoAriyama
by New Contributor II
  • 965 Views
  • 2 replies
  • 0 kudos

Can't create table in unity catalog.

Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...

  • 965 Views
  • 2 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor
  • 0 kudos

Hi RyoCan you share the code, how you try to create the table? 

  • 0 kudos
1 More Replies
ChristianRRL
by Valued Contributor
  • 559 Views
  • 1 replies
  • 0 kudos

Databricks "Preferred" Approaches To Backfilling Single Column In Wide Tables

Hi there,I've tried thinking through this and googling as well, but I'm not sure if there's a better approach that I might be missing. We have *wide* tables with hundreds of columns, and on a day-to-day basis these tables are incrementally filled in ...

  • 559 Views
  • 1 replies
  • 0 kudos
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @ChristianRRL ,If I understand correctly you have an API to get historical data for a column.If yes, you can use MERGE clause.You will join the by key columns with the target table to backfill and when there is a match, then you will UPDATE SET ta...

  • 0 kudos
elgeo
by Valued Contributor II
  • 29885 Views
  • 5 replies
  • 2 kudos

SQL Stored Procedure in Databricks

Hello. Is there an equivalent of SQL stored procedure in Databricks? Please note that I need a procedure that allows DML statements and not only Select statement as a function provides.Thank you in advance

  • 29885 Views
  • 5 replies
  • 2 kudos
Latest Reply
Biswajit
New Contributor II
  • 2 kudos

I have recently went through a video link from databricks which says its possible. But, when I tried it, it did not worked.https://www.youtube.com/watch?v=f4TxNBfSNqMWas anyone able to create stored procedure in databricks ?

  • 2 kudos
4 More Replies
RKNutalapati
by Valued Contributor
  • 4134 Views
  • 5 replies
  • 4 kudos

Read and saving Blob data from oracle to databricks S3 is slow

I am trying to import a table from oracle which has around 1.3 mill rows and one of the column is a Blob, the total size of data on oracle is around 250+ GB. read and save to S3 as delta table is taking around 60 min. I tried with parallel(200 thread...

  • 4134 Views
  • 5 replies
  • 4 kudos
Latest Reply
vinita_mehta
New Contributor II
  • 4 kudos

Any update on this topic what should be the best option to read from oracle and write in ADLS. 

  • 4 kudos
4 More Replies
NLearn
by New Contributor II
  • 600 Views
  • 1 replies
  • 0 kudos

Lakehouse monitoring

I created snapshot and time series lake house monitoring on 2 different tables. After execution, metrics table got created and dashboard also got created but monitoring data is not populating in dashboard and metrics tables even after refresh of moni...

  • 600 Views
  • 1 replies
  • 0 kudos
Latest Reply
mandree
New Contributor II
  • 0 kudos

Do you refreshed the metrics?Table -> Quality -> Refresh metrics

  • 0 kudos
Volker
by New Contributor III
  • 1438 Views
  • 5 replies
  • 0 kudos

Structured Streaming: How to handle Schema Changes in source and target Delta Table

Hey Community,we have a streaming pipeline, starting with autoloader to ingest data into a bronze table and this data gets then picked up by another streaming job that transforms this data and writes into a silver table.Now there are some schema chan...

  • 1438 Views
  • 5 replies
  • 0 kudos
Latest Reply
Volker
New Contributor III
  • 0 kudos

I adjusted the schema in both bronze and silver, such that I do not need schema evolution. The problem is that the DataStreamReader does not pick up the schema changes in bronze.I already figured out that it has something to do with providing also a ...

  • 0 kudos
4 More Replies
EricCournarie
by New Contributor III
  • 406 Views
  • 1 replies
  • 0 kudos

Resolved! With the JDBC Driver, ResultSet metadata gives the column type as column name

Hello,Using the JDBC Driver , when I check the metadata of a result set, when retrieving the column name, it's the column type that is given . Here what it looks like in the meta data . Using version 2.6.40, with the query SELECT "id" string, "year" ...

Screenshot 2024-10-16 at 11.28.36.png
  • 406 Views
  • 1 replies
  • 0 kudos
Latest Reply
EricCournarie
New Contributor III
  • 0 kudos

forget about it, the select clause is wrong of course .. too tired I guess sorry

  • 0 kudos
Vladif1
by New Contributor II
  • 7705 Views
  • 8 replies
  • 1 kudos

Error when reading delta lake files with Auto Loader

Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = (    spark.readStream    .format('cloudFiles')    .option("cloudFiles.format", "delta")    .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint")    .load(bronz...

  • 7705 Views
  • 8 replies
  • 1 kudos
Latest Reply
Panda
Valued Contributor
  • 1 kudos

@Vladif1 The error occurs because the cloudFiles format in Auto Loader is meant for reading raw file formats like CSV, JSON ... for ingestion for more Format Support. For Delta tables, you should use the Delta format directly. #Sample Example bronze...

  • 1 kudos
7 More Replies
dataslicer
by Contributor
  • 2425 Views
  • 3 replies
  • 1 kudos

How to export/clone Databricks Notebook without results via web UI?

When a Databricks Notebook exceeds size limit, it suggests to `clone/export without results`.  This is exactly what I want to do, but the current web UI does not provide the ability to bypass/skip the results in either the `clone` or `export` context...

  • 2425 Views
  • 3 replies
  • 1 kudos
Latest Reply
dataslicer
Contributor
  • 1 kudos

Thank you @Yeshwanth for the response. I am looking for a way without clearing up the current outputs. This is necessary because I want to preserve the existing outputs and fork off another notebook instance to run with few parameter changes and come...

  • 1 kudos
2 More Replies
dbx_687_3__1b3Q
by New Contributor III
  • 11677 Views
  • 10 replies
  • 4 kudos

Resolved! Databricks Asset Bundle (DAB) from a Git repo?

My earlier question was about creating a Databricks Asset Bundle (DAB) from an existing workspace. I was able to get that working but after further consideration and some experimenting, I need to alter my question. My question is now "how do I create...

  • 11677 Views
  • 10 replies
  • 4 kudos
Latest Reply
mflyingget
New Contributor II
  • 4 kudos

How can i deploy a custom Gitrepo to the .bundle workshapce?

  • 4 kudos
9 More Replies
shadowinc
by New Contributor III
  • 5405 Views
  • 7 replies
  • 4 kudos

Resolved! Databricks SQL endpoint as Linked Service in Azure Data Factory

We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this. 

Data Engineering
SQl endpoint
  • 5405 Views
  • 7 replies
  • 4 kudos
Latest Reply
yashrg
New Contributor III
  • 4 kudos

Azure Databricks Delta Lake (Dataset) uses a linked service that can only connect to a All Purpose/Interactive cluster.If you want to use the SQL Endpoint, you would need a Self Hosted Integration Runtime for ADF with Databricks ODBC driver Installed...

  • 4 kudos
6 More Replies
tonylax6
by New Contributor
  • 597 Views
  • 0 replies
  • 0 kudos

Azure Databricks to Adobe Experience Platform

I'm using Azure databricks and am attempting to stream near real-time data from databricks into the Adobe Experience Platform to ingest into the AEP schema for profile enrichment.We are running into an issue with the API and streaming, so we are curr...

Data Engineering
Adobe
Adobe Experience Platform
CDP integration
  • 597 Views
  • 0 replies
  • 0 kudos
andreasmherman
by New Contributor II
  • 3654 Views
  • 4 replies
  • 1 kudos

DLT SCD type 2 in bronze, silver and gold? Is it possible?

I have a question related to when we are using  #DLT. Let me try to describe the DLT problem: Objective: Process data end-to-end (bronze, silver gold) using DLTWant bronze to hold a complete raw replica of the data, leveraging apply_changes SCD to wr...

  • 3654 Views
  • 4 replies
  • 1 kudos
Latest Reply
thedatacrew
New Contributor III
  • 1 kudos

Hi,At the moment, my process is:-I am using ETL (Data Factory) to land paquet files in a raw landing zone. I keep all the source data here. So I can fully rebuild the data if I need to.i.e.source_system/schema/table/loadon_year=2024/loadon_month=08/l...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels