- 1396 Views
- 2 replies
- 0 kudos
How to load single line mode json file?
Hi there,The activity log store in adls gen2 container is a single line mode json file.How to load single line mode json file, save data to delta table? Thanks & Regards,zmsoft
- 1396 Views
- 2 replies
- 0 kudos
- 0 kudos
@zmsoft Since the JSON is a single-line file, ensure it is being read correctly. Try setting the multiLine option to false (it defaults to false, but explicitly setting it ensures correct handling). stageDf = ( spark.read.format("json") .opti...
- 0 kudos
- 2671 Views
- 2 replies
- 1 kudos
Resolved! what api calls does autoloader make on s3?
Hey guys I'm trying to find the estimate for an ingestion pipeline that uses autoloader on an S3 bucket every 2 minutes.I found the pricing for s3 bucket api consumption but I am not certain what api calls will autoloader make.Talking to chatGPT it t...
- 2671 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SakuraDev1 ,LIST and GET make sense.How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.So a LIST request is needed to check the files in the source directory, and th...
- 1 kudos
- 1187 Views
- 1 replies
- 0 kudos
Mongo server to Delta Live Tables
I want to fetch data from mongodb server and then show that in delta live table in gcp.
- 1187 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Rutuja_3641 Have a look here: https://docs.databricks.com/en/connect/external-systems/mongodb.htmlI think you can easily adapt the code to DLT
- 0 kudos
- 2536 Views
- 2 replies
- 0 kudos
Can't create table in unity catalog.
Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...
- 2536 Views
- 2 replies
- 0 kudos
- 2082 Views
- 1 replies
- 0 kudos
Databricks "Preferred" Approaches To Backfilling Single Column In Wide Tables
Hi there,I've tried thinking through this and googling as well, but I'm not sure if there's a better approach that I might be missing. We have *wide* tables with hundreds of columns, and on a day-to-day basis these tables are incrementally filled in ...
- 2082 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ChristianRRL ,If I understand correctly you have an API to get historical data for a column.If yes, you can use MERGE clause.You will join the by key columns with the target table to backfill and when there is a match, then you will UPDATE SET ta...
- 0 kudos
- 7783 Views
- 5 replies
- 4 kudos
Read and saving Blob data from oracle to databricks S3 is slow
I am trying to import a table from oracle which has around 1.3 mill rows and one of the column is a Blob, the total size of data on oracle is around 250+ GB. read and save to S3 as delta table is taking around 60 min. I tried with parallel(200 thread...
- 7783 Views
- 5 replies
- 4 kudos
- 4 kudos
Any update on this topic what should be the best option to read from oracle and write in ADLS.
- 4 kudos
- 1503 Views
- 1 replies
- 0 kudos
Lakehouse monitoring
I created snapshot and time series lake house monitoring on 2 different tables. After execution, metrics table got created and dashboard also got created but monitoring data is not populating in dashboard and metrics tables even after refresh of moni...
- 1503 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you refreshed the metrics?Table -> Quality -> Refresh metrics
- 0 kudos
- 8591 Views
- 5 replies
- 0 kudos
Structured Streaming: How to handle Schema Changes in source and target Delta Table
Hey Community,we have a streaming pipeline, starting with autoloader to ingest data into a bronze table and this data gets then picked up by another streaming job that transforms this data and writes into a silver table.Now there are some schema chan...
- 8591 Views
- 5 replies
- 0 kudos
- 0 kudos
I adjusted the schema in both bronze and silver, such that I do not need schema evolution. The problem is that the DataStreamReader does not pick up the schema changes in bronze.I already figured out that it has something to do with providing also a ...
- 0 kudos
- 2701 Views
- 1 replies
- 0 kudos
Resolved! With the JDBC Driver, ResultSet metadata gives the column type as column name
Hello,Using the JDBC Driver , when I check the metadata of a result set, when retrieving the column name, it's the column type that is given . Here what it looks like in the meta data . Using version 2.6.40, with the query SELECT "id" string, "year" ...
- 2701 Views
- 1 replies
- 0 kudos
- 0 kudos
forget about it, the select clause is wrong of course .. too tired I guess sorry
- 0 kudos
- 7403 Views
- 3 replies
- 1 kudos
How to export/clone Databricks Notebook without results via web UI?
When a Databricks Notebook exceeds size limit, it suggests to `clone/export without results`. This is exactly what I want to do, but the current web UI does not provide the ability to bypass/skip the results in either the `clone` or `export` context...
- 7403 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you @Yeshwanth for the response. I am looking for a way without clearing up the current outputs. This is necessary because I want to preserve the existing outputs and fork off another notebook instance to run with few parameter changes and come...
- 1 kudos
- 23937 Views
- 10 replies
- 4 kudos
Resolved! Databricks Asset Bundle (DAB) from a Git repo?
My earlier question was about creating a Databricks Asset Bundle (DAB) from an existing workspace. I was able to get that working but after further consideration and some experimenting, I need to alter my question. My question is now "how do I create...
- 23937 Views
- 10 replies
- 4 kudos
- 16644 Views
- 7 replies
- 4 kudos
Resolved! Databricks SQL endpoint as Linked Service in Azure Data Factory
We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this.
- 16644 Views
- 7 replies
- 4 kudos
- 4 kudos
Azure Databricks Delta Lake (Dataset) uses a linked service that can only connect to a All Purpose/Interactive cluster.If you want to use the SQL Endpoint, you would need a Self Hosted Integration Runtime for ADF with Databricks ODBC driver Installed...
- 4 kudos
- 9063 Views
- 4 replies
- 1 kudos
DLT SCD type 2 in bronze, silver and gold? Is it possible?
I have a question related to when we are using #DLT. Let me try to describe the DLT problem: Objective: Process data end-to-end (bronze, silver gold) using DLTWant bronze to hold a complete raw replica of the data, leveraging apply_changes SCD to wr...
- 9063 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi,At the moment, my process is:-I am using ETL (Data Factory) to land paquet files in a raw landing zone. I keep all the source data here. So I can fully rebuild the data if I need to.i.e.source_system/schema/table/loadon_year=2024/loadon_month=08/l...
- 1 kudos
- 5974 Views
- 1 replies
- 0 kudos
TypeError: ColSpec.__init__() got an unexpected keyword argument 'required'
Hi Team, one of my customer is facing the below issue.. Anyone faced this issue before ? Any help would be appreciated.import mlflowmlflow.set_registry_uri("databricks-uc")catalog_name = "system"embed = mlflow.pyfunc.spark_udf(spark, f"models:/system...
- 5974 Views
- 1 replies
- 0 kudos
- 0 kudos
Upgrade the version of the mlflow package. In 2.7.1 there is no such parameter. https://mlflow.org/docs/2.7.1/python_api/mlflow.types.htmlWhile in the latest (2.17.0) there is such https://mlflow.org/docs/2.17.0/python_api/mlflow.types.html%pip insta...
- 0 kudos
- 2252 Views
- 2 replies
- 1 kudos
Resolved! autoloader error using unity catalog
Hello! I'm new on Databricks and I'm exploring some of its features.I've successfully configured a workspace with unity catalog, one external storage location (ADLSg2) and the associated storage credential. I provided all privileges for all account u...
- 2252 Views
- 2 replies
- 1 kudos
- 1 kudos
hey @garf could you please try to create an external volume using your external location and then use the file path in the volume as the input file path?
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 577 | |
| 349 | |
| 287 |
