- 12953 Views
- 9 replies
- 2 kudos
Is Delta Live Tables not supported anymore? How do I use it in Python?
Hi!Any time I try to import "dlt" in a notebook session to develop Pipelines, I get an error message saying DLT is not supported on Spark Connect clusters. These are very generic clusters, I've tried runtime 14, 15 and the latest 16, using shared clu...
- 12953 Views
- 9 replies
- 2 kudos
- 2 kudos
Oakhill, we do provide free onboard training. You might be interested in the "Get Started with Data Engineering on Databricks" session. You can register here: https://www.databricks.com/training/catalog. When you are searching the catalog of traini...
- 2 kudos
- 2226 Views
- 1 replies
- 0 kudos
DLT Autoloader stuck in reading Avro files from Azure blob storage
I have a DLT pipeline joining data from streaming tables to metadata of Avro files located in Azure blob storage. The avro files are loaded using autoloader. Up until 25.3. (about 20:00UTC) the pipeline worked fine, but then suddenly got stuck in ini...
- 2226 Views
- 1 replies
- 0 kudos
- 0 kudos
Based off of your screenshot, a Spark job has started, and 33/34 tasks are completed. This is usually indicative of some kind of skewed processing. Please refer to this documentation for help identifying and resolving skew
- 0 kudos
- 44678 Views
- 1 replies
- 1 kudos
Autoloader exclude one directory
Hi,I have a bunch of csv files in directories within an azure blob container and I am using autoloader to ingest them into a raw (bronze) table, all csvs apart from one have the same schema. Is there a way to get autoloader to ignore the directory wi...
- 44678 Views
- 1 replies
- 1 kudos
- 1 kudos
Auto Loader accept globs as input, including negative globs. You can use this to exclude a directory as long as the path is known ahead of time
- 1 kudos
- 2029 Views
- 1 replies
- 0 kudos
VACUUM seems to be deleting Autoloader's log files.
Hello everyone,I have a workflow setup that updates a few Delta tables incrementally with autoloader three times a day. Additionally, I run a separate workflow that performs VACUUM and OPTIMIZE on these tables once a week.The issue I'm facing is that...
- 2029 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message suggests that autoloader's state is being improperly deleted, most likely by a separate process. If your checkpoint exists inside of the root of a delta table, then VACUUM can delete its files. Make sure that you do not store checkp...
- 0 kudos
- 1102 Views
- 1 replies
- 0 kudos
Connectivity failure to on-prem databases
Hi All,We have more than 100 jobs right now in Databricks which connect to on-prem database like Oracle. Connection to oracle is made through notebooks using jdbc thin client and using com.oracle.ojdbc:ojdbc10:19.3.0.0 or com.oracle.ojdbc:ojdbc8:19.3...
- 1102 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you have any custom DNS in your set up? If yes, are you aware of any changes being performed on the same pointing to the databases?
- 0 kudos
- 1077 Views
- 1 replies
- 0 kudos
How to increase autoloader speed while working with s3 (AWS)
Hey everyone! I am trying to switch from a batch processing job to a autoloader (directory listing mode) on my s3 bucket that has millions of files. I am using modifiedAfter to create my initial checkpoint for the autoloader and want to speed up the ...
- 1077 Views
- 1 replies
- 0 kudos
- 0 kudos
What mode are you using? File notification?
- 0 kudos
- 3328 Views
- 2 replies
- 2 kudos
How to connect GENIE Space output Visualize to AI/BI dashboards
I am looking the way to connect the GENIE Space to AI/BI dashboard to link SQL QUERY Visualizations, currently i am not able to do that.
- 3328 Views
- 2 replies
- 2 kudos
- 2 kudos
thanks for sharingYes , but I am looking other way around...GENIE Space rooms ---> Add the Generated Query ( Visualize) ---? to my AI/BI dashboard.Looks like I am doing the wrong way, the best way to create AI/BI dashboards and Genie use with that. L...
- 2 kudos
- 5394 Views
- 4 replies
- 5 kudos
- 5394 Views
- 4 replies
- 5 kudos
- 5 kudos
Hello everyone,How did you solve this issue? Did you manage to get the secrets through global init script?Thanks.
- 5 kudos
- 1895 Views
- 1 replies
- 0 kudos
Autoloader: Read old version of file. Read modification time is X, latest modification time is X
I'm recieving this error from autoloader. It seems to be stuck on this one file. I don't care when it was read and last modified, I just want to ingest it. Any ideas?java.io.IOException: Read old version of file s3a://<file-path>.json. Read modificat...
- 1895 Views
- 1 replies
- 0 kudos
- 0 kudos
@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read. Please refe...
- 0 kudos
- 4223 Views
- 2 replies
- 1 kudos
AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS
Spark write method (df.write.parquet) to parquet files is transactional. I mean after write is sucessfull file _SUCCESS is created in path where parquet files was loaded.Is it possible to configure AutoLoader to load parquet files only in case when w...
- 4223 Views
- 2 replies
- 1 kudos
- 1 kudos
@Marcin_U Please use the below option in the readStream to load only parquet files .option("pathGlobfilter", "*.parquet") Please refer to the below documentation: https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/options.html...
- 1 kudos
- 3398 Views
- 3 replies
- 0 kudos
Connecting AWS MSK from Databricks
Hello ,I am new to AWS and MSK. I have created MSK on VPC with public subnet and I am trying to connect it from Databricks on AWS. I see that both MSK and VPC are in two different VPC. Do we need to create VPC peering to establish the connection be...
- 3398 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello,Thanks for your quick update. I will continue with the setup. I also have one more question. It is mentioned in the set up that, we have to have the bellow in the Kafka client properties.In addition, if you choose to configure your connection ...
- 0 kudos
- 1075 Views
- 2 replies
- 0 kudos
Updates are going as insert in a databricks job
There is no code change in a databricks job or notebook but we have observed the malfunction in cdc. The records which are meant for updates are going for inserts and causing duplicacy. At the sametime we had make sure the PKs based on which merge ru...
- 1075 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sashikanth, I would recommend opening a case with us further investigate this behavior since this requires a code logic review and additional checks. Please refer to: https://docs.databricks.com/en/resources/support.html
- 0 kudos
- 1803 Views
- 1 replies
- 1 kudos
Databricks ODBC driver tooks long time to list columns
I'm testing the performance of databricks, when I use ODBC driver to submit query, I found it slower than notebook,and it looks like odbc driver will send "list columns" request to databricks, 1. Is there a way to prevent the ODBC driver from sending...
- 1803 Views
- 1 replies
- 1 kudos
- 1 kudos
@Youngwb Did you fixed this issue?
- 1 kudos
- 21935 Views
- 7 replies
- 3 kudos
Resolved! Delta Live Tables error pivot
I'm facing an error in Delta Live Tables when I want to pivot a table. The error is the following: And the code to replicate the error is the following:import pandas as pd import pyspark.sql.functions as F pdf = pd.DataFrame({"A": ["foo", "foo", "f...
- 21935 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi, Was this a specific design choice to not allow Pivots in DLT? I'm under the impression they expect fixed table structures in DLT design for a reason, but I don't understand the reason? Conceptually, I understand the fixed structures makes lineage...
- 3 kudos
- 1854 Views
- 2 replies
- 1 kudos
Resolved! Getting
payload = {"clusters": [{"num_workers": 4}],"pipeline_id": pipeline_id}update_url = f"{workspace_url}/api/2.0/pipelines/{pipeline_id}"response = requests.put(update_url, headers=headers, json=payload)for this, i'm getting below output with status cod...
- 1854 Views
- 2 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
