- 1476 Views
- 1 replies
- 0 kudos
Autoloader: Read old version of file. Read modification time is X, latest modification time is X
I'm recieving this error from autoloader. It seems to be stuck on this one file. I don't care when it was read and last modified, I just want to ingest it. Any ideas?java.io.IOException: Read old version of file s3a://<file-path>.json. Read modificat...
- 1476 Views
- 1 replies
- 0 kudos
- 0 kudos
@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read. Please refe...
- 0 kudos
- 3346 Views
- 2 replies
- 1 kudos
AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS
Spark write method (df.write.parquet) to parquet files is transactional. I mean after write is sucessfull file _SUCCESS is created in path where parquet files was loaded.Is it possible to configure AutoLoader to load parquet files only in case when w...
- 3346 Views
- 2 replies
- 1 kudos
- 1 kudos
@Marcin_U Please use the below option in the readStream to load only parquet files .option("pathGlobfilter", "*.parquet") Please refer to the below documentation: https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/options.html...
- 1 kudos
- 2030 Views
- 3 replies
- 0 kudos
Connecting AWS MSK from Databricks
Hello ,I am new to AWS and MSK. I have created MSK on VPC with public subnet and I am trying to connect it from Databricks on AWS. I see that both MSK and VPC are in two different VPC. Do we need to create VPC peering to establish the connection be...
- 2030 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello,Thanks for your quick update. I will continue with the setup. I also have one more question. It is mentioned in the set up that, we have to have the bellow in the Kafka client properties.In addition, if you choose to configure your connection ...
- 0 kudos
- 664 Views
- 2 replies
- 0 kudos
Updates are going as insert in a databricks job
There is no code change in a databricks job or notebook but we have observed the malfunction in cdc. The records which are meant for updates are going for inserts and causing duplicacy. At the sametime we had make sure the PKs based on which merge ru...
- 664 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sashikanth, I would recommend opening a case with us further investigate this behavior since this requires a code logic review and additional checks. Please refer to: https://docs.databricks.com/en/resources/support.html
- 0 kudos
- 1298 Views
- 1 replies
- 1 kudos
Databricks ODBC driver tooks long time to list columns
I'm testing the performance of databricks, when I use ODBC driver to submit query, I found it slower than notebook,and it looks like odbc driver will send "list columns" request to databricks, 1. Is there a way to prevent the ODBC driver from sending...
- 1298 Views
- 1 replies
- 1 kudos
- 1 kudos
@Youngwb Did you fixed this issue?
- 1 kudos
- 15688 Views
- 7 replies
- 3 kudos
Resolved! Delta Live Tables error pivot
I'm facing an error in Delta Live Tables when I want to pivot a table. The error is the following: And the code to replicate the error is the following:import pandas as pd import pyspark.sql.functions as F pdf = pd.DataFrame({"A": ["foo", "foo", "f...
- 15688 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi, Was this a specific design choice to not allow Pivots in DLT? I'm under the impression they expect fixed table structures in DLT design for a reason, but I don't understand the reason? Conceptually, I understand the fixed structures makes lineage...
- 3 kudos
- 1207 Views
- 2 replies
- 1 kudos
Resolved! Getting
payload = {"clusters": [{"num_workers": 4}],"pipeline_id": pipeline_id}update_url = f"{workspace_url}/api/2.0/pipelines/{pipeline_id}"response = requests.put(update_url, headers=headers, json=payload)for this, i'm getting below output with status cod...
- 1207 Views
- 2 replies
- 1 kudos
- 877 Views
- 2 replies
- 0 kudos
ETL job execution failure in serverless compute
Hi,We facing an issue when executing the ELT pipeline on serverless compute. In the ETL pipeline, for some users, a task gets stuck every time we run the job. However, a similar ETL pipeline works fine for other users. Furthermore, there are canceled...
- 877 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,Could you please share the error message or any logs you’re seeing when the task gets stuck? This will help in diagnosing the issue and identifying potential solutions.
- 0 kudos
- 1305 Views
- 1 replies
- 0 kudos
Auto Loader
from pyspark.sql.types import StructType, StructField, LongType, StringType, TimestampTypefrom pyspark.sql import functions as F, Windowfrom delta.tables import DeltaTableimport logging# Set up logginglogging.basicConfig(level=logging.INFO)logger = l...
- 1305 Views
- 1 replies
- 0 kudos
- 0 kudos
If you intend to capture data changes, take a look at this doc, which talks about change data feed in Databricks.
- 0 kudos
- 920 Views
- 2 replies
- 1 kudos
Historical Migration of Data from Delta Tables to Delta Live Tables
Our team is planning to migrate to the Delta Live Tables (DLT) framework for data ingestion. We currently have Delta tables populated with several years of data from ingested files and wish to avoid re-ingesting these files. What is the best approach...
- 920 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @nalindhar I assume you want the target table to have the same name once you declare your ETL operations in the DLT pipeline, if that's the case, begin with renaming your delta table to specify that they contain historical data, you can do this v...
- 1 kudos
- 1788 Views
- 1 replies
- 0 kudos
Bloomberg API and Databricks
Dear Databricks Community,Do you have any experience connecting Databricks (Python) to the Bloomberg Terminal to retrieve data in an automated way? I tried running the following code without success:%python%pip install blpapiThanks for your help,Kevi...
- 1788 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kfloresip ,As per documentation you need to run:%pip install --index-url=https://blpapi.bloomberg.com/repository/releases/python/simple/ blpapi It means that the package is not available on the standard Python Package Index (PyPI) and Bloomberg h...
- 0 kudos
- 4725 Views
- 6 replies
- 1 kudos
Resolved! Does Databricks supports the Pytorch Distributed Training for multiple devices?
Hi, Im trying to use the databricks platform to do the pytorch distributed training, but I didnt find any info about this. What I expected is using multiple clusters to run a common job using pytorch distributed data parallel (DDP) with the code belo...
- 4725 Views
- 6 replies
- 1 kudos
- 1 kudos
If only the driver is active, this probably means you are not using Spark. When running pure python,... code, the driver will execute that.If Spark is active, workers receive their tasks from the driver. Generally the driver is not that active, the...
- 1 kudos
- 808 Views
- 1 replies
- 0 kudos
Can i send multiple post requests to an API endpoint and get the info if all succeeded ?
Hello I am trying to send multiple post requests to an endpoint, i have a spark dataframe and each column of this dataframe is sent through the payload of the post request.However when i run this in my notebook, no exception is raised. I'm guessing i...
- 808 Views
- 1 replies
- 0 kudos
- 0 kudos
The return type for foreachPartition is None, so this is expected. If you're looking to do arbitrary code execution and return a result, mapInPandas or Pandas UDFs are good choices - you'd want to combine those with something like a .toLocalIterator ...
- 0 kudos
- 3408 Views
- 8 replies
- 4 kudos
How to execute .sql file in volume
I have giant queries (SELECT.. FROM) that i store in .sql files. I want to put those files in the Volume, and run the queries from a workflow task.I can load the file content into a 'text' format string, then run the query. My question is, is there...
- 3408 Views
- 8 replies
- 4 kudos
- 4 kudos
issue resolved:for .py, i was using spark, and I have to explicitly create the spark session so that it can be run properly and insert data.
- 4 kudos
- 1501 Views
- 2 replies
- 1 kudos
Resolved! How can I force a data frame to evaluate without saving it?
The problem in my hand requires me to take a set of actions on a very large data frame df_1. This set of actions results in a second data frame df_2, and from this second data frame, I have multiple downstream tasks, task_1, task_2 ... By default, t...
- 1501 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @100databricks,Hi, yes, you can run df_2.cache() or df_2.persist()(df_2.cache() is a shortcut for df_2.persist(StorageLevel.MEMORY_ONLY)Here is the pseudo-code:# df_1 is your large initial DataFrame df_1 = ... # Perform expensive transformations ...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
