- 4223 Views
- 7 replies
- 7 kudos
I have S3 as a data source containing sample TPC dataset (10G, 100G).I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?I also need the data to be partitioned. And withi...
- 4223 Views
- 7 replies
- 7 kudos
Latest Reply
Hi @Vikas Goel We haven't heard from you since the last response from @Werner Stinckens , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...
6 More Replies
- 2124 Views
- 5 replies
- 0 kudos
Is it possible to connect and execute query directly from rds in sql editor without using unity catelog
- 2124 Views
- 5 replies
- 0 kudos
Latest Reply
Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks
4 More Replies
- 568 Views
- 2 replies
- 0 kudos
I recently started exploring the field of Data Engineering and came across some difficulties. I have a bucket in GCS with millions of parquet files and I want to create an Anomaly Detection model with them. I was trying to ingest that data into Datab...
- 568 Views
- 2 replies
- 0 kudos
Latest Reply
@Pedro Barbosa :It seems like you are running out of memory when trying to convert the PySpark dataframe to an H2O frame. One possible approach to solve this issue is to partition the PySpark dataframe before converting it to an H2O frame.You can us...
1 More Replies
by
ramz
• New Contributor II
- 1819 Views
- 4 replies
- 1 kudos
Hi, I am using pyspark and i am reading a bunch of parquet files and doing the count on each of them. Driver memory shoots up about 6G to 8G. My setup:I have a cluster of 1 driver node and 2 worker node (all of them 16 core 128 GB RAM). This is th...
- 1819 Views
- 4 replies
- 1 kudos
Latest Reply
Hi @ramz siva Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
3 More Replies
- 3060 Views
- 1 replies
- 0 kudos
ProblemI have a directory in S3 with a bunch of data files, like "data-20221101.parquet". They all have the same columns: timestamp, reading_a, reading_b, reading_c. In the earlier files, the readings are floats, but in the later ones they are double...
- 3060 Views
- 1 replies
- 0 kudos
Latest Reply
1) Can you let us know what was the error message when you don't set the schema & use mergeSchema2) What happens when you define schema (with FloatType) & use mergeSchema ? what error message do you get ?
- 3619 Views
- 6 replies
- 2 kudos
Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...
- 3619 Views
- 6 replies
- 2 kudos
Latest Reply
It's been a while since the question was asked, and in the meantime Delta Lake 2.2.0 hit the shelves with the exact feature the OP asked about, i.e. LIMIT pushdown:LIMIT pushdown into Delta scan. Improve the performance of queries containing LIMIT cl...
5 More Replies
- 1905 Views
- 3 replies
- 4 kudos
Spark 3.3.1 supports the brotli compression codec, but when I use it to read parquet files from S3, I get:INVALID_ARGUMENT: Unsupported codec for Parquet page: BROTLIExample code:df = (spark.read.format("parquet")
.option("compression", "brotli")...
- 1905 Views
- 3 replies
- 4 kudos
Latest Reply
Given the new information I appended, I looked into the Delta caching and I can disable it:.option("spark.databricks.io.cache.enabled", False)This works as a work around while I read these files in to save them locally in DBFS, but does it have perfo...
2 More Replies
- 2442 Views
- 4 replies
- 7 kudos
In my findings I have found a lot of delta tables in the lake house to be sparse so just wondering what space data lake takes to store null data and also any suggestions to handle sparse data tables in lake house would be appreciated.I also want to o...
- 2442 Views
- 4 replies
- 7 kudos
Latest Reply
Hi @Akash Ragothu, We haven’t heard from you since the last response from @Ajay Pandey, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to othe...
3 More Replies
- 1704 Views
- 2 replies
- 17 kudos
I'm thinking of using autoloader to process files being put on our data lake.Let's say f.e. every 15 minutes, a parquet files is written. These files however contain overlapping data.Now, every 2 hours I want to process the new data (autoloader) and...
- 1704 Views
- 2 replies
- 17 kudos
Latest Reply
What about forEachBatch and then MERGE?Alternatively, run another process that will clean updates using the window function, as you said.
1 More Replies
by
elgeo
• Valued Contributor II
- 574 Views
- 0 replies
- 3 kudos
Hello. We would like to understand how many parquet files are created per data table. To be more specific, we refer to the current snapshot of the table. For example, we noticed that while we performed initial inserts to a table, one parquet file was...
- 574 Views
- 0 replies
- 3 kudos
- 3979 Views
- 4 replies
- 4 kudos
When i tried to read file from dbfs, it throws error - Caused by: FileReadException: Error while reading file dbfs:/.......................parquet is not a Parquet file. Expected magic number at tail [80, 65, 82, 49] but found [105, 108, 101, 115].Bu...
- 3979 Views
- 4 replies
- 4 kudos
Latest Reply
Hi @KARTHICK N, What's the one-line code you're trying to read the file, precisely the path?Can you confirm if your file is a CSV or Parquet file?Are you trying to read it in python or scala?
3 More Replies
- 686 Views
- 0 replies
- 4 kudos
Hi Team,I am trying to run a streaming job in databricks, used Autoloader approach for reading the files from the Azure Datalake Gen2 which is in parquet format. I have created a new checkpoint, so first offset is getting created but throwing an erro...
- 686 Views
- 0 replies
- 4 kudos
by
Mayank
• New Contributor III
- 4133 Views
- 8 replies
- 4 kudos
I am trying to load parquet files using Autoloader. Below is the code def autoload_to_table (data_source, source_format, table_name, checkpoint_path):
query = (spark.readStream
.format('cloudFiles')
.option('cl...
- 4133 Views
- 8 replies
- 4 kudos
Latest Reply
Hi again @Mayank Srivastava Thank you so much for getting back to us and marking the answer as best.We really appreciate your time.Wish you a great Databricks journey ahead!
7 More Replies
- 11116 Views
- 3 replies
- 2 kudos
- 11116 Views
- 3 replies
- 2 kudos
Latest Reply
Hello @Karl Saycon Can you try setting this config to prevent additional parquet summary and metadata files from being written? The result from dataframe write to storage should be a single file.https://community.databricks.com/s/question/0D53f00001...
2 More Replies
- 4717 Views
- 4 replies
- 4 kudos
I am trying to access the Azure Blob table using Pyspark but getting an Authentication Error. Here I am passing SAS token (HTTP and HTTPS enabled) but it's working only with WASBS (HTTPS) URL, not with WASB (HTTP) URL.Even I tried with Account key as...
- 4717 Views
- 4 replies
- 4 kudos
Latest Reply
Hi @Arvind Ravish The issue got fixed after passing HTTP and HTTPS enabled token to spark executors.Thanks again for your help
3 More Replies
