- 6917 Views
- 2 replies
- 1 kudos
Resolved! Log has failed integrity check error when altering a table property
Below is the integrity check error we are getting when trying to set the deletedRetentionFileDuration table property to 10 days. Observation: The table data is sitting in S3. The size of all the files in S3 is in TB. There are millions of files for t...
- 6917 Views
- 2 replies
- 1 kudos
- 1 kudos
Please backup your table, then run the repair of filesFSCK REPAIR TABLE table_nameyou can also try to make dry run firstFSCK REPAIR TABLE table_name DRY RUNif data is partitioned can be helpful to refresh metastoreMSCK REPAIR TABLE mytable
- 1 kudos
- 2779 Views
- 2 replies
- 0 kudos
How to pass job task parameters to another task in scala
Hi Team,​I have a requirement in workflow job. Job has two tasks, one is python-task and another one is scala-task (both are running their own cluster).​I have defined dbutils.job.taskValue in python which is not able to read value in scala because o...
- 2779 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Sreekanth Nallapa​ please refer this link This might help you in this
- 0 kudos
- 5610 Views
- 1 replies
- 5 kudos
Optimal Batch Size for Batch Insert Queries using JDBC for Delta Tables
Hi,I am currently experimenting with databricks-jdbc : 2.6.29 and trying to execute batch insert queries What is the optimal batch size recommended by Databricks for performing Batch Insert queries?Currently it seems that values are inserted row by r...
- 5610 Views
- 1 replies
- 5 kudos
- 5 kudos
Just an observation : By using auto optimize table level property, I was able to see batch inserts inserting records in single file.https://docs.databricks.com/optimizations/auto-optimize.html
- 5 kudos
- 13412 Views
- 14 replies
- 9 kudos
Suggestion Needed for a Orchestrator/Scheduler to schedule and execute Jobs in an automated way
Hello Friends,We have an application which extracts dat from various tables in Azure Databricks and we extract it to postgres tables (postgres installed on top of Azure VMs). After extraction we apply transformation on those datasets in postgres tabl...
- 13412 Views
- 14 replies
- 9 kudos
- 9 kudos
You can leverage Airflow, which provides a connector for databricks jobs API, or can use databricks workflow to orchestrate your jobs where you can define several tasks and set dependencies accordingly.
- 9 kudos
- 12486 Views
- 7 replies
- 5 kudos
Resolved! Custom library import fails randomly with error: not found: value it
Hello,I have an issue with the import of a custom library, in Azure Databricks.(roughly) 95% of the times it works fine, but sometimes it fails.I searched the internet and this community with no luck, so far.It is a scala library in a scala notebook,...
- 12486 Views
- 7 replies
- 5 kudos
- 5 kudos
Even I also encountered the same error. While Importing a file getting an error as "Import failed with error: Could not deserialize: Exceeded 16777216 bytes (current = 16778609)"
- 5 kudos
- 4844 Views
- 2 replies
- 0 kudos
Resolved! Configure Autoloader with the file notification mode for production
I configured ADLS Gen2 standard storage and successfully configured Autoloader with the file notification mode.In this documenthttps://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html"ADLS Gen2 provides different event notificati...
- 4844 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, @Chris Konsur​. You do not need anything with the FlushWithClose event REST API that is just the event type that we listen to. As for backfill setting, this is for handling late data or late event that are being triggered. This setting largely de...
- 0 kudos
- 3325 Views
- 4 replies
- 1 kudos
"approxQuantile" not working as part of a delta live table workflow pipeline.
I am trying to compute outliers using approxQuantile on a Dataframe. It works fine in a Databricks notebook, but the call doesn't work correctly when it is part of a delta live table pipeline. (This is python.) Here is the line that isn't working a...
- 3325 Views
- 4 replies
- 1 kudos
- 1 kudos
Good recommendation. I was able to do something similar that appears to work.# Step 2. Compute the is outlier sessions based on duration_minutes lc = session_agg_df.selectExpr("percentile(duration_minutes, 0.25) lower_quartile") session_agg_d...
- 1 kudos
- 1219 Views
- 0 replies
- 0 kudos
www.meetup.com
DEC 13 MEETUP: Arbitrary Stateful Stream Processing in PySparkFor folks in the Bay Area- Dr. Karthik Ramasamy, Databricks' Head of Streaming, will be joined by engineering experts on the streaming and PySpark teams at Databricks for this in-person me...
- 1219 Views
- 0 replies
- 0 kudos
- 4081 Views
- 3 replies
- 5 kudos
Weekly Release Notes Recap�� Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notes...
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notes December 1-6, 2022Partner Connect supports connecting to AtScale:You can now easily create a connection betwe...
- 4081 Views
- 3 replies
- 5 kudos
- 5 kudos
@Uma Maheswara Rao Desula​ if i am not wrong, below ideas portal should help you Ideas Portal | Databricks on AWS
- 5 kudos
- 8567 Views
- 4 replies
- 7 kudos
How to properly convert DUB's consumed into Doller Amount in Databricks AWS/GCP/AZURE
HI Team,I have gone through Lot of articles, but it looks there is some gap on pricing. can anyone please let me know accurate way to calculate DBU Pricing into dollarsas per my understandingTotal DBU Cost: DBU /hour * total job ran in hours (Shows a...
- 8567 Views
- 4 replies
- 7 kudos
- 5236 Views
- 4 replies
- 9 kudos
Construct Dataframe or RDD from S3 bucket with Delta tables
Hi all! I have an S3 bucket with Delta parquet files/folders with different schemas each. I need to create an RDD or DataFrame from all those Delta Tables that should contain the path, name and different schema of each.How could I do that?Thank you!P...
- 5236 Views
- 4 replies
- 9 kudos
- 9 kudos
You can mount S3 bucket or read directly from it.access_key = dbutils.secrets.get(scope = "aws", key = "aws-access-key") secret_key = dbutils.secrets.get(scope = "aws", key = "aws-secret-key") sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", ac...
- 9 kudos
- 8112 Views
- 2 replies
- 3 kudos
%run not printing notebook output when using 'Run All' command
I have been using the %run command to run auxiliary notebooks from an "orchestration" notebook. I like using %run over dbutils.notebook.run because of the variable inheritance, troubleshooting ease, and the printing of the output from the auxiliary n...
- 8112 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Aaron Petry​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks
- 3 kudos
- 5620 Views
- 5 replies
- 29 kudos
Resolved! databricks community
I have points in databricks community 461 but in reward store only reflecting 23 points can any one look into this issue
- 5620 Views
- 5 replies
- 29 kudos
- 29 kudos
Hi rewards account needs to be created with same email id and points may take a week to reflect in your rewards account
- 29 kudos
- 6448 Views
- 4 replies
- 8 kudos
Databricks Runtime Support
How Long are Databricks runtimes supported for? How often are they updated?You can learn more about the Databricks runtime support lifecycle here (AWS | Azure | GCP).Long Term Support (LTS) runtimes are released every 6 months and supported for 2 yea...
- 6448 Views
- 4 replies
- 8 kudos
- 10411 Views
- 7 replies
- 11 kudos
Data bricks SQL is allowing 10 queries only ?
•Power BI is a publisher that uses AD group authentication to publish result sets. Since the publisher's credentials are maintained, the same user can access the data bricks database.•Number of the users are retrieving the data from the power bi or i...
- 10411 Views
- 7 replies
- 11 kudos
- 11 kudos
I believe 10 is a limit as of now. See if you can increase the concurrency limit from the source.
- 11 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
