- 2181 Views
- 1 replies
- 0 kudos
- 2181 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Live Table supports the data quality checks via expectations. On encountering invalid records you can choose to either retain them, drop them or fail/stop the pipeline. See the link below for additional detailshttps://docs.databricks.com/data-e...
- 0 kudos
- 1947 Views
- 1 replies
- 0 kudos
- 1947 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes. You can specify a "target" database as part of your DLT pipeline configuration to publish results to a target database in the metastore. See - https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-quickstart.html#publi...
- 0 kudos
- 1586 Views
- 1 replies
- 0 kudos
- 1586 Views
- 1 replies
- 0 kudos
- 0 kudos
DLT Pipeline results are published to the "Storage Location" defined as part of configuring the Pipeline. Ex:- https://docs.databricks.com/_images/dlt-create-notebook-pipeline.pngIf an explicit Storage Location is not specified, the pipeline results ...
- 0 kudos
- 2218 Views
- 1 replies
- 1 kudos
- 2218 Views
- 1 replies
- 1 kudos
- 1 kudos
Notebooks with Delta Live Table/View definition just contain the pipeline definition. In order to execute Delta Live Tables Notebooks you need to define a Pipeline via the Jobs UI. Pipeline carries with it the logic to build the dependency graph betw...
- 1 kudos
- 2357 Views
- 1 replies
- 1 kudos
- 2357 Views
- 1 replies
- 1 kudos
- 1 kudos
You could have the single node cluster where proxy is installed monitored by one of the tools like cloudwatch, azure monitor, datadog etc and have it configured to send alerts on node failure
- 1 kudos
- 1701 Views
- 1 replies
- 0 kudos
DBFS root resides in the Customer account or Databricks Account
IF I installed the root Bucket I see a root bucket is created with workspace, Does this bucket resided in Customer account or Databricks Account. How can I Access the bucket and can i see this bucket directly in s3 or ADLS
- 1701 Views
- 1 replies
- 0 kudos
- 0 kudos
Didin't get the reference about installing bucket ? did you mean configured a workspace with root bucket. If so, you'd have probably gathered that root storage for a workspace resides in customer's account
- 0 kudos
- 5886 Views
- 2 replies
- 1 kudos
- 5886 Views
- 2 replies
- 1 kudos
- 1 kudos
Delta cache accelerates data reads by creating copies of remote files in nodes’ local storage using a fast intermediate data format. The data is cached automatically whenever a file has to be fetched from a remote location. Successive reads of the sa...
- 1 kudos
- 4744 Views
- 1 replies
- 1 kudos
Converting data that is in Delta format to plain parquet format
Many a times there is a need to convert Delta tables from Delta format to plain parquet format for a number of reasons, what is the best way to do that?
- 4744 Views
- 1 replies
- 1 kudos
- 1 kudos
You can easily convert a Delta table back to a Parquet table using the following steps:If you have performed Delta Lake operations that can change the data files (for example, delete or merge, run vacuum with retention of 0 hours to delete all data f...
- 1 kudos
- 6260 Views
- 1 replies
- 0 kudos
Metaexception [Version information not found in metastore] during cluster [re]start
Trying to configure new external metastore and running into the following exception during cluster initialization - Caused by: MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.RetryingHMSHandl...
- 6260 Views
- 1 replies
- 0 kudos
- 0 kudos
The above exception happens when the hive schema is not available in the metastore instance. Please check in your init scripts to make sure the following flag is enabled to create hive Schema and tables if not already present. datanucleus.autoCreateA...
- 0 kudos
- 2127 Views
- 1 replies
- 0 kudos
- 2127 Views
- 1 replies
- 0 kudos
- 0 kudos
The below code snippet can be used to get the DBR details on a HC clusterprint("hadoopVersion:" + sc._gateway.jvm.org.apache.hadoop.util.VersionInfo.getVersion()) print("baseVersion:" + sc._gateway.jvm.org.apache.spark.BuildInfo.sparkBranch()) print(...
- 0 kudos
- 1508 Views
- 1 replies
- 0 kudos
- 1508 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks notebooks can be exported and stored in S3 or any other object storage. The internal storage of the databricks notebook cannot be changed or configured. The implementation is internal to Databicks control plane and not user configurable.
- 0 kudos
- 4108 Views
- 1 replies
- 0 kudos
- 4108 Views
- 1 replies
- 0 kudos
- 0 kudos
The below code snippet is useful to get the modification time of files. %scala import scala.util.Try import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.hadoop.io.IOUtils import java.io.IOExcep...
- 0 kudos
- 7730 Views
- 2 replies
- 0 kudos
Resolved! How and when to capture the thread dump of the Spark driver?
What is the best way to capture the thread dump of the Spark driver process. Also, when should I capture the thread dump?
- 7730 Views
- 2 replies
- 0 kudos
- 0 kudos
For Spark driver the process is the same. Choose the driver from the Executor page and view the thread dump. A thread dump is the footprints of the JVM they are very useful in debugging issues where the JVM process is stuck or making extremely slow p...
- 0 kudos
- 3306 Views
- 2 replies
- 0 kudos
Resolved! Autoloader: How to identify the backlog in RocksDB
With S3-SQS it was easier to identify the backlog ( the messages that are fetched from SQS and not consumed by the streaming job) How to find the same with Auto-loader
- 3306 Views
- 2 replies
- 0 kudos
- 0 kudos
For DBR 8.2 or later, the backlog details are captured in the Streaming metricsEg:
- 0 kudos
- 3780 Views
- 1 replies
- 0 kudos
Resolved! How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?
I am running jobs on Databricks using the Run Submit API with Airflow. I have noticed that rarely, a particular run is run more than one time at once. Why?
- 3780 Views
- 1 replies
- 0 kudos
- 0 kudos
Idempotency can be ensured by providing the idempotency token. It's easy to pass the same through REST API as mentioned in the below doc:https://kb.databricks.com/jobs/jobs-idempotency.htmlThe primary reason for multiple runs is the client submits t...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
