- 4348 Views
- 4 replies
- 0 kudos
Feedback on the data quality and consistency checks in Spark
I'm seeking validation from experts regarding the data quality and consistency checks we're implementing as part of a data migration using Spark and Databricks.Our migration involves transferring data from Azure Data Lake to a different data lake. As...
- 4348 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Coders, I'd also consider some profiling checks for column stats and distribution just to be sure everything is consistent after the migration.Afterwards, you should consider the best-practice of implementing some data quality validations on the ...
- 0 kudos
- 6353 Views
- 5 replies
- 3 kudos
What kind of data quality rules that can be run using unity catalog
We are trying to build data quality process for initial file level or data ingestion level for bronze and add more specific business times for silver and business related aggregates for golden layer.
- 6353 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @laksh!You could take a look at Rudol Data Quality, it has native Databricks integration and covers both basic an advanced data quality checks. Basic checks can be configured by non-technical roles using a no-code interface, but there's also the o...
- 3 kudos
- 11033 Views
- 5 replies
- 0 kudos
Data Quality in Databricks
Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.
- 11033 Views
- 5 replies
- 0 kudos
- 2485 Views
- 4 replies
- 1 kudos
Resolved! getting Status code: 301 Moved Permanently error
getting this error while running the cells Failed to upload command result to DBFS. Error message: Status code: 301 Moved Permanently, Error message: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>PermanentRedirect</Code><Message>The bucket you ...
- 2485 Views
- 4 replies
- 1 kudos
- 1 kudos
Same problem and I don't know how to solve.. Here an example of cell that has always worked correctly but from yesterday it stopped.# Compute the correlation matrixcorrelation_matrix = data.corr()# Set up the matplotlib figureplt.figure(figsize=(14, ...
- 1 kudos
- 3687 Views
- 3 replies
- 3 kudos
Resolved! Delta Live Table automatic table removal and schema update
Hello, I made a delta live table workflow that created 3 streaming tables in unity catalog. I then removed the source code for the 3rd table from the workflow and reran. After about a week, the 3rd streaming table is no longer available in unity cata...
- 3687 Views
- 3 replies
- 3 kudos
- 3 kudos
This makes sense @raphaelblg! Just to confirm my understanding, is the following statement true:If I remove the source code for a unity catalog DLT streaming table from a DLT pipeline and wait 7 days, that table will be dropped from unity catalog, an...
- 3 kudos
- 1986 Views
- 2 replies
- 0 kudos
Returing and reusing the identity value
Hello I have a table that has a column defined as an identity (BIGINT GENERATED ALWAYS AS IDENTITY)I will be inserting rows into this table in parallelHow can I get the identity and use that within a pipelineParallel is relevant as there will be mult...
- 1986 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @dpc ,What you're trying to achieve does not make sense in the context of identity columns. Look at below entry from documentation. So, the answer is - if you want to have concurrent transaction, don't use identity columns Declaring an identity co...
- 0 kudos
- 3310 Views
- 4 replies
- 4 kudos
Resolved! Temporary streaming tables (CDC)
I am currently using the `apply_changes` feature. I saw for the regular decorator `dlt.table` you can create temporary tables. I do not see the option you could use this feature with `dlt.create_streaming_table(`, in the sql version it looks it is su...
- 3310 Views
- 4 replies
- 4 kudos
- 4 kudos
Read this articles:Delta Live Tables Python language reference - Azure Databricks | Microsoft LearnThe APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables - Azure Databricks | Microsoft LearnEven you could define as temporary, it ...
- 4 kudos
- 3777 Views
- 3 replies
- 1 kudos
Resolved! Delete folder using Databricks CLI
Hello,I am trying to delete a folder and its content using databricks cli, but I'm getting the following error:databricks workspace delete /Workspace/Users/XXX/XXX --profile DEFAULT --recursive trueError: expected to have the absolute path of the not...
- 3777 Views
- 3 replies
- 1 kudos
- 14355 Views
- 4 replies
- 6 kudos
Resolved! Redirect error in access to web app in Azure Databricks with private front endpoint
I have created a workspace with private endpoint in Azure following this guide:https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-linkOnce I have created the private link of type browser_authent...
- 14355 Views
- 4 replies
- 6 kudos
- 6 kudos
You don't need a CNAME record.Go to your private link resource in Azure and click on Settings > DNS Configuration. Make sure you have created private link A records for all the FQDNs listed under 'Custom DNS records'. You have most likely missed one ...
- 6 kudos
- 1718 Views
- 2 replies
- 3 kudos
Resolved! Databricks DLT with Hive Metastore and ADLS Access Issues
We are currently working on Databricks DLT tables to transform data from bronze to silver. we are specifically instructed us not to use mount paths for accessing data from ADLS Gen 2. To comply, I configured storage credentials and created an externa...
- 1718 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @yvishal519 ,Since you're using hive metastore you have no other option than mount points. Storage credentials and external locations are only supported in Unity Catalog
- 3 kudos
- 1458 Views
- 3 replies
- 3 kudos
Unavailable system schemas
When I list the available schemas I get the following:{"schemas":[{"schema":"storage","state":"AVAILABLE"},{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"access","state":"AVAILABLE"},{"schema":"billing","state":"ENABLE_COMPLETED"},{"s...
- 1458 Views
- 3 replies
- 3 kudos
- 3 kudos
I have the same issue for the compute schema. Workspace is UC enabled and I'm account admin.
- 3 kudos
- 902 Views
- 1 replies
- 0 kudos
Auto-tuning capability available for external tables?
If I am using Databricks Runtime 11.3 and above to create managed Delta tables cataloged in Unity Catalog (Databricks’ data catalog), I don’t need to worry about optimizing the underlying file sizes or configuring a target file size for my Delta tabl...
- 902 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Amit_Dass_Chmp ,Yep, according to documentation. As of second question, such capability will be available in the future. If you are using Databricks Runtime 11.3 and above to create managed Delta tables cataloged in Unity Catalog (Databricks’ dat...
- 0 kudos
- 3237 Views
- 3 replies
- 4 kudos
Resolved! Approach to monthly data snapshots
HelloI'm building a datawarehouse with all the usual facts and dimensionsIt will flush (truncate) and rebuild on a monthly basisUsers have the need to not only view the data now but also view it historically i.e. what it was a point in timeMy initial...
- 3237 Views
- 3 replies
- 4 kudos
- 1901 Views
- 3 replies
- 3 kudos
Resolved! getting error while accessing dbfs from databricks community account and couldnt upload any files
Hi, I have enabled dbfs in my databricks community account and started the cluster. while accessing dbfs its throwing an error.
- 1901 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Retired_mod I am no longer facing this issue, Thanks to the team for fixing it !
- 3 kudos
- 1842 Views
- 2 replies
- 0 kudos
DatabricksSession and SparkConf
Hi,I want to initialize a Spark session using `DatabricksSession`. However, it seems not possible to call `.config()` and pass it a `SparkConf` instance. The following works:# Initialize the configuration for the Spark session confSettings = [ ("...
- 1842 Views
- 2 replies
- 0 kudos
- 0 kudos
In almost all cases you don't need to create a new spark session, as Databricks will do it for you automatically.If it's only about spark configurations, there are multiple ways to set it:Cluster settingsspark.conf.set
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
