- 315 Views
- 1 replies
- 1 kudos
Retrieve data older than the one year window : system.access.table_lineage
Hello,I am currently using table_lineage from system.access.table_lineage. It is a great feature but I am experiencing missing data. After some search I have seen that "Because lineage is computed on a one-year rolling window, lineage collected more ...
- 315 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Clara ,I don't think so. But you can build history such history tables by yourself. Desing ETL process that will extract data from system tables and store them in your own data tables.
- 1 kudos
- 490 Views
- 2 replies
- 1 kudos
Can we create Materialized view n exsting view and table?
Dear All,Is it possible to create Materialized view on view and table (Joining view and table)?I suspect it is not possible. Please suggest.Also please provide best way to schedule the refresh of Materialized view. Regards,Surya
- 490 Views
- 2 replies
- 1 kudos
- 5063 Views
- 4 replies
- 1 kudos
Resolved! Help - Can't create table from tutorial. Is my setup wrong?
Trying out databricks for the first time and followed the Get Started steps. I managed to successfully create a cluster and ran the simple sql tutorial to query data from a notebook. However, got the following error:Query:DROP TABLE IF EXISTS diamond...
- 5063 Views
- 4 replies
- 1 kudos
- 1 kudos
It seems as though you're doing great with your Databricks arrangement, however this sort of mistake could be connected with a couple of expected issues. In light of the subtleties you've shared, here are a few things you should check:Group Setup: Gu...
- 1 kudos
- 987 Views
- 5 replies
- 1 kudos
Empty Streaming tables in dlt
I want to create empty streaming tables in dlt with only schema specified. Is it possible ?I want to do it in dlt python.
- 987 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @ashraf1395, The term "rate" refers to a special source in Apache Spark's Structured Streaming that generates data at a specified rate. This source is primarily used for testing and benchmarking purposes. When you use spark.readStream.format("rate...
- 1 kudos
- 1281 Views
- 2 replies
- 2 kudos
Resolved! applying column tags
Can anyone tell me the correct syntax for applying a column tag to a specific tableThese are what I tried ALTER TABLE accounts_and_customer.bronze.BB1123_loans ALTER/CHANGE COLUMN loan_number SET TAGS ('classification' = 'confidential')<p>I got thi...
- 1281 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Takuya-Omi ,I agree. The syntax was correct. I was facing some completely different problems with schemas and I solved it. Thanks though. Or I would have spent hours banging my head to find the reason for error.
- 2 kudos
- 509 Views
- 1 replies
- 1 kudos
CDF table partition - Real time Data
Hi team,We are currently working in loading CDF table using data events from Kafka. The table is going to hold data across geographies. When we tried partitioning it is slowing down the ingestion time. But without partition the downstream application...
- 509 Views
- 1 replies
- 1 kudos
- 1 kudos
1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...
- 1 kudos
- 976 Views
- 3 replies
- 2 kudos
Resolved! UNPIVOT VARIANT data in SQL
Hi All,Have a VARIANT column with the following data;CREATE TABLE unpivot_valtype AS SELECT parse_json( '{ "Id": 1234567, "Result": { "BodyType": "NG", "ProdType": "Auto", "ResultSets": [ { "R1": { "AIn...
- 976 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @binsel ,You need to use variant_explode function.Here is the working code:WITH first_explode AS ( SELECT uv.rowData:Id AS Id, uv.rowData:Result:BodyType AS BodyType, uv.rowData:Result:ProdType AS ProdType, v.value AS result_se...
- 2 kudos
- 414 Views
- 1 replies
- 0 kudos
Error Querying Shallow Clones: Couldn't Initialize File System for Path
Hi,We are offering data products through a central catalog for our users. To minimize data duplication and to display relationships between tables, we use shallow clones to provide access to the data.However, since implementing this approach, we occa...
- 414 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @schluca ,I’ve encountered an issue where an error occurred when trying to reference a table after deleting and recreating the source table for a Shallow Clone, and then performing the Shallow Clone again. As a solution, try deleting the destinati...
- 0 kudos
- 704 Views
- 2 replies
- 1 kudos
Databricks table lineage
Hi,I wanted to know if it is possible to edit the lineage that we see in databricks, like the one shown below.Can I edit this lineage graph, like add other ETL tools (at the start of the tables) that I have used to get data in aws and then in databri...
- 704 Views
- 2 replies
- 1 kudos
- 1 kudos
This will be extremely beneficial. We have certain use cases where we do not leverage Spark in our pipelines and lose the lineage. I would prefer to set an extra parameter when writing a table to specify the lineage.
- 1 kudos
- 422 Views
- 2 replies
- 0 kudos
Error while running a notebook job using git repo (Gitlab)
I am trying to run a notebook job with a git repo hosted on Gitlab. I have Linked my Gitlab account using Gitlab tokenYet i am getting the following error on running the job How to resolve this?
- 422 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @vinitkhandelwal, Looks like the token could be missing required permissions for the operation. Please refer to: You can clone public remote repositories without Git credentials (a personal access token and a username). To modify a public remote r...
- 0 kudos
- 2480 Views
- 4 replies
- 3 kudos
Azure IMDS is not accesbile selecting shared compute policy
Hi, Databricks community,I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user id...
- 2480 Views
- 4 replies
- 3 kudos
- 3 kudos
I'm having a similar problem, my aim is from an Azure DataBricks notebook to invoke an AzureDataDactory pipeline I created an Access Connector for Azure Databricks to which I gave Data Factory Contributor permissions. Using these lines pythonfrom azu...
- 3 kudos
- 892 Views
- 1 replies
- 0 kudos
Resolved! how to pass using DABs the parameters to use
Hello Community, I want to pass parameters to my Databricks job through the DABs CLI. Specifically, I'd like to be able to run a job with parameters directly using the command: databricks bundle run -t prod --params [for example: table_name="client"...
- 892 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeremy98 ,You can pass parameters using CLI in following way: databricks bundle run -t ENV --params Param1=Value1,Param2=Value2 Job_Name And in your yml file you should define parameters in similar way to following:You can find more info in follo...
- 0 kudos
- 552 Views
- 1 replies
- 0 kudos
Pydeequ is not working with shared mode clusters on a unity catalog enabled databricks account
Hi folks,Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :It works fine with the single mode clusters. Does anyone have ...
- 552 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @parasupadhyay ,I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.
- 0 kudos
- 776 Views
- 2 replies
- 0 kudos
Get user who ran a job
From the databricks API/ CLI is it possible to get the user who triggered a job run programatically?The information can be found in the job "event log" and can be queried in the "audit log" but neither of these seem to have a API option. Is there a w...
- 776 Views
- 2 replies
- 0 kudos
- 0 kudos
with the databricks cli you can get all infos about a job run with this command:databricks jobs get-run <run-id> replace <run-id> with your actual run-id
- 0 kudos
- 20158 Views
- 19 replies
- 10 kudos
Resolved! Lakehouse Fundamentals Certificate/Badge not appearing
Hello! I just passed the Lakehouse Fundamentals Accreditation and I haven't received any badge or certificate for it. I understand that I need to go to credentials.databricks.com but it is not there. How long before it appears? Need help
- 20158 Views
- 19 replies
- 10 kudos
- 10 kudos
Hello can you help me, I have the same problem. I don't have the certificate available on credentials.databricks.com.
- 10 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
