- 458 Views
- 2 replies
- 2 kudos
Resolved! applying column tags
Can anyone tell me the correct syntax for applying a column tag to a specific tableThese are what I tried ALTER TABLE accounts_and_customer.bronze.BB1123_loans ALTER/CHANGE COLUMN loan_number SET TAGS ('classification' = 'confidential')<p>I got thi...
- 458 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Takuya_Omi ,I agree. The syntax was correct. I was facing some completely different problems with schemas and I solved it. Thanks though. Or I would have spent hours banging my head to find the reason for error.
- 2 kudos
- 285 Views
- 1 replies
- 1 kudos
CDF table partition - Real time Data
Hi team,We are currently working in loading CDF table using data events from Kafka. The table is going to hold data across geographies. When we tried partitioning it is slowing down the ingestion time. But without partition the downstream application...
- 285 Views
- 1 replies
- 1 kudos
- 1 kudos
1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...
- 1 kudos
- 459 Views
- 3 replies
- 2 kudos
Resolved! UNPIVOT VARIANT data in SQL
Hi All,Have a VARIANT column with the following data;CREATE TABLE unpivot_valtype AS SELECT parse_json( '{ "Id": 1234567, "Result": { "BodyType": "NG", "ProdType": "Auto", "ResultSets": [ { "R1": { "AIn...
- 459 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @binsel ,You need to use variant_explode function.Here is the working code:WITH first_explode AS ( SELECT uv.rowData:Id AS Id, uv.rowData:Result:BodyType AS BodyType, uv.rowData:Result:ProdType AS ProdType, v.value AS result_se...
- 2 kudos
- 302 Views
- 2 replies
- 1 kudos
Infor Data Lake to Databricks
I'm working on this project which involves moving data from Infor to Databricks.Infor is somewhat of an enterprise solution. I could not find much resources on this. I could not even find any free trial option on their site.If anyone has experience w...
- 302 Views
- 2 replies
- 1 kudos
- 1 kudos
@devagya wrote:I'm working on this project which involves moving data from Infor to Databricks.Infor is somewhat of an enterprise solution. I could not find much resources on this. I could not even find any free trial option on their site.If anyone h...
- 1 kudos
- 209 Views
- 1 replies
- 0 kudos
Error Querying Shallow Clones: Couldn't Initialize File System for Path
Hi,We are offering data products through a central catalog for our users. To minimize data duplication and to display relationships between tables, we use shallow clones to provide access to the data.However, since implementing this approach, we occa...
- 209 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @schluca ,I’ve encountered an issue where an error occurred when trying to reference a table after deleting and recreating the source table for a Shallow Clone, and then performing the Shallow Clone again. As a solution, try deleting the destinati...
- 0 kudos
- 365 Views
- 2 replies
- 1 kudos
Databricks table lineage
Hi,I wanted to know if it is possible to edit the lineage that we see in databricks, like the one shown below.Can I edit this lineage graph, like add other ETL tools (at the start of the tables) that I have used to get data in aws and then in databri...
- 365 Views
- 2 replies
- 1 kudos
- 1 kudos
This will be extremely beneficial. We have certain use cases where we do not leverage Spark in our pipelines and lose the lineage. I would prefer to set an extra parameter when writing a table to specify the lineage.
- 1 kudos
- 242 Views
- 2 replies
- 0 kudos
Error while running a notebook job using git repo (Gitlab)
I am trying to run a notebook job with a git repo hosted on Gitlab. I have Linked my Gitlab account using Gitlab tokenYet i am getting the following error on running the job How to resolve this?
- 242 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @vinitkhandelwal, Looks like the token could be missing required permissions for the operation. Please refer to: You can clone public remote repositories without Git credentials (a personal access token and a username). To modify a public remote r...
- 0 kudos
- 1292 Views
- 3 replies
- 2 kudos
Azure IMDS is not accesbile selecting shared compute policy
Hi, Databricks community,I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user id...
- 1292 Views
- 3 replies
- 2 kudos
- 2 kudos
I'm having a similar problem, my aim is from an Azure DataBricks notebook to invoke an AzureDataDactory pipeline I created an Access Connector for Azure Databricks to which I gave Data Factory Contributor permissions. Using these lines pythonfrom azu...
- 2 kudos
- 419 Views
- 1 replies
- 0 kudos
Resolved! how to pass using DABs the parameters to use
Hello Community, I want to pass parameters to my Databricks job through the DABs CLI. Specifically, I'd like to be able to run a job with parameters directly using the command: databricks bundle run -t prod --params [for example: table_name="client"...
- 419 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeremy98 ,You can pass parameters using CLI in following way: databricks bundle run -t ENV --params Param1=Value1,Param2=Value2 Job_Name And in your yml file you should define parameters in similar way to following:You can find more info in follo...
- 0 kudos
- 236 Views
- 1 replies
- 0 kudos
Pydeequ is not working with shared mode clusters on a unity catalog enabled databricks account
Hi folks,Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :It works fine with the single mode clusters. Does anyone have ...
- 236 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @parasupadhyay ,I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.
- 0 kudos
- 347 Views
- 2 replies
- 0 kudos
Get user who ran a job
From the databricks API/ CLI is it possible to get the user who triggered a job run programatically?The information can be found in the job "event log" and can be queried in the "audit log" but neither of these seem to have a API option. Is there a w...
- 347 Views
- 2 replies
- 0 kudos
- 0 kudos
with the databricks cli you can get all infos about a job run with this command:databricks jobs get-run <run-id> replace <run-id> with your actual run-id
- 0 kudos
- 17664 Views
- 19 replies
- 10 kudos
Resolved! Lakehouse Fundamentals Certificate/Badge not appearing
Hello! I just passed the Lakehouse Fundamentals Accreditation and I haven't received any badge or certificate for it. I understand that I need to go to credentials.databricks.com but it is not there. How long before it appears? Need help
- 17664 Views
- 19 replies
- 10 kudos
- 10 kudos
Hello can you help me, I have the same problem. I don't have the certificate available on credentials.databricks.com.
- 10 kudos
- 38693 Views
- 6 replies
- 5 kudos
how to read the CSV file from users workspace
Hi!I have been carrying out a POC, so I created the CSV file in my workspace and tried to read the content using the techniques below in a Python notebook, but did not work.Option1:repo_file = "/Workspace/Users/u1@org.com/csv files/f1.csv"tmp_file_na...
- 38693 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Dev ,Generally, What happens spark reader APIs point to the DBFS by default. And, to read the file from User workspace, we need to append 'file:/' in the prefix.Thanks
- 5 kudos
- 1077 Views
- 3 replies
- 0 kudos
Databricks Genie AI Next Level Q&A
Hi Team Could Please you answer these three questions to onboard Genie Space for Analytics?Do you know if we can use GenieSpace in our web application through API or SDK?Is there any way to manage access control other than the Unity catalog?Can we ad...
- 1077 Views
- 3 replies
- 0 kudos
- 0 kudos
@AH my loyola wrote:Hi Team Could Please you answer these three questions to onboard Genie Space for Analytics?Do you know if we can use GenieSpace in our web application through API or SDK?Is there any way to manage access control other than the Uni...
- 0 kudos
- 11725 Views
- 5 replies
- 9 kudos
Spark 3.3.0 connect kafka problem
I am trying to connect to my Kafka from spark but getting an error:Kafka Version: 2.4.1Spark Version: 3.3.0I am using jupyter notebook to execute the pyspark code below:```from pyspark.sql.functions import *from pyspark.sql.types import *#import libr...
- 11725 Views
- 5 replies
- 9 kudos
- 9 kudos
Hi @avnish26, did you added the Jar files to the cluster? do you still have issues? please let us know
- 9 kudos