- 5745 Views
- 4 replies
- 1 kudos
Resolved! Help - Can't create table from tutorial. Is my setup wrong?
Trying out databricks for the first time and followed the Get Started steps. I managed to successfully create a cluster and ran the simple sql tutorial to query data from a notebook. However, got the following error:Query:DROP TABLE IF EXISTS diamond...
- 5745 Views
- 4 replies
- 1 kudos
- 1 kudos
It seems as though you're doing great with your Databricks arrangement, however this sort of mistake could be connected with a couple of expected issues. In light of the subtleties you've shared, here are a few things you should check:Group Setup: Gu...
- 1 kudos
- 2034 Views
- 2 replies
- 2 kudos
Resolved! applying column tags
Can anyone tell me the correct syntax for applying a column tag to a specific tableThese are what I tried ALTER TABLE accounts_and_customer.bronze.BB1123_loans ALTER/CHANGE COLUMN loan_number SET TAGS ('classification' = 'confidential')<p>I got thi...
- 2034 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Takuya-Omi ,I agree. The syntax was correct. I was facing some completely different problems with schemas and I solved it. Thanks though. Or I would have spent hours banging my head to find the reason for error.
- 2 kudos
- 679 Views
- 1 replies
- 1 kudos
CDF table partition - Real time Data
Hi team,We are currently working in loading CDF table using data events from Kafka. The table is going to hold data across geographies. When we tried partitioning it is slowing down the ingestion time. But without partition the downstream application...
- 679 Views
- 1 replies
- 1 kudos
- 1 kudos
1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...
- 1 kudos
- 1415 Views
- 3 replies
- 2 kudos
Resolved! UNPIVOT VARIANT data in SQL
Hi All,Have a VARIANT column with the following data;CREATE TABLE unpivot_valtype AS SELECT parse_json( '{ "Id": 1234567, "Result": { "BodyType": "NG", "ProdType": "Auto", "ResultSets": [ { "R1": { "AIn...
- 1415 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @binsel ,You need to use variant_explode function.Here is the working code:WITH first_explode AS ( SELECT uv.rowData:Id AS Id, uv.rowData:Result:BodyType AS BodyType, uv.rowData:Result:ProdType AS ProdType, v.value AS result_se...
- 2 kudos
- 602 Views
- 1 replies
- 0 kudos
Error Querying Shallow Clones: Couldn't Initialize File System for Path
Hi,We are offering data products through a central catalog for our users. To minimize data duplication and to display relationships between tables, we use shallow clones to provide access to the data.However, since implementing this approach, we occa...
- 602 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @schluca ,I’ve encountered an issue where an error occurred when trying to reference a table after deleting and recreating the source table for a Shallow Clone, and then performing the Shallow Clone again. As a solution, try deleting the destinati...
- 0 kudos
- 1106 Views
- 2 replies
- 1 kudos
Databricks table lineage
Hi,I wanted to know if it is possible to edit the lineage that we see in databricks, like the one shown below.Can I edit this lineage graph, like add other ETL tools (at the start of the tables) that I have used to get data in aws and then in databri...
- 1106 Views
- 2 replies
- 1 kudos
- 1 kudos
This will be extremely beneficial. We have certain use cases where we do not leverage Spark in our pipelines and lose the lineage. I would prefer to set an extra parameter when writing a table to specify the lineage.
- 1 kudos
- 588 Views
- 2 replies
- 0 kudos
Error while running a notebook job using git repo (Gitlab)
I am trying to run a notebook job with a git repo hosted on Gitlab. I have Linked my Gitlab account using Gitlab tokenYet i am getting the following error on running the job How to resolve this?
- 588 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @vinitkhandelwal, Looks like the token could be missing required permissions for the operation. Please refer to: You can clone public remote repositories without Git credentials (a personal access token and a username). To modify a public remote r...
- 0 kudos
- 1320 Views
- 1 replies
- 0 kudos
Resolved! how to pass using DABs the parameters to use
Hello Community, I want to pass parameters to my Databricks job through the DABs CLI. Specifically, I'd like to be able to run a job with parameters directly using the command: databricks bundle run -t prod --params [for example: table_name="client"...
- 1320 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeremy98 ,You can pass parameters using CLI in following way: databricks bundle run -t ENV --params Param1=Value1,Param2=Value2 Job_Name And in your yml file you should define parameters in similar way to following:You can find more info in follo...
- 0 kudos
- 744 Views
- 1 replies
- 0 kudos
Pydeequ is not working with shared mode clusters on a unity catalog enabled databricks account
Hi folks,Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :It works fine with the single mode clusters. Does anyone have ...
- 744 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @parasupadhyay ,I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.
- 0 kudos
- 1121 Views
- 2 replies
- 0 kudos
Get user who ran a job
From the databricks API/ CLI is it possible to get the user who triggered a job run programatically?The information can be found in the job "event log" and can be queried in the "audit log" but neither of these seem to have a API option. Is there a w...
- 1121 Views
- 2 replies
- 0 kudos
- 0 kudos
with the databricks cli you can get all infos about a job run with this command:databricks jobs get-run <run-id> replace <run-id> with your actual run-id
- 0 kudos
- 22098 Views
- 19 replies
- 10 kudos
Resolved! Lakehouse Fundamentals Certificate/Badge not appearing
Hello! I just passed the Lakehouse Fundamentals Accreditation and I haven't received any badge or certificate for it. I understand that I need to go to credentials.databricks.com but it is not there. How long before it appears? Need help
- 22098 Views
- 19 replies
- 10 kudos
- 10 kudos
Hello can you help me, I have the same problem. I don't have the certificate available on credentials.databricks.com.
- 10 kudos
- 1930 Views
- 3 replies
- 0 kudos
Databricks Genie AI Next Level Q&A
Hi Team Could Please you answer these three questions to onboard Genie Space for Analytics?Do you know if we can use GenieSpace in our web application through API or SDK?Is there any way to manage access control other than the Unity catalog?Can we ad...
- 1930 Views
- 3 replies
- 0 kudos
- 0 kudos
@AH my loyola wrote:Hi Team Could Please you answer these three questions to onboard Genie Space for Analytics?Do you know if we can use GenieSpace in our web application through API or SDK?Is there any way to manage access control other than the Uni...
- 0 kudos
- 13862 Views
- 5 replies
- 9 kudos
Spark 3.3.0 connect kafka problem
I am trying to connect to my Kafka from spark but getting an error:Kafka Version: 2.4.1Spark Version: 3.3.0I am using jupyter notebook to execute the pyspark code below:```from pyspark.sql.functions import *from pyspark.sql.types import *#import libr...
- 13862 Views
- 5 replies
- 9 kudos
- 9 kudos
Hi @avnish26, did you added the Jar files to the cluster? do you still have issues? please let us know
- 9 kudos
- 1917 Views
- 3 replies
- 4 kudos
Managing streaming checkpoints with unity catalog
This is partly a question, partly a feature request: How do you guys handle streaming checkpoints in combination with unity catalog managed tables?It seems like the only way is to create a volume, and manually specify paths in it as streaming checkpo...
- 1917 Views
- 3 replies
- 4 kudos
- 4 kudos
For Structured Streaming Applications, this would be a nice feature. Delta Live Tables manages checkpoints for you out of the box - you don't even have to reason about checkpoints at all, would recommend checking it out!
- 4 kudos
- 15236 Views
- 10 replies
- 4 kudos
How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API
Hi Community,I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the APII have tried passing JSON in this format:{ "full_refresh": "true", "configuration": [ ...
- 15236 Views
- 10 replies
- 4 kudos
- 4 kudos
You cannot pass parameters from a Databricks job to a DLT pipeline. Atleast not yet. You can see from the DLT rest API that there is no option for it to accept any parameters.But there is a workaround.But there is a workaround.With the assumption tha...
- 4 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
