- 3092 Views
- 1 replies
- 0 kudos
How to pass task name as parameter in scheduled job that will be used as a schema name in query
I want to run a parametrized sql query in a task. Query: select * from {{client}}.catalog.table with client value being {{task.name}}.if client is a string parameter, it is replaced with quotes which throws an error.if table is a dropdown list parame...
- 3092 Views
- 1 replies
- 0 kudos
- 0 kudos
Try this:select * from identifier(:catalog||'.schema.table') The :catalog is a parameter within DBSQLReplace schema and table with actual names
- 0 kudos
- 4289 Views
- 3 replies
- 2 kudos
How to use custom whl file + pypi repo with a job cluster in asset bundles?
I tried looking through the documentation but it is confusing at best and misses important parts at worst. Is there any place where the entire syntax and ALL options for asset bundle YAMLs are described? I found this https://docs.databricks.com/en/d...
- 4289 Views
- 3 replies
- 2 kudos
- 2 kudos
It took me a while to realize the distinction of the keys inside the task - so for anyone else looking into this: only one of the following keys can exist in a task definition: tasks: - task_key: ingestion_delta # existing_c...
- 2 kudos
- 4328 Views
- 5 replies
- 1 kudos
Databricks Excel ODBC driver bug
Hello!I might have experienced a bug with the ODBC driver. We have an issue where given certain priviledges in databricks, the ODBC driver is unable to show any schemas/tables.When we click the 'expand' button on any catalog in the list (of which we ...
- 4328 Views
- 5 replies
- 1 kudos
- 1 kudos
Following this post - we are also faced with the same issue. @KTheJoker- when I'm connecting and trying to expand a catalog, I do see the query fire off in the SQL Warehouse query history but in Excel nothing is returned. I can see the schemas/tables...
- 1 kudos
- 5170 Views
- 6 replies
- 1 kudos
Structured streaming in Databricks using delta table
Hi everyone, I’m new to Databricks and exploring its features. I’m trying to implement Change Data Capture (CDC) from the bronze layer to the silver layer using streaming. Could anyone share sample code or reference materials for implementing CDC wit...
- 5170 Views
- 6 replies
- 1 kudos
- 1 kudos
You can also look at https://www.databricks.com/resources/demos#tutorials
- 1 kudos
- 2110 Views
- 2 replies
- 0 kudos
DLT pipeline observability questions (and maybe suggestions)
All my questions is around this code block@Dlt.append_flow(target=”target_table”): def flow_01(): df = spark.readStream.table(“table_01”) @dlt.append_flow(target=”target_table”): def flow_02(): df = spark.readStream.table(“table_02”)The first qu...
- 2110 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @guangyi , I am getting back to you with some insights Regarding your first question about checkpointing You can manually check the checkpointing location of your stream table. The checkpoints of your Delta Live Tables are under Storage locatio...
- 0 kudos
- 2184 Views
- 1 replies
- 0 kudos
Databricks Asset Bundle - variables for job trigger
I'm using the Databricks CLI to deploy an asset bundle for a job. I'm attempting to setup the configuration such that the "dev" target does not have a trigger on it, and the "prod" target does. Essentially the dev job is not scheduled to run and th...
- 2184 Views
- 1 replies
- 0 kudos
- 0 kudos
in your dev target, you can add mode to pause all trigger.targets: dev: mode: development DAB also have new update,you can also use preset to handle different target setting.targets: dev: presets: name_prefix: "testing_" # pre...
- 0 kudos
- 27728 Views
- 10 replies
- 6 kudos
We've encountered an error logging you in.
I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...
- 27728 Views
- 10 replies
- 6 kudos
- 6 kudos
Hey, I am not able to login to my work databrick's account. I am getting the above errors.
- 6 kudos
- 1816 Views
- 1 replies
- 0 kudos
Resolved! Using sql inside Notebook using question marks
Hi all,I have a very quick question that I hope someone can help with.I want to execute a very simple sql statement like %sql select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm$RITWebsader$911a%'However, the sql does not ...
- 1816 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @JensV , Question marks are used as parameter placeholders, so could you please try to escape the question mark using Backslashes?select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm\\$RITWebsader\\$911a%'Alternatively, w...
- 0 kudos
- 3256 Views
- 1 replies
- 0 kudos
use_cached_result
i am trying to execute same query on 3 different platforms - dbeaver, python notebook and sql workflow.I was expecting after first execution of the query irrespective of the platform, subsequent execution of same query should NOT re-compute. However ...
- 3256 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think its possible unless the results written into a table and its being used in the queries across the client. Pls refer to this https://docs.databricks.com/en/sql/user/queries/query-caching.html
- 0 kudos
- 6348 Views
- 1 replies
- 0 kudos
JSON string object with nested Array and Struct column to dataframe in pyspark
I am trying to convert JSON string stored in variable into spark dataframe without specifying schema, because I have a big number of different tables, so it has to be dynamically. I managed to do it with sc.parallelize, but since we are moving to Uni...
- 6348 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi filipjankovic, SparkContext sc is a Spark 1.0 API and is deprecated on Standard and Serverless compute. However, your input data is a list of dictionaries, which are supported with spark.createDataFrame. This should give you identical output witho...
- 0 kudos
- 2727 Views
- 2 replies
- 3 kudos
Using Managed Identity Authentication in Unity Catalog using pyodbc
Hello,I am having trouble using Managed Identity Authentication in Unity Catalog using pyodbc in Azure Databricks. The same code works on a "Legacy Shared Compute". The code snippet is below: import pyodbc jdbc_url = ( "DRIVER={ODBC 17 DRIVER PATH...
- 2727 Views
- 2 replies
- 3 kudos
- 3 kudos
Thank you very much!I have spent an enormous amount of hours fighting with this and in the end it was the type of cluster... I hope that this problem will be solved in the future, because affects the developments when you use databricks-connect and s...
- 3 kudos
- 3121 Views
- 3 replies
- 0 kudos
DLT apply_changes not accepting upsert
I have a DLT pipeline that has bronze -> silver -> gold -> platinum. I need to include a table that is joined to the gold layer for platinum that allows upserts in the DLT pipeline. This table is managed externally via Databricks API. Anytime a chang...
- 3121 Views
- 3 replies
- 0 kudos
- 0 kudos
You obtain this error: "Detected a data update in the source table at version 1. This is currently not supported...."becuse DLT is based on Structured Streaming.... and for Structured Streaming any changes (deletes, updates) in the source table are n...
- 0 kudos
- 11371 Views
- 8 replies
- 10 kudos
Resolved! Pandas.spark.checkpoint() doesn't broke lineage
Hi, I'm doing some something simple on Databricks notebook:spark.sparkContext.setCheckpointDir("/tmp/") import pyspark.pandas as ps sql=("""select field1, field2 From table Where date>='2021-01.01""") df = ps.sql(sql) df.spark.checkpoint()That...
- 11371 Views
- 8 replies
- 10 kudos
- 10 kudos
checkpoint() returns a checkpointed DataFrame, so you need to assign it to a new variable:checkpointedDF = df.checkpoint()
- 10 kudos
- 1494 Views
- 1 replies
- 1 kudos
Resolved! Oracle DB connection works in single-user compute but not shared compute
I can connect to an onprem oracle DB using my single-user compute but when i switch over to a shared compute, I get invalid username/password error. I can connect to my onprem singlestore db using either compute so not sure why oracle would be diffe...
- 1494 Views
- 1 replies
- 1 kudos
- 1 kudos
Based on internal research I found that shared access mode does not currently support Oracle JDBC connector only in Single/Assigned access mode. There is a feature request to include Oracle connector as part of the Lakehouse Federation. Once it is in...
- 1 kudos
- 7549 Views
- 12 replies
- 8 kudos
Resolved! Reading a csv file
while try to read a csv file using data frame , read csv using a file format , but fail in case of formatting and column error while loading how the data in databricks ,the code i used fordf = spark.read.format("csv") \ .option("header", "true") ...
- 7549 Views
- 12 replies
- 8 kudos
- 8 kudos
You can try add multiline option: df = ( spark.read.format("csv") .option("header", "true") .option("quote", '"') .option("delimiter", ",") .option("nullValue", "") .option("emptyValue", "NULL") .option("multiline", True) .schema(schem...
- 8 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
