- 3236 Views
- 5 replies
- 0 kudos
How to replace ${param} by :param
Hello,We previously used ${param} in our SQL queries:SELECT * FROM json.`${source_path}/file.json`However, this syntax is now deprecated. The recommended approach is to use :param instead.But when I attempt to replace ${param} with :param, I encounte...
- 3236 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ZD Please try this syntax in Your notebook for SQL:%sqldeclare _my_path = 'some_path';select _my_path;
- 0 kudos
- 1769 Views
- 2 replies
- 1 kudos
Resolved! Job cluster has no permission to create folder in Unity Catalog Volume
Hello everybody,I want to run a job that collects some csv files from a SFTP server and saves them on my Unity Catalog Volume. While my personal cluster defined like the following has access to create folders on the volume my job cluster doesn't.Defi...
- 1769 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you, that helped although I had to use "SINGLE_USER" instead of "DATA_SECURITY_MODE_DEDICATED". According to the docs (https://docs.databricks.com/api/workspace/clusters/create) "SINGLE_USER" is an alias for "DATA_SECURITY_MODE_DEDICATED".
- 1 kudos
- 2731 Views
- 4 replies
- 2 kudos
Resolved! Not able to read data from volume and data is in JSON format
Not able to read data from volume and data is in JSON format data = spark.read.json("/Volumes/mydatabricksaviral/datatesting/datavolume/mytest.json") display(data) ############################################# Py4JJavaError: An error occurred while...
- 2731 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Aviral-Bhardwaj ,please double check:- if volume path is correct- if you have READ VOLUME permission on this volume- if your cluster has access to unity catalog- if json file exists
- 2 kudos
- 1259 Views
- 1 replies
- 0 kudos
Resolved! Low Level Design for Moving Data from Databricks A to Databricks B
Hello Techie,May someone please help me with Low level design point what all we should considered while moving data from One Delta lake instance to another delta lake.For example :-Service principle creation.IP Whitelisting.Any gitlab / devops relate...
- 1259 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Pratikmsbsvm Here's a brief low-level design checklist for Delta Lake to Delta Lake data migration:1. Security & Authentication- Create service principals for both environments- Set up Azure Key Vault for credential management- Configure IP white...
- 0 kudos
- 1659 Views
- 1 replies
- 0 kudos
finishes execution in 6 seconds but the notebook still shows "waiting"
Issue:Although my SQL query completes execution in approximately 2-3 seconds, the notebook interface continues to show "waiting" for an extended period before displaying results. Only way to see the results of my cell execution is by refreshing the w...
- 1659 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @root92 This is a known Databricks interface issue, not related to query performance or account type.Most Likely Causes:WebSocket connection timeout between browser and DatabricksBrowser memory issues with long-running notebook sessionsNetwork pro...
- 0 kudos
- 1915 Views
- 5 replies
- 0 kudos
in-home built predictive optimization
Hello allHas anyone attempted to look at the internals of predictive optimization and built an in-home solution mimicking its functionality? I understood that there are no plans from Databricks to roll-out this feature for external tables, and hence,...
- 1915 Views
- 5 replies
- 0 kudos
- 0 kudos
@LinlinH thanks for the details. Can you please share any Github link where the community work is put so I can verify if any code can be re-used...
- 0 kudos
- 4595 Views
- 2 replies
- 0 kudos
Collecting Job Usage Metrics Without Unity Catalog
hi,I would like to request assistance on how to collect usage metrics and job execution data for my Databricks environment. We are currently not using Unity Catalog, but I would still like to monitor and analyze usageCould you please provide guidance...
- 4595 Views
- 2 replies
- 0 kudos
- 4491 Views
- 2 replies
- 0 kudos
Append-only table from non-streaming source in Delta Live Tables
I have a DLT pipeline, where all tables are non-streaming (materialized views), except for the last one, which needs to be append-only, and is therefore defined as a streaming table.The pipeline runs successfully on the first run. However on the seco...
- 4491 Views
- 2 replies
- 0 kudos
- 0 kudos
@Oliver_Angelil - I was wondering if you found a solution? I have a similar use-case. I want to create a archive table using DLT from non-streaming (MV). I would prefer a DLT solution. I was able to get it to work using traditional merge approach (no...
- 0 kudos
- 3040 Views
- 3 replies
- 1 kudos
Resolved! Deploying Data Assets Bundle with VSCode Add-in
Deploying a bundle containing a Pipeline that references a DLT Notebook.In the YAML I am passing the relative path to the Notebook from the repository root (where the YAML lives).Deploying says 'success' but when validating, the Pipeline cannot find ...
- 3040 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @simonB2025 May you share a snippet of your folder/project structure and where the notebook resides, so we can suggest toyou the exact solution. Thank you!Best, Ilir
- 1 kudos
- 3547 Views
- 1 replies
- 1 kudos
Resolved! connecting to unity catalog with power BI
i created a group of powerBI users, i grant consumer access permission to this groupe on unity catalog i started a shared compute cluster (Standard_D4ds_v5). when trying to connect from power BI, i will connect to tables only if i grant access to wor...
- 3547 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Miloud_G Instead of granting full workspace access, you can grant "Databricks SQL access" entitlement, which provides limited workspace access specifically for BI tools.Go to Workspace Settings → Identity and Access → Groups/Users/SPs Manage → Se...
- 1 kudos
- 3178 Views
- 1 replies
- 1 kudos
Resolved! Complex pipeline with many tasks and dependencies: orchestration in Jobs or in Notebook?
We need to set up a job that consists of several hundred tasks with many dependencies between each other. We are considering two different directions:1. Databricks job with tasks, with dependencies defined as code and deployed with Databricks asset b...
- 3178 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @smoortema To my best knowlegde:Option 1)You can create jobs that contain up to 1000 tasks, however, it is recommended to split tasks into logical subgroups.Jobs with more than 100 tasks require API 2.2 and above Jobs with a large number of tasks ...
- 1 kudos
- 3103 Views
- 4 replies
- 0 kudos
Time Travel vs Bronze historical archive
HelloI am working on building a pipeline using Medallion architecture. The source tables in this pipeline are overwritten each time the table is updated. In the bronze ingestion layer, I plan to append this new table to the current bronze table, addi...
- 3103 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Locomo_Dncr Time travel isn't recomended to store historical data. It's for backup, audit purpose. You can store snapshot data or use SCD2 to keep history."Databricks does not recommend using Delta Lake table history as a long-term backup solutio...
- 0 kudos
- 2818 Views
- 6 replies
- 0 kudos
Understanding infra costs of Databricks compute
Hi.I can see in our Azure cost analysis tool that a not insignificant part of our costs come from the managed Databricks RG deployed with the workspace, and that it relates particularly to VMs (so compute, I assume?) and storage, the later which, tho...
- 2818 Views
- 6 replies
- 0 kudos
- 0 kudos
Thank you all for your replies. The issue is not getting an overview of costs - I already have that from the Cost Management Export function in Azure, and by using the system.billing tables in Databricks. The issue is understanding the relation betwe...
- 0 kudos
- 2831 Views
- 4 replies
- 2 kudos
Not able to get the location of the table in Databricks Free edition
Hi Team,I am exploring the delta table properties and wants to see the internal folder structure of the delta table. I created a delta table using Databricks Free edition and when I executed the describe extended command, the location field is coming...
- 2831 Views
- 4 replies
- 2 kudos
- 3327 Views
- 4 replies
- 6 kudos
Job Failures Observed In DBR 13.3 due to changes in pip package - aiosignal
Problem Statement Few jobs running on DBR 13.3 have started failing with error mentioned below - TypeError: TypeVarTuple.__init__() got an unexpected keyword argument 'default'--------------------------------------------------------------------------...
- 3327 Views
- 4 replies
- 6 kudos
- 6 kudos
Pinning down aiosignal==1.3.2 worked for me, are you sure the package you are building is using the pinned version?
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
