- 2500 Views
- 1 replies
- 0 kudos
Connecting Power BI to Delta Share: Error on data retrieval
Hi everybody,sharing data with an access token and Databricks Connector works fine in Power BI (desktop). Now we wanted to switch to Delta Sharing.We setup a delta share to distribute data via open share to anyone outside our organization. Unity Cata...
- 2500 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi everybody,for anybody running into the same issue.It is a bug in the current Power Bi version (2.121.644.0). I reverted back to the April release (2.116.404.0), which does work as expected.
- 0 kudos
- 4566 Views
- 1 replies
- 0 kudos
How to create a databricks job with parameters via CLI?
I'm creating a new job in databricks using the databricks-cli:databricks jobs create --json-file ./deploy/databricks/config/job.config.jsonWith the following json:{ "name": "Job Name", "new_cluster": { "spark_version": "4.1.x-scala2.1...
- 4566 Views
- 1 replies
- 0 kudos
- 0 kudos
This is an old post but still relevant for future readers, so will answer how it is done. You need to add base_parameters flag in the notebook_task config, like the following. "notebook_task": { "notebook_path": "...", "base_parameters": { ...
- 0 kudos
- 8608 Views
- 9 replies
- 6 kudos
Resolved! Spark UI reverse Proxy blocked on GCP
Using the 9.1ML cluster atm but also tried the 7.3 and 8.1.Databricks is deployed on google platform and I was using the trial.It is quite difficult to debug if the spark ui is only semi accessible.Part of the results in raw html are visible but all ...
- 8608 Views
- 9 replies
- 6 kudos
- 5793 Views
- 2 replies
- 0 kudos
ADD_NODES_FAILED Cluster Does Not Start
Hello everyone, I tried to change a Databricks Runtime Cluster from 12.2 LTS ML to 13.3 LTS ML, however I got this error: Failed to add 1 container to the compute. Will attempt retry: false. Reason: Global init script failureGlobal init script Instal...
- 5793 Views
- 2 replies
- 0 kudos
- 12187 Views
- 6 replies
- 2 kudos
Using run_job_task in Databricks Asset Bundles
Do Databrick Asset Bundles support run_job_task tasks?I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've...
- 12187 Views
- 6 replies
- 2 kudos
- 2 kudos
Ah, I see it is a known bug in the Databricks CLI: Asset bundle run_job_task fails · Issue #812 · databricks/cli (github.com). Anyone facing this issue should comment on and keep an eye on that ticket for resolution.
- 2 kudos
- 10876 Views
- 3 replies
- 0 kudos
Resolved! Pull Cluster Tags
Does anybody know any in-notebook or JAR code to pull cluster tags from the runtime environment? Something like... dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply('user')but for the cluster name?
- 10876 Views
- 3 replies
- 0 kudos
- 0 kudos
Did you find any documentation for spark.conf.get properties? I am trying to get some metadata about the environment my notebook is running in (specifically cluster custom tags)? But cannot find any information beside a couple of forum posts.
- 0 kudos
- 3825 Views
- 1 replies
- 1 kudos
Resolved! Job Concurrency Queue not working as expected
I have a process that should run the same notebook with varying parameters, thus translating to a job with queue and concurrency enabled. When the first executions are triggered the Jobs Runs work as expected, i.e. if the job has a max concurrency se...
- 3825 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod, we double-checked everything, the resources are enough and all settings are properly set. I'll reach out the support by filing a new ticket. Thank you for your help.
- 1 kudos
- 2226 Views
- 2 replies
- 1 kudos
to_timstamp function in non-legacy mode does not parse this format: yyyyMMddHHmmssSS
I have this datetime string in my dataset: '2023061218154258' and I want to convert it to datetime, using below code. However the format that I expect to work, doesn't work, namely: yyyyMMddHHmmssSS. This code will reproduce the issue:from pyspark.sq...
- 2226 Views
- 2 replies
- 1 kudos
- 1 kudos
Is there anybody who has the same issue or knows that this is in fact an issue?
- 1 kudos
- 11329 Views
- 1 replies
- 0 kudos
Resolved! Java - FAILED_WITH_ERROR when saving to snowflake
I'm trying to move data from database A to B on Snowflake. There's no permission issue since using the Python package snowflake.connector works Databricks runtime version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)Insert into database B fail...
- 11329 Views
- 1 replies
- 0 kudos
- 0 kudos
Found the problem. The sub-roles didn't have grants to the warehouse.I hope it will help someone one day
- 0 kudos
- 2939 Views
- 1 replies
- 0 kudos
Trigger Events in data pipeline
Hi,I am running datapipeline in databrick using matillion architecture. I am facing inconsistent events in silver to gold layer in case any row deleted/updated from a partition. Let me explain with example.e.g. I have data in silver layer with partit...
- 2939 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you Kaniz. Further queries on this.1. If I have nested partitions e.g. on department & date, finance->09, finance->10 and if am updating one record in finance->09 then will then updates partition finance->10 as well2. Is it good idea to have sm...
- 0 kudos
- 7550 Views
- 4 replies
- 5 kudos
Resolved! DLT overwrite part of the table
Hello !We're currently building a pipeline of file ingestion using a Delta Live Tables pipeline and autoloader. The bronze tables are pretty much the following schema : file_name | file_upload_date | colA | colB (Well, there are actually 250+ columns...
- 7550 Views
- 4 replies
- 5 kudos
- 3365 Views
- 3 replies
- 1 kudos
DLT: Autoloader Perf
Hi Team,I am looking for some advice to perf tune my bronze layer using DLT.I have the following code very simple and yet very effective. @dlt.create_table(name="bronze_events", comment = "New raw data ingested from storage account ...
- 3365 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Gilg You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on ...
- 1 kudos
- 3994 Views
- 1 replies
- 1 kudos
APPLY_CHANGES late arriving data
Hi Team,I have a DLT pipeline that uses APPLY_CHANGES to our Silver tables. I am using Id as keys and timestamp to know the sequence of the incoming data. Question: How does APPLY_CHANGES handles late arriving data?i.e., for silver_table_1, the data ...
- 3994 Views
- 1 replies
- 1 kudos
- 16820 Views
- 0 replies
- 0 kudos
Access denied error while reading file from S3 to spark
I'm seeing the access denied error from spark cluster while reading s3 file into notebook.Running on personal single user compute with LTS 13.3 ML.configs setup looks like this:spark.conf.set("spark.hadoop.fs.s3a.access.key", access_id)spark.conf.set...
- 16820 Views
- 0 replies
- 0 kudos
- 2191 Views
- 1 replies
- 0 kudos
Resolved! Databricks Academy - Advanced Data Engineering - Notebook Error while loading configurations
Hi Databricks Academy team,I am getting below errors while running classroom setup command in Databricks Academy - Advanced data engineering course Notebooks in databricks community edition. Please help me resolve it. #databricksacademy #advanceddat...
- 2191 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, all. Our apologies - the Advanced Data Engineering with Databricks course will not run on Databricks Community Edition. If you would like a lab environment to run this course on, please see the new paid lab subscription available via the Databric...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
