- 1461 Views
- 5 replies
- 0 kudos
Accessing Workspace / Repo file works in notebook, but not from job
In a notebook attached to.a normal personal cluster I can successfully do this:%fs ls file:/Workspace/Repos/$userName/$repoName/$folderNameWhen I run an init-script on a UC volume that the does the same thing, I'm getting this error:ls: cannot access...
- 1461 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @dollyb, Can you try with just "ls /Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name" I'm not sure dbutils will be useful in an init script, I will try to test it out
- 0 kudos
- 13165 Views
- 8 replies
- 5 kudos
Resolved! No Course Materials Widget below Lesson
Hello everyone,In my Databricks partner academy account, there is no course material while it should be under the lesson video. How can I resolve this problem? Does anyone else face the same problem? I had submitted a ticket to ask Databricks team bu...
- 13165 Views
- 8 replies
- 5 kudos
- 5 kudos
I am still having this problem, cant find the slides and DLC for Data Engineering with Databricks
- 5 kudos
- 2094 Views
- 1 replies
- 0 kudos
Resolved! Run a notebook as a different user or role
Outside of running jobs with different users, is there any way for me to run a notebook (or even better a cell within a notebook) as either a different user or a specific role that is not my user default?I'm trying to find an easy way to test data ma...
- 2094 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately there is no direct way to run a notebook using additional principal, only option is to set up a notebook job task and on the Run As specify the principal that will run the job that can be a user or a Service Principal.
- 0 kudos
- 4327 Views
- 3 replies
- 0 kudos
Is it possible to share a Dashboard with a user inside , org that doesn't have a Databricks account?
Hello Team,Is it possible to share a Dashboard with a user inside the organization that doesn't have a Data-bricks account?Assign a cluster to one notebook/dashboard and share it inside the organization with a SSO login possibility?Suppose we want to...
- 4327 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello, thank you for this since I am also looking for this functionality. However, doing more digging with my engineers we found that this feature may allow us to share dashboards to users not in the Workspace, but it still requires the user to have ...
- 0 kudos
- 1561 Views
- 1 replies
- 1 kudos
how to use dlt module in streaming pipeline
If anyone has example code for building a CDC live streaming pipeline generated by AWS DMS using import dlt, I'd love to see it.I'm currently able to see the parquet file starting with Load on the first full load to S3 and the cdc parquet file after ...
- 1561 Views
- 1 replies
- 1 kudos
- 1 kudos
There is a blogpost for this that includes example code that you can find here
- 1 kudos
- 26600 Views
- 4 replies
- 26 kudos
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace �� Private link In your storage accounts please go to “Networ...
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace Private linkIn your storage accounts please go to “Networking” -> “Private endpoint connections” and click Add Private Endpoint.It is important to add private links in ...
- 26600 Views
- 4 replies
- 26 kudos
- 1237 Views
- 1 replies
- 0 kudos
STATEMENT_TIMEOUT on a specific SQL Warehouse
Hi, I would like to se STATEMENT_TIMEOUT for a specific SQL warehouse and not on a global level.How would I do that?P.s. I would like to avoid it on a session level, just one-time configuration for a given SQL warehouse.
- 1237 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately we do not support that. We only support Global and Session level settings. We have an internal feature request for this (DB-I-6556 ) but it has not been prioritized in the Roadmap.
- 0 kudos
- 726 Views
- 2 replies
- 0 kudos
Error when trying to run model in staging
i am learning dbt and pache airflow i am using the samples catalog and tpch schema/databasewhen i try to run a sql query in my staging folder I get this error - I am using the premium trial version for databricks09:30:21 Running with dbt=1.8.8using l...
- 726 Views
- 2 replies
- 0 kudos
- 0 kudos
The error indicates that the command you are trying to run is not supported in UC, can you please share what is the SQL command you are currently running?
- 0 kudos
- 736 Views
- 2 replies
- 0 kudos
Configuring DLT _delta_logs with Log Analytics Workspace on Job Clusters
Hi,How do I configure my DLT (Delta Live Table pipeline notebook) _delta_logs with my Azure Log Analytics workspace? I'm encountering issues because the pipeline runs on a job cluster, which doesn't allow me to specify the destination of the log file...
- 736 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Alberto_Umana ,The error I received was related to cells not being connected to the DLT pipeline, as mentioned in my other post, "Cannot run a cell when connected to the pipeline Databricks." However, after browsing the web, I realized that there...
- 0 kudos
- 2531 Views
- 1 replies
- 0 kudos
How could we automatically re run the complete job if any of its associted task fails.?
I need to re run the compete job automatically if any of its associated task gets failed, any help would be appreciable. Thanks
- 2531 Views
- 1 replies
- 0 kudos
- 0 kudos
You could use the suggestions being provided in Community post https://community.databricks.com/t5/data-engineering/need-to-automatically-rerun-the-failed-jobs-in-databricks/td-p/89074
- 0 kudos
- 870 Views
- 1 replies
- 1 kudos
Resolved! DLT: "cannot run a cell when connected to pipeline databricks"
Hi,I have several different cells in my notebook that are connected to a DLT pipeline. Why are some cells skipped and others aren't?I get the message "cannot run a cell when connected to the pipeline Databricks" when try running a cell when I'm conne...
- 870 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, @mkEngineer When working with Delta Live Tables (DLT) in Databricks, you cannot run individual cells interactively as you would in a standard Databricks notebook.DLT Pipeline Behavior:Delta Live Tables notebooks are executed as part of a managed ...
- 1 kudos
- 3184 Views
- 9 replies
- 5 kudos
Resolved! How to access bronze dlt in silver dlt
I have a job in Workflows thatt runs two DLT pipelines, one for Bronze_Transaction and on for Silver_Transaction. The reason for two DLT pipelines is because i want the tables to be created in bronze catalog and erp schema, and silver catalog and erp...
- 3184 Views
- 9 replies
- 5 kudos
- 5 kudos
Final solution for the Bronze:# Define view as the source@dlt.viewdef Transactions_Bronze_View(): return ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option("inferSchema", True) .option(...
- 5 kudos
- 1138 Views
- 3 replies
- 0 kudos
Resolved! Set up Loacation using widget
I'm struggeling using the databricks widget to set up the location in an sql create table statement. I tried the following to set up the location:Step1: Creating a notebook (Notebook1) to define the variable.Location_Path = 'abfss:xxxxx@xxxx.xxx.ne...
- 1138 Views
- 3 replies
- 0 kudos
- 0 kudos
I finaly solved my problem by using the parameters in python F-Strings:Location_Path = dbutils.widgets.text("Location_Path","") -- create table using widget query = f""" CREATE OR REPLACE TABLE schema.Tabelname1 LOCATION '{Location_Path}' AS SELECT...
- 0 kudos
- 943 Views
- 1 replies
- 0 kudos
Running child job under parent job using run_job_task
Hi Community,I am trying to call another job under a workflow job using run_job_task. Currently I am manually providing job_id of the child job. I want to know if there is any way to pass job_name instead of run_id. This will automate the deployment ...
- 943 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @holychs ,It is possible to do using lookup in Databricks Asset Bundles.You define the job id variable that finds id of the job based on its name and use this variable to specify job_id in the run_job_task. Here is the code: variables: my_job_id...
- 0 kudos
- 825 Views
- 2 replies
- 0 kudos
Resolved! Concurrent Workflow Jobs
Hi Community, I am trying to run a Databricks workflow job using run_job_task under a for_loop. I have set the concurrent jobs as 2. I can see 2 iteration jobs getting triggered successfully. But both fail with an error:"ConnectException: Connection ...
- 825 Views
- 2 replies
- 0 kudos
- 0 kudos
It was an internal bug resolved with managing different parameters for each loop jobs.
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
