- 2603 Views
- 1 replies
- 0 kudos
Giving coworker "runing" permision on workflow but without allowing him access to notebooks.
I noticed that there is can_manage_run permission on workflow level, and someone can run a workflow only with these permission (without needing can_run permission on notebook level). Problem is that coworker can go to run details and then click on ta...
- 2603 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @CaptainJack , In Databricks, the can_manage_run permission lets a user manage workflow executions but does not hide the code in the tasks. If someone has this permission, they can see the details and code of the workflow runs. At...
- 0 kudos
- 1204 Views
- 0 replies
- 0 kudos
question of changing cluster key in liquid cluster
If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...
- 1204 Views
- 0 replies
- 0 kudos
- 4706 Views
- 5 replies
- 1 kudos
Resolved! How to migrate Delta live table.
I need guidance on migrating the Delta live table to another pipeline without a full refresh
- 4706 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi, Yes you can use the same checkpoint location in the new pipeline.
- 1 kudos
- 8184 Views
- 1 replies
- 1 kudos
AWS CLI Commands
I wish to run aws CLI command in databricks, is there a way i can achieve the same, to be more specific i would like to run:aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-...
- 8184 Views
- 1 replies
- 1 kudos
- 1 kudos
@yatharth please check this: https://docs.databricks.com/en/compute/access-mode-limitations.html#network-and-file-system-access-limitations-for-unity-catalog-shared-access-mode:~:text=Cannot%20connect%20to%20the%20instance%20metadata%20service%20(IMD...
- 1 kudos
- 2170 Views
- 1 replies
- 0 kudos
Missing Spark SQL Warehouse Folder in New Databricks Workspace (GCP)
We recently created a new Databricks workspace on the GCP platform. We noticed that the Spark SQL warehouse folder (/user/hive/warehouse) is not present by default, unlike in our previous workspaces. In our earlier Databricks workspaces, this folder ...
- 2170 Views
- 1 replies
- 0 kudos
- 0 kudos
@harichand, could you please attach the screenshot where you see this path '/user/hive/warehouse' in both workspaces? [Old one and the new one]
- 0 kudos
- 3853 Views
- 2 replies
- 0 kudos
Different error handling behavior after DB runtime upgrade from 13.3 to 14.3
Hi! We want to upgrade the DB runtime on our clusters from 13.3 LTS to 14.3 LTS. Currently, everything looks good except for the different error-handling in the new runtime.For example, the error in the 13.3 LTS runtime looks familiar:while the same ...
- 3853 Views
- 2 replies
- 0 kudos
- 5655 Views
- 1 replies
- 0 kudos
Resolved! mount bucket s3
Hi, I have Databricks configured on AWS, I need to mount some S3 buckets on Databricks in /mnt, but I have some questions:- How can a bucket be mounted for all clusters and users to have access to, so as not to need to mount it every time the cluster...
- 5655 Views
- 1 replies
- 0 kudos
- 0 kudos
@thiagoawstest To mount an S3 bucket in Databricks on AWS so that all clusters and users have access to it without needing to remount each time, and without creating an access key in AWS, follow these steps: Mounting an S3 Bucket Using an AWS Instan...
- 0 kudos
- 5432 Views
- 7 replies
- 0 kudos
DATABRICKS DATA ENGINEER ASSOCIATE EXAM GOT SUSPENDED
I encountered Pathetic experience while attempting my Databricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times and then suspended my exam, saying I have exceeded eyes movement and I almost comple...
- 5432 Views
- 7 replies
- 0 kudos
- 0 kudos
@Cert-Team @Cert-TeamOPS @Retired_mod Can you please look into this issue. I have to complete my exam asap
- 0 kudos
- 7287 Views
- 4 replies
- 0 kudos
Run failed with error message ContextNotFound
Hi all!Recently we've been getting lots of these errors when running Databricks notebooks:At that time we observed DRIVER_NOT_RESPONDING (Driver is up but is not responsive, likely due to GC.) log on the single-user cluster we use.Previously when thi...
- 7287 Views
- 4 replies
- 0 kudos
- 0 kudos
In case somebody else runs into the same issue: After investigation from Databricks support the conclusion was that the driver's memory was overloaded ('Driver Not Responding' error message in the event log) but it can happen that we don't get the co...
- 0 kudos
- 3304 Views
- 1 replies
- 0 kudos
SQL function does not work in 'Create Function'
This SQL statement works fine by itself SELECT COUNT(1) FROM tablea f INNER JOIN tableb t ON lower(f.col1) = t.col1but if I want to use it inside a function:​CREATE OR REPLACE FUNCTION fn_abc(var1 ...
- 3304 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Lily99 , I hope this message finds you well. Could you please try the code below and let me know the results? CREATE OR REPLACE FUNCTION fn_abc(var1 STRING, var2 STRING) RETURNS DOUBLECOMMENT 'test function'RETURN SELECT CASE WHEN EXISTS...
- 0 kudos
- 2753 Views
- 1 replies
- 0 kudos
DLT aggregation problem
I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...
- 2753 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Yash_542965 , I hope this message finds you well. Could you please share a sample of code you are using so that we can check it further? Best regards,Lucas Rocha
- 0 kudos
- 3165 Views
- 1 replies
- 0 kudos
Databricks Notebook error : Analysis Exception with multiple datasets
Hi All,I am getting below error when trying to execute the code.AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to ...
- 3165 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @vijaykumarbotla , I hope you're doing well. This is probably because both DataFrames contain a column with the same name, and Spark is unable to determine which one you are referring to in the select statement. To resolve this issue, you can u...
- 0 kudos
- 2523 Views
- 1 replies
- 0 kudos
- 2523 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @DataAlchemist28 , I hope this message finds you well. Delta Live Tables supports loading data from any data source supported by Databricks. You can find the datasources supported here Connect to data sources, and JDBC is one of them. You can a...
- 0 kudos
- 2038 Views
- 1 replies
- 0 kudos
Exceptions are Not Getting Handled In Autoloader Write Stream
I have below logic implemented using Databricks Autoloader. ## Autoloader Write stream: Its calling forEachBatch function to write into respective datatype catalog table# and using checkpoint to keeps track of processing files.try: ##Observe raw ...
- 2038 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Sambit_S ,In your scenario, there is a merge failure. Your query won't be able to progress as the problematic batch can't be committed to sink. Even if you handle the exception in a try catch block, it's impossible for the autoloader to update...
- 0 kudos
- 3400 Views
- 1 replies
- 0 kudos
Increase stack size Databricks
Hi everyoneI'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this...
- 3400 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
