cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

nistrate
by New Contributor III
  • 6365 Views
  • 2 replies
  • 5 kudos

Resolved! Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

Hello Databricks Community,I am seeking assistance understanding the possibility and procedure of implementing a workflow restriction mechanism in Databricks. Our aim is to promote a better workflow management and ensure the quality of the notebooks ...

  • 6365 Views
  • 2 replies
  • 5 kudos
Latest Reply
Avvar2022
Contributor
  • 5 kudos

I believe this has to happen in 2 steps.step1: Currently admin can't restrict workflow creation in databricks  currently any user with workspace access can create workflows. Admins should be able to restrict workflow creation. Databricks doesn't have...

  • 5 kudos
1 More Replies
CaptainJack
by New Contributor II
  • 205 Views
  • 1 replies
  • 0 kudos

Giving coworker "runing" permision on workflow but without allowing him access to notebooks.

I noticed that there is can_manage_run permission on workflow level, and someone can run a workflow only with these permission (without needing can_run permission on notebook level). Problem is that coworker can go to run details and then click on ta...

  • 205 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ravivarma
New Contributor III
  • 0 kudos

Hello @CaptainJack , In Databricks, the can_manage_run permission lets a user manage workflow executions but does not hide the code in the tasks. If someone has this permission, they can see the details and code of the workflow runs. At...

  • 0 kudos
AhsanKhawaja
by New Contributor
  • 3386 Views
  • 4 replies
  • 0 kudos

using databricks sql warehouse as web app backend

Hi,I wanted to ask if anyone is using Databricks SQL Warehouse as backend for small to large scale web application? What are your thoughts about it, specially what Databricks team thinks of it ?Kind Regards,A

  • 3386 Views
  • 4 replies
  • 0 kudos
Latest Reply
Robert-Scott
New Contributor II
  • 0 kudos

Using Databricks SQL Warehouse as a backend for a web application involves integrating Databricks with your web app to handle data processing, querying, and analytics. Here are the steps to achieve this:1. Set Up Databricks SQL WarehouseCreate a Data...

  • 0 kudos
3 More Replies
yj940525
by New Contributor II
  • 325 Views
  • 0 replies
  • 0 kudos

question of changing cluster key in liquid cluster

If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...

  • 325 Views
  • 0 replies
  • 0 kudos
yatharth
by New Contributor III
  • 4073 Views
  • 1 replies
  • 1 kudos

AWS CLI Commands

I wish to run aws CLI command in databricks, is there a way i can achieve the same, to be more specific i would like to run:aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-...

  • 4073 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 1 kudos

@yatharth please check this: https://docs.databricks.com/en/compute/access-mode-limitations.html#network-and-file-system-access-limitations-for-unity-catalog-shared-access-mode:~:text=Cannot%20connect%20to%20the%20instance%20metadata%20service%20(IMD...

  • 1 kudos
harichand
by New Contributor
  • 358 Views
  • 1 replies
  • 0 kudos

Missing Spark SQL Warehouse Folder in New Databricks Workspace (GCP)

We recently created a new Databricks workspace on the GCP platform. We noticed that the Spark SQL warehouse folder (/user/hive/warehouse) is not present by default, unlike in our previous workspaces. In our earlier Databricks workspaces, this folder ...

Data Engineering
warehouse hive
  • 358 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@harichand, could you please attach the screenshot where you see this path '/user/hive/warehouse' in both workspaces? [Old one and the new one]

  • 0 kudos
Karlo_Kotarac
by New Contributor III
  • 492 Views
  • 2 replies
  • 0 kudos

Different error handling behavior after DB runtime upgrade from 13.3 to 14.3

Hi! We want to upgrade the DB runtime on our clusters from 13.3 LTS to 14.3 LTS. Currently, everything looks good except for the different error-handling in the new runtime.For example, the error in the 13.3 LTS runtime looks familiar:while the same ...

Karlo_Kotarac_0-1717743717317.png Karlo_Kotarac_1-1717743788282.png Karlo_Kotarac_3-1717743912995.png
  • 492 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@Karlo_Kotarac Where do you see this error:   

  • 0 kudos
1 More Replies
Volker
by New Contributor III
  • 413 Views
  • 0 replies
  • 0 kudos

Asset Bundles cannot run job with single node job cluster

Hello community,we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:resources: jobs: example_job: name: example_job tasks: - task_key: main_task ...

  • 413 Views
  • 0 replies
  • 0 kudos
thiagoawstest
by Contributor
  • 734 Views
  • 1 replies
  • 0 kudos

Resolved! mount bucket s3

Hi, I have Databricks configured on AWS, I need to mount some S3 buckets on Databricks in /mnt, but I have some questions:- How can a bucket be mounted for all clusters and users to have access to, so as not to need to mount it every time the cluster...

  • 734 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@thiagoawstest To mount an S3 bucket in Databricks on AWS so that all clusters and users have access to it without needing to remount each time, and without creating an access key in AWS, follow these steps:  Mounting an S3 Bucket Using an AWS Instan...

  • 0 kudos
NaeemS
by New Contributor III
  • 381 Views
  • 2 replies
  • 0 kudos

Custom transformers with mlflow

Hi Everyone,I have created a spark pipeline in which I have a stage which is a Custom Transformer. Now I am using feature stores to log my model. But the issue is that the custom Transformer stage is not serialized properly and is not logged along wi...

  • 381 Views
  • 2 replies
  • 0 kudos
Latest Reply
NaeemS
New Contributor III
  • 0 kudos

Hi @Kaniz_Fatma , Can you please guide me what are the additional steps I'll need to handle serialization of Custom transformers so I can use it in my model pipeline via feature stores.Thanks!

  • 0 kudos
1 More Replies
saikumar_ganji
by New Contributor III
  • 879 Views
  • 8 replies
  • 0 kudos

DATABRICKS DATA ENGINEER ASSOCIATE EXAM GOT SUSPENDED

I encountered Pathetic experience while attempting my Databricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times and then suspended my exam, saying I have exceeded eyes movement and I almost comple...

  • 879 Views
  • 8 replies
  • 0 kudos
Latest Reply
saikumar_ganji
New Contributor III
  • 0 kudos

@Cert-Team @Cert-TeamOPS @Kaniz_Fatma Can you please look into this issue. I have to complete my exam asap

  • 0 kudos
7 More Replies
Karlo_Kotarac
by New Contributor III
  • 1062 Views
  • 4 replies
  • 0 kudos

Run failed with error message ContextNotFound

Hi all!Recently we've been getting lots of these errors when running Databricks notebooks:At that time we observed DRIVER_NOT_RESPONDING (Driver is up but is not responsive, likely due to GC.) log on the single-user cluster we use.Previously when thi...

Karlo_Kotarac_0-1713422302017.png
  • 1062 Views
  • 4 replies
  • 0 kudos
Latest Reply
Karlo_Kotarac
New Contributor III
  • 0 kudos

In case somebody else runs into the same issue: After investigation from Databricks support the conclusion was that the driver's memory was overloaded ('Driver Not Responding' error message in the event log) but it can happen that we don't get the co...

  • 0 kudos
3 More Replies
thiagoawstest
by Contributor
  • 571 Views
  • 1 replies
  • 0 kudos

Resolved! add active directory group permission

Hi, I'm using Databricks on AWS, I did the single sign-on integration with Azure extra ID (active directory), everything is working fine, I can add users, but when I try to add a group that was created in AD, it can't be found the group.How should I ...

  • 571 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @thiagoawstest, To make Databricks find groups created in Azure Active Directory (AD), you can use the System for Cross-domain Identity Management (SCIM).SCIM is an open standard that allows you to automate user provisioning in Databricks.

  • 0 kudos
Lily99
by New Contributor
  • 763 Views
  • 1 replies
  • 0 kudos

SQL function does not work in 'Create Function'

This SQL statement works fine by itself SELECT COUNT(1) FROM tablea f INNER JOIN tableb t ON lower(f.col1) = t.col1but if I want to use it inside a function:​CREATE OR REPLACE FUNCTION fn_abc(var1 ...

  • 763 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
New Contributor III
  • 0 kudos

Hello @Lily99 , I hope this message finds you well. Could you please try the code below and let me know the results? CREATE OR REPLACE FUNCTION fn_abc(var1 STRING, var2 STRING) RETURNS DOUBLECOMMENT 'test function'RETURN SELECT    CASE    WHEN EXISTS...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels