cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

akshayauser
by New Contributor
  • 1416 Views
  • 2 replies
  • 1 kudos

Create a table name without back tick when using set variable

When i tried to create a table name with variable like this-- Set a string variableSET table_suffix = 'suffix';-- Use dynamic SQL to create a table with the variable as a suffix in the table nameCREATE TABLE IF NOT EXISTS <dbname>.my_table_${table_su...

  • 1416 Views
  • 2 replies
  • 1 kudos
Latest Reply
brockb
Databricks Employee
  • 1 kudos

Hi,It's possible that the `identifier` clause is what you're looking for (https://docs.databricks.com/en/sql/language-manual/sql-ref-names-identifier-clause.html#identifier-clause). If so, this basic example should work: DECLARE mytab = '`mycatalog`....

  • 1 kudos
1 More Replies
nehaa
by New Contributor II
  • 1217 Views
  • 1 replies
  • 0 kudos

Filter in DBX dashboards

How to add a column from Table1 as a filter to Table2 (Also called as on-click action filter) in databricks Dashboards?Both the tables are getting data through sql query 

  • 1217 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

To add a column from Table1 as a filter to Table2 in Databricks Dashboards, you can use the dashboard parameters feature. Here are the steps: Create a visualization for each of your SQL queries. You can do this by clicking the '+' next to the Resul...

  • 0 kudos
high-energy
by New Contributor III
  • 2267 Views
  • 3 replies
  • 2 kudos

Resolved! Union and Column data types

I have three data frames that I create in python. I want to write all three of these to the same delta table. In code I bring the three of them together using the union operation.When I do this the data in the columns is no longer aligned correctly.I...

  • 2267 Views
  • 3 replies
  • 2 kudos
Latest Reply
high-energy
New Contributor III
  • 2 kudos

Aligning the data types and column order across all three data frames before attempting to union them together solved the problem. The below snippet highlights what was happening.data = [[2021, "test", "Albany", "M", 42]] df1 = spark.createDataFrame...

  • 2 kudos
2 More Replies
nistrate
by New Contributor III
  • 7925 Views
  • 2 replies
  • 5 kudos

Resolved! Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

Hello Databricks Community,I am seeking assistance understanding the possibility and procedure of implementing a workflow restriction mechanism in Databricks. Our aim is to promote a better workflow management and ensure the quality of the notebooks ...

  • 7925 Views
  • 2 replies
  • 5 kudos
Latest Reply
Avvar2022
Contributor
  • 5 kudos

I believe this has to happen in 2 steps.step1: Currently admin can't restrict workflow creation in databricks  currently any user with workspace access can create workflows. Admins should be able to restrict workflow creation. Databricks doesn't have...

  • 5 kudos
1 More Replies
CaptainJack
by New Contributor III
  • 677 Views
  • 1 replies
  • 0 kudos

Giving coworker "runing" permision on workflow but without allowing him access to notebooks.

I noticed that there is can_manage_run permission on workflow level, and someone can run a workflow only with these permission (without needing can_run permission on notebook level). Problem is that coworker can go to run details and then click on ta...

  • 677 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ravivarma
Databricks Employee
  • 0 kudos

Hello @CaptainJack , In Databricks, the can_manage_run permission lets a user manage workflow executions but does not hide the code in the tasks. If someone has this permission, they can see the details and code of the workflow runs. At...

  • 0 kudos
yj940525
by New Contributor II
  • 554 Views
  • 0 replies
  • 0 kudos

question of changing cluster key in liquid cluster

If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...

  • 554 Views
  • 0 replies
  • 0 kudos
Therdpong
by New Contributor III
  • 1878 Views
  • 5 replies
  • 1 kudos

Resolved! How to migrate Delta live table.

I need guidance on migrating the Delta live table to another pipeline without a full refresh

  • 1878 Views
  • 5 replies
  • 1 kudos
Latest Reply
Hkesharwani
Contributor II
  • 1 kudos

Hi, Yes you can use the same checkpoint location in the new pipeline.

  • 1 kudos
4 More Replies
yatharth
by New Contributor III
  • 5494 Views
  • 1 replies
  • 1 kudos

AWS CLI Commands

I wish to run aws CLI command in databricks, is there a way i can achieve the same, to be more specific i would like to run:aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-...

  • 5494 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 1 kudos

@yatharth please check this: https://docs.databricks.com/en/compute/access-mode-limitations.html#network-and-file-system-access-limitations-for-unity-catalog-shared-access-mode:~:text=Cannot%20connect%20to%20the%20instance%20metadata%20service%20(IMD...

  • 1 kudos
harichand
by New Contributor
  • 713 Views
  • 1 replies
  • 0 kudos

Missing Spark SQL Warehouse Folder in New Databricks Workspace (GCP)

We recently created a new Databricks workspace on the GCP platform. We noticed that the Spark SQL warehouse folder (/user/hive/warehouse) is not present by default, unlike in our previous workspaces. In our earlier Databricks workspaces, this folder ...

Data Engineering
warehouse hive
  • 713 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@harichand, could you please attach the screenshot where you see this path '/user/hive/warehouse' in both workspaces? [Old one and the new one]

  • 0 kudos
Karlo_Kotarac
by New Contributor III
  • 1457 Views
  • 2 replies
  • 0 kudos

Different error handling behavior after DB runtime upgrade from 13.3 to 14.3

Hi! We want to upgrade the DB runtime on our clusters from 13.3 LTS to 14.3 LTS. Currently, everything looks good except for the different error-handling in the new runtime.For example, the error in the 13.3 LTS runtime looks familiar:while the same ...

Karlo_Kotarac_0-1717743717317.png Karlo_Kotarac_1-1717743788282.png Karlo_Kotarac_3-1717743912995.png
  • 1457 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@Karlo_Kotarac Where do you see this error:   

  • 0 kudos
1 More Replies
thiagoawstest
by Contributor
  • 2303 Views
  • 1 replies
  • 0 kudos

Resolved! mount bucket s3

Hi, I have Databricks configured on AWS, I need to mount some S3 buckets on Databricks in /mnt, but I have some questions:- How can a bucket be mounted for all clusters and users to have access to, so as not to need to mount it every time the cluster...

  • 2303 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@thiagoawstest To mount an S3 bucket in Databricks on AWS so that all clusters and users have access to it without needing to remount each time, and without creating an access key in AWS, follow these steps:  Mounting an S3 Bucket Using an AWS Instan...

  • 0 kudos
NaeemS
by New Contributor III
  • 828 Views
  • 1 replies
  • 0 kudos

Custom transformers with mlflow

Hi Everyone,I have created a spark pipeline in which I have a stage which is a Custom Transformer. Now I am using feature stores to log my model. But the issue is that the custom Transformer stage is not serialized properly and is not logged along wi...

  • 828 Views
  • 1 replies
  • 0 kudos
Latest Reply
NaeemS
New Contributor III
  • 0 kudos

Hi @Retired_mod , Can you please guide me what are the additional steps I'll need to handle serialization of Custom transformers so I can use it in my model pipeline via feature stores.Thanks!

  • 0 kudos
saikumar_ganji
by New Contributor III
  • 2400 Views
  • 7 replies
  • 0 kudos

DATABRICKS DATA ENGINEER ASSOCIATE EXAM GOT SUSPENDED

I encountered Pathetic experience while attempting my Databricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times and then suspended my exam, saying I have exceeded eyes movement and I almost comple...

  • 2400 Views
  • 7 replies
  • 0 kudos
Latest Reply
saikumar_ganji
New Contributor III
  • 0 kudos

@Cert-Team @Cert-TeamOPS @Retired_mod Can you please look into this issue. I have to complete my exam asap

  • 0 kudos
6 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels