- 13080 Views
- 6 replies
- 0 kudos
Resolved! Why is Photon increasing DBU used per hour?
I noticed that enabling photon acceleration is increasing the number of DBU utilized per hour which in turn increases our cost.In light of this, I am interested in gaining clarity on the costing of Photon acceleration as I was led to believe that Pho...
- 13080 Views
- 6 replies
- 0 kudos
- 0 kudos
well that depends on what kinds of tests you do. In data warehousing there are different kinds of loads.What have you tested? Data transformations or analytical queries. Because for the latter databricks sql is a better choice than a common spark ...
- 0 kudos
- 1910 Views
- 1 replies
- 0 kudos
Resolved! Accessing a series in a DataFrame
Frequently I see this syntax to access a series in DBX. df['column_name'] However, I get this as my output from that.Column<'derived_value'>What's the correct way to access a series
- 1910 Views
- 1 replies
- 0 kudos
- 0 kudos
I realized I was looking at the wrong dataframe type.I needed a Pandas dataframe, not a databricks dataframe.
- 0 kudos
- 5099 Views
- 2 replies
- 1 kudos
Create a table name without back tick when using set variable
When i tried to create a table name with variable like this-- Set a string variableSET table_suffix = 'suffix';-- Use dynamic SQL to create a table with the variable as a suffix in the table nameCREATE TABLE IF NOT EXISTS <dbname>.my_table_${table_su...
- 5099 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,It's possible that the `identifier` clause is what you're looking for (https://docs.databricks.com/en/sql/language-manual/sql-ref-names-identifier-clause.html#identifier-clause). If so, this basic example should work: DECLARE mytab = '`mycatalog`....
- 1 kudos
- 3449 Views
- 1 replies
- 0 kudos
Filter in DBX dashboards
How to add a column from Table1 as a filter to Table2 (Also called as on-click action filter) in databricks Dashboards?Both the tables are getting data through sql query
- 3449 Views
- 1 replies
- 0 kudos
- 0 kudos
To add a column from Table1 as a filter to Table2 in Databricks Dashboards, you can use the dashboard parameters feature. Here are the steps: Create a visualization for each of your SQL queries. You can do this by clicking the '+' next to the Resul...
- 0 kudos
- 1733 Views
- 0 replies
- 0 kudos
Metric Flow Commands Not Recognised in Data bricks dbt core Workflows
Hi Databricks community,I'm working on a project that combines dbt core with MetricFlow for metric management. I've set up my pipeline within a Databricks workflow, using the "dbt" workflow task type. While standard dbt commands (like dbt debug or d...
- 1733 Views
- 0 replies
- 0 kudos
- 8837 Views
- 3 replies
- 2 kudos
Resolved! Union and Column data types
I have three data frames that I create in python. I want to write all three of these to the same delta table. In code I bring the three of them together using the union operation.When I do this the data in the columns is no longer aligned correctly.I...
- 8837 Views
- 3 replies
- 2 kudos
- 2 kudos
Aligning the data types and column order across all three data frames before attempting to union them together solved the problem. The below snippet highlights what was happening.data = [[2021, "test", "Albany", "M", 42]] df1 = spark.createDataFrame...
- 2 kudos
- 2144 Views
- 0 replies
- 0 kudos
Asset Bundles - YAML file replacing job cluster with an all-purpose cluster
Hi all,I have been having some trouble running a workflow that consists of 3 tasks that run sequentially. Task1 runs on an all-purpose cluster and kicks off Task2 that needs to run on a job cluster. Task2 kicks off Task3 which also uses a job cluster...
- 2144 Views
- 0 replies
- 0 kudos
- 11407 Views
- 2 replies
- 5 kudos
Resolved! Restricting Workflow Creation and Implementing Approval Mechanism in Databricks
Hello Databricks Community,I am seeking assistance understanding the possibility and procedure of implementing a workflow restriction mechanism in Databricks. Our aim is to promote a better workflow management and ensure the quality of the notebooks ...
- 11407 Views
- 2 replies
- 5 kudos
- 5 kudos
I believe this has to happen in 2 steps.step1: Currently admin can't restrict workflow creation in databricks currently any user with workspace access can create workflows. Admins should be able to restrict workflow creation. Databricks doesn't have...
- 5 kudos
- 2457 Views
- 1 replies
- 0 kudos
Giving coworker "runing" permision on workflow but without allowing him access to notebooks.
I noticed that there is can_manage_run permission on workflow level, and someone can run a workflow only with these permission (without needing can_run permission on notebook level). Problem is that coworker can go to run details and then click on ta...
- 2457 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @CaptainJack , In Databricks, the can_manage_run permission lets a user manage workflow executions but does not hide the code in the tasks. If someone has this permission, they can see the details and code of the workflow runs. At...
- 0 kudos
- 1162 Views
- 0 replies
- 0 kudos
question of changing cluster key in liquid cluster
If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...
- 1162 Views
- 0 replies
- 0 kudos
- 4372 Views
- 5 replies
- 1 kudos
Resolved! How to migrate Delta live table.
I need guidance on migrating the Delta live table to another pipeline without a full refresh
- 4372 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi, Yes you can use the same checkpoint location in the new pipeline.
- 1 kudos
- 8012 Views
- 1 replies
- 1 kudos
AWS CLI Commands
I wish to run aws CLI command in databricks, is there a way i can achieve the same, to be more specific i would like to run:aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-...
- 8012 Views
- 1 replies
- 1 kudos
- 1 kudos
@yatharth please check this: https://docs.databricks.com/en/compute/access-mode-limitations.html#network-and-file-system-access-limitations-for-unity-catalog-shared-access-mode:~:text=Cannot%20connect%20to%20the%20instance%20metadata%20service%20(IMD...
- 1 kudos
- 2025 Views
- 1 replies
- 0 kudos
Missing Spark SQL Warehouse Folder in New Databricks Workspace (GCP)
We recently created a new Databricks workspace on the GCP platform. We noticed that the Spark SQL warehouse folder (/user/hive/warehouse) is not present by default, unlike in our previous workspaces. In our earlier Databricks workspaces, this folder ...
- 2025 Views
- 1 replies
- 0 kudos
- 0 kudos
@harichand, could you please attach the screenshot where you see this path '/user/hive/warehouse' in both workspaces? [Old one and the new one]
- 0 kudos
- 3660 Views
- 2 replies
- 0 kudos
Different error handling behavior after DB runtime upgrade from 13.3 to 14.3
Hi! We want to upgrade the DB runtime on our clusters from 13.3 LTS to 14.3 LTS. Currently, everything looks good except for the different error-handling in the new runtime.For example, the error in the 13.3 LTS runtime looks familiar:while the same ...
- 3660 Views
- 2 replies
- 0 kudos
- 5460 Views
- 1 replies
- 0 kudos
Resolved! mount bucket s3
Hi, I have Databricks configured on AWS, I need to mount some S3 buckets on Databricks in /mnt, but I have some questions:- How can a bucket be mounted for all clusters and users to have access to, so as not to need to mount it every time the cluster...
- 5460 Views
- 1 replies
- 0 kudos
- 0 kudos
@thiagoawstest To mount an S3 bucket in Databricks on AWS so that all clusters and users have access to it without needing to remount each time, and without creating an access key in AWS, follow these steps: Mounting an S3 Bucket Using an AWS Instan...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
