- 3556 Views
- 1 replies
- 0 kudos
Billing of Databricks Job clusters
Hi All,Please help me understand how the billing is calculated for using the Job cluster.Document says they are charged hourly basis, so if my job ran for 1hr 30mins then will be charged for the 30mins based on the hourly rate or it will be charged f...
- 3556 Views
- 1 replies
- 0 kudos
- 0 kudos
Job clusters consume DBUs per hour depending on the VM size. The Databricks billing happens at "per second granularity", see here. That means if you run your job for 1.5 hours, you will be charged DBUs/hour*1.5*SKU_price; accordingly, if you run your...
- 0 kudos
- 6437 Views
- 5 replies
- 0 kudos
SQL code against tables with '>' in headers suddenly failing?
Just want to post this issue we're experiencing here in case other people are facing something similar. Below is the wording of the support ticket request I've raised:SQL code that has been working is suddenly failing due to syntax errors today. Ther...
- 6437 Views
- 5 replies
- 0 kudos
- 0 kudos
The point that we've got to with this is that MS Support / Databricks have acknowledged that they did something and are working on a fix. "The issue occurred due to the regression in the recent DBR maintenance release...Our engineering team is workin...
- 0 kudos
- 7772 Views
- 6 replies
- 2 kudos
Autoingest not working with Unity Catalog in DLT pipeline
Hey Everyone,I've built a very simple pipeline with a single DLT using auto ingest, and it works, provided I don't specify the output location. When I build the same pipeline but set UC as the output location, it fails when setting up S3 notification...
- 7772 Views
- 6 replies
- 2 kudos
- 2 kudos
Hey @Babu_Krishnan I was! I had to reach out to my Databricks support engineer directly and the resolution was to add "cloudfiles.awsAccessKey" and "cloudfiles.awsSecretKey" to the params as in the screenshot below (apologies, i don't know why the sc...
- 2 kudos
- 22657 Views
- 4 replies
- 4 kudos
Resolved! Using "Select Expr" and "Stack" to Unpivot PySpark DataFrame doesn't produce expected results
I am trying to unpivot a PySpark DataFrame, but I don't get the correct results.Sample dataset:# Prepare Data data = [("Spain", 101, 201, 301), \ ("Taiwan", 102, 202, 302), \ ("Italy", 103, 203, 303), \ ("China", 104, 204, 304...
- 22657 Views
- 4 replies
- 4 kudos
- 4 kudos
You can also use backticks around the column names that would otherwise be recognised as numbers.from pyspark.sql import functions as F unpivotExpr = "stack(3, '2018', `2018`, '2019', `2019`, '2020', `2020`) as (Year, CPI)" unPivotDF = df.select("C...
- 4 kudos
- 4074 Views
- 1 replies
- 0 kudos
Delete on streaming table and starting startingVersion
I deleted for mistake some records from a streaming table, and of course, the streaming job stopped working. So I restored the table at the version before the delete was done, and attempted to restart the job using the startingVersion to the new vers...
- 4074 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @6502, It appears you've used the `startingVersion` parameter in your streaming query, which causes the stream to begin processing data from the version prior to the DELETE operation version. However, the DELETE operation will still be processe...
- 0 kudos
- 1642 Views
- 0 replies
- 0 kudos
BUG: Unity Catalog kills UDF
We have UDFs in a few locations and today we noticed they died in performance. This seems to be caused by Unity Catalog.Test environment 1:Databricks Runtime Environment: 14.3 / 15.1Compute: 1 master, 4 nodesPolicy: UnrestrictedAccess Mode: SharedTes...
- 1642 Views
- 0 replies
- 0 kudos
- 3190 Views
- 2 replies
- 0 kudos
Query table based on table_name from information_schema
Hi,I have one table that changes the name every 60 days. The name simple increases the number version, for example:* Firtst 60 days: table_name_v1. After 60 days: table_name_v2 and so on.What i want is to query the table wich name returned in the que...
- 3190 Views
- 2 replies
- 0 kudos
- 0 kudos
The simpliest way would be propably using spark.sql%py tbl_name = 'table_v1' df = spark.sql(f'select * from {tbl_name}') display(df) From there, You can simply create temporary view:%py df.createOrReplaceTempView('table_act')and query it using SQL st...
- 0 kudos
- 5576 Views
- 5 replies
- 2 kudos
AutoLoader File notification mode Configuration with AWS
from pyspark.sql import functions as F from pyspark.sql import types as T from pyspark.sql import DataFrame, Column from pyspark.sql.types import Row import dlt S3_PATH = 's3://datalake-lab/XXXXX/' S3_SCHEMA = 's3://datalake-lab/XXXXX/schemas/' ...
- 5576 Views
- 5 replies
- 2 kudos
- 5005 Views
- 4 replies
- 1 kudos
Resolved! DLT with UC Access Denied sqs
I am going to use the newly released DLT with UC.But it keeps getting access denied. As I keep tracking the reasons, it seems that an account. ID other than my account ID or Databricks account ID is being requested.I cannot use '*' in principal attri...
- 5005 Views
- 4 replies
- 1 kudos
- 1 kudos
Every service on AWS, an SQS queue, and all the other services in your stack using that queue will be configured with minimal permissions, leading to access issues. So, make sure you get your IAM policies set up correctly before deploying to producti...
- 1 kudos
- 10348 Views
- 4 replies
- 3 kudos
Change {{job.start_time.[iso_date]}} Timezone
I am trying to schedule some jobs using workflows and leveraging dynamic variables. One caveat is that when I try to use {{job.start_time.[iso_date]}} it seems to be defaulted to UTC, is there a way to change it?
- 10348 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi, all the dynamic values are in UTC (documentation). Maybe you can use the code like the one presented below + pass the variables between tasks (see Share information between tasks in a Databricks job) ? %python from datetime import datetime, timed...
- 3 kudos
- 4572 Views
- 2 replies
- 0 kudos
Update data type of a column within a table that has a GENERATED ALWAYS AS IDENTITY-column
I want to cast the data type of a column "X" in a table "A" where column "ID" is defined as GENERATED ALWAYS AS IDENTITY. Databricks refer to overwrite to achieve this: https://docs.databricks.com/delta/update-schema.htmlThe following operation:(spar...
- 4572 Views
- 2 replies
- 0 kudos
- 0 kudos
Update is not working but delete is when i changed to DEFAULT property AnalysisException: UPDATE on IDENTITY column "XXXX_ID" is not supported.
- 0 kudos
- 1960 Views
- 0 replies
- 0 kudos
Pytest on Notebook
I am currently exploring testing methodologies for Databricks notebooks and would like to inquire whether it's possible to write pytest tests for notebooks that contain code not encapsulated within functions or classes.***********************a = 4b ...
- 1960 Views
- 0 replies
- 0 kudos
- 9262 Views
- 4 replies
- 0 kudos
Parallel execution of SQL cell in Databricks Notebooks
Hi Team,Please provide guidance on enabling SQL cells parallel execution in a notebook containing multiple SQL cells. Currently, when we execute notebook and all the SQL cells they run sequentially. I would appreciate assistance on how to execute th...
- 9262 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Phani1 Yes you can achieve this scenario with the help of Databricks Workflow jobs where you can create task and dependencies for each other.
- 0 kudos
- 5646 Views
- 5 replies
- 0 kudos
why the userIdentity is anonymous?
Do you know why the userIdentity is anonymous in AWS Cloudtail's logs even though I have specified an instance profile?
- 5646 Views
- 5 replies
- 0 kudos
- 0 kudos
If you're using AssumeRole to switch roles, make sure that the assumed role session is being used correctly. The Security Token Service (STS) is responsible for issuing temporary security credentials when assuming roles. Ensure that your EC2 instance...
- 0 kudos
- 12489 Views
- 3 replies
- 2 kudos
My cluster can't access any tables in data catalogs
My goal is to have table access control in the data science and engineering workspace. So I enabled access control to my cluster using this config "spark.databricks.acl.dfAclsEnabled": "true" and my cluster is shown as Table ACLs enabled now (shield ...
- 12489 Views
- 3 replies
- 2 kudos
- 2 kudos
Here is my use case: https://community.databricks.com/t5/data-engineering/structured-streaming-using-delta-as-source-and-delta-as-sink-and/td-p/67825And I get this error: "py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.Datase...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
