- 990 Views
- 1 replies
- 1 kudos
Unable to change/cast column datatype using Delta IO
I have created a delta table using Delta IO library, with following detailsTable Name: EmployeeColumns {Id Integer, name String, gender String, Salary decimal(5,2)}Now I want to upcast the salary from decimal(5,2) to decimal(10,4). If I use delta IO ...
- 990 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Srikanthn , Yes, you can change the datatype of a column in a Delta table without losing the delta log. The delta log provides transaction history which is essential for time travel in delta tables. Here is the method to change the datatype of a...
- 1 kudos
- 2000 Views
- 1 replies
- 0 kudos
Update values in DataFrame with values from another DataFrame
Hi, I have to data sources: "admin table" which contains active and inactive employees and I have "hr table" which contains only active employees.I need to update admin table with data from hr table:You can see that I need to update employee with num...
- 2000 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AleksandraFrolo , To update your "admin" table with data from the "hr" table, you can use the MERGE INTO statement provided by Databricks Delta. This operation is known as an upsert (update or insert). Here's an example of how you can do it: sqlM...
- 0 kudos
- 989 Views
- 1 replies
- 0 kudos
Connecting to databricks ipython kernel from VSCode
I'd like to run python notebooks (.ipynb) from VSCode connecting to the ipython kernel from databricks. I have already connected to an execution cluster from VSCode and am able to run python scripts (.py files) and see the output on my local console....
- 989 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rotate_employee, I recommend checking Databricks or VSCode documentation or file a support ticket for further assistance.
- 0 kudos
- 2396 Views
- 6 replies
- 0 kudos
Need help migrating company customer and partner academy accounts to work properly
Hi, originally I accidentally made a customer academy account with my company that is a databricks partner. Then I made an account using my personal email and listed my company email as the partner email for the partner academy account. that account ...
- 2396 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Maria_fed Thanks again, i have assigned your case to my colleague and you should hearing from them soon. Regards, Akshay
- 0 kudos
- 1052 Views
- 2 replies
- 1 kudos
Getting FileNotFoundException while using cloudFiles
Hi,Following is the code i am using the ingest the data incrementally (weekly).val ssdf = spark.readStream.schema(schema) .format("cloudFiles").option("cloudFiles.format", "parquet").load(sourceUrl).filter(criteriaFilter)val transformedDf = ssdf.tran...
- 1052 Views
- 2 replies
- 1 kudos
- 792 Views
- 1 replies
- 0 kudos
RBAC, Security & Privacy controls
Could you please share us best practices on implementation of RBAC, Security & Privacy controls in Databricks
- 792 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please check on https://docs.databricks.com/en/lakehouse-architecture/security-compliance-and-privacy/best-practices.html and see if this helping? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
- 659 Views
- 1 replies
- 0 kudos
ganglia metrics
Hello Everyone, I have build this script in order to collect ganglia metrics but the size of stderr and sdtout ganglia is 0. It doesn't work. I Have put this script on Workspace due to migration databricks all init-script should be place on Workspace...
- 659 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Is there any error you are getting? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
- 791 Views
- 2 replies
- 0 kudos
GCP hosted Databricks - DBFS temp files - Not Found
I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a "file path n...
- 791 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, the error code snippet with the whole error may help to determine the issue, also, considering the above points may also work as a fix.
- 0 kudos
- 1231 Views
- 2 replies
- 0 kudos
Resolved! Unable to find permission button in Sql Warehouse for providing Access
Hi Everyone, am unable to see the permission button in sql warehouse to provide access to other users.I have admin rights and databricks is premium subscription.
- 1231 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Could you please provide a screenshot of the SQL warehouse? Also, you can go through: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.htmlAlso, please tag @Debayan with your next comment which will notify me. Th...
- 0 kudos
- 623 Views
- 1 replies
- 1 kudos
SQL Serverless - cost view
Hi,Anyone knows how I'm able to monitor cost of the SQL Serverless? I'm using Databricks in Azure and I'm not sure where to find cost generated by compute resources hosted on Databricks.
- 623 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can calculate the pricing in https://www.databricks.com/product/pricing/databricks-sql also, https://azure.microsoft.com/en-in/pricing/details/databricks/#:~:text=Sign%20in%20to%20the%20Azure,asked%20questions%20about%20Azure%20pricing. For A...
- 1 kudos
- 589 Views
- 1 replies
- 0 kudos
ListBucket
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<s3-bucket-name>" ] }, { "Effect": "Allow", "Action": [ "s3:Pu...
- 589 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Cryptocurentcyc, The version in the given JSON is "Privilege Model version 1.0". The statement in the JSON is about upgrading to Privilege Model version 1.0 to take advantage of privilege inheritance and new features. It also highlights the diffe...
- 0 kudos
- 1995 Views
- 2 replies
- 0 kudos
Monitoring job metrics
Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...
- 1995 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Bagger, To monitor the metrics you specified, you can use a combination of Databricks features and Prometheus: 1. **Failed Jobs:** You can monitor failed jobs using Databricks’ built-in job monitoring capabilities. The status of each job run, inc...
- 0 kudos
- 1377 Views
- 1 replies
- 0 kudos
Spark Remote error when connecting to cluster
Hi, I am using the latest version of pyspark and I am trying to connect to a remote cluster with runtime 13.3.My doubts are:- Do i need databricks unity catalog enabled?- My cluster is already in a Shared policy in Access Mode, so what other configur...
- 1377 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Is your workspace is already unity catalog enabled? Also, did you go through the considerations for enabling workspace for unity catalog? https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html#considerations-before-yo...
- 0 kudos
- 5303 Views
- 6 replies
- 1 kudos
Resolved! Unable to create table with primary key
Hi Team,Getting below error while creating a table with primary key,"Table constraints are only supported in Unity Catalog."Table script : CREATE TABLE persons(first_name STRING NOT NULL, last_name STRING NOT NULL, nickname STRING,CONSTRAINT persons_...
- 5303 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi, this needs further investigation, could you please raise a support case with Databricks?
- 1 kudos
- 1028 Views
- 1 replies
- 0 kudos
Problem starting cluster
I try to start cluster that i used to start it 7 times before and it gave me this error Cloud provider is undergoing a transient resource throttling. This is retryable. 1 out of 2 pods scheduled. Failed to launch cluster in kubernetes in 1800 seconds...
- 1028 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This error "GCE out of resources" typically means that Google compute engine is out of resources as in out of nodes (can be a quota issue or can be node issues in that particular region in GCP). Could you please raise a google support case on thi...
- 0 kudos
