Hi Team,Could you please suggest the best way to track the cost of Databricks objects/components? Could you please share any best practices for optimizing costs and conducting detailed cost analysis?Regards,Phanindra
Could you please suggest how to capture changes at the object level in Databricks, such as notebooks changes, table DDL changes, view DDL , functions DDL, and workflows etc. changes? We would like to build a dashboard for changes.
Hi @Phani1 , Good Day! Could you kindly clarify on your question about the capture change? What type of change do you want to capture? If you want to see what modifications the user made to the Notebooks, tables, and workflow, you can check the audi...
Hi, I am trying to create a query definition using API '/api/2.0/preview/sql/queries' in postman but getting internal server error. Below is the snap for the snap. Let me know If I am doing anything wrong here.
I'm interested to know how this error was resolved. I'm getting an "Internal Server Error" returned when trying to create queries with version 1.36.1 of the Databricks Terraform Provider. The error provides no other information.
I was wondering if someone could help us with implementation here. Our current program will spin up 5 jobs through the Databricks API using the same Databricks cluster but each one needs their own spark context (specifically each one will connect to ...
you can set up buckets with different credentials, endpoints, and so on.https://docs.databricks.com/en/connect/storage/amazon-s3.html#per-bucket-configuration
Hi friends;I'm working on a project where we are 4 programmers. We are working in a single environment, using only the "Workspaces" folder. Each has its own user, which is managed by Azure AD.We had a peak in consumption on the 5th Feb. So I can see ...
Hi @Retired_mod , thanks for your quick answer.There is no other way to monitor notebook runs. I ask this because adding tags to the cluster and workspace does not solve my problem, considering that everyone uses the same cluster and the same workspa...
Data Engineer – Databricks - RemoteApply Here: Job Application for Data Engineer – Databricks at Jenzabar (greenhouse.io)Jenzabar Website: Higher Education Software Solutions - JenzabarFor over four decades, the higher education experts at Jenzabar h...
Hi, Newcomer here. I am experimenting with the community version of databrick.I wanted to run the notebook example provided here https://community.cloud.databricks.com/?o=6085264701896358#notebook/2691200955149229It failed because it cannot import th...
Each year, we celebrate the amazing customers that rely on Databricks to innovate and transform their organizations — and the world — with the power of data and AI. The nomination form is now open to submit nominations. Nominations will close on Marc...
I see the current conversion of dataframe to xml need to be improved.My dataframe schema is a perfect nested schema based on structs but when I create a xml I have the follow issues:1) I can't add elements to root2) rootTag and rowTag are requiredIn ...
Here is one of the ways to use the struct field name as rowTag: import org.apache.spark.sql.types._ val schema = new StructType().add("Record", new StructType().add("age", IntegerType).add("name", StringType)) val data = Seq(Row(Row(18, "John ...
Taking from my experience, the default action of widgets triggering code execution upon value change poses risks that outweigh the convenience in certain scenarios. While this feature may seem advantageous in some cases, it can lead to unintended con...
I definitely have to agree with the original point- if you have a notebook that you import, and you touch any widget value you're running code, most likely accidentally. I'd love to see a workspace or user type option where you can change the default...
Hi.I'm trying to generate XML as output base on my nested dataframe. Everything is ok except by I don't know how to add elements to rootTag.I can add elements from rowtag but not in rootTag. Same problems to add attributes to root <books version = "...
Hi @RobsonNLPT ,Thanks for bringing up your concerns, always happy to help Can you please refer to the below document to read and write the XML files? https://docs.databricks.com/en/query/formats/xml.html Please let me know if this helps and leave a...
Databricks experts, I'm new to Databricks, and encounter an issue with the ANALYZE TABLE command in the Notebook. I created two tables nyc_taxi and nyc_taxi2, from one csv file.When executing the following command in Notebook, analyze table nyc_taxi2...
Hello I have an executable file which i want to host and run from databricks. is this possible in databricks using DBFS ?If NOT what are the other ways to it in databricks ?
Hello, I don't have much information on what kind of executables you would like to run in databricks however, I can think of two solutions : Solution 1: Deploy your code in azure container registry as an image and use the endpoint in data bricks. Sol...
My secret value in Azure key vault is like below.private_key="""-----BEGIN RSA PRIVATE KEY-----********-----END RSA PRIVATE KEY-----"""Running this command in Databricks notebook - jwt.encode(claim_set,private_key,algorithm='RS256')While using the ab...
Thanks much for your troubleshooting methods.Validated the secret scopes, accessing secrets. These looks fine.Key format - I feel problem is with the key format only. As of now I'm awaiting on Azure subscription access. But I printed the secret value...
Hi,I am running autoloader with continuous trigger. How can I stop this trigger during some specific time, only if no data pending and current batch process is complete. How to check how many records pending in queue and current state.Regards,Sanjay
