- 1884 Views
- 3 replies
- 0 kudos
Databricks CI/CD
Hi Team,How can we implement CI/CD in Databricks and its best practices, and tools to consider?Regards,Janga
- 1884 Views
- 3 replies
- 0 kudos
- 2580 Views
- 2 replies
- 1 kudos
What is the quota limit for using create user token api?
Hi Community, I was going through this doc: https://docs.databricks.com/api/workspace/tokens/create to and got to know, that there is a quota limit to how many token one can generate using the api: POST /api/2.0/token/create, having breached the thre...
- 2580 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello@Surajv, Q1: What is and how to find out the quota limit? The quota limit for creating user tokens via the API (POST /api/2.0/token/create) is essential to manage token usage. Each user can have multiple personal access tokens in a Databricks wo...
- 1 kudos
- 2094 Views
- 2 replies
- 0 kudos
Resolved! Can I give different git branches in the same repo for different tasks in a data bricks workflow
I have 2 tasks (T1 &T2) that run in branch B1 of Repo1.I have created a new task (depends on T2 ) which points to a different branch B2 of same Repo1.Is it possible to run them in the same workflow pipeline? When I tried to set this up, databricks c...
- 2094 Views
- 2 replies
- 0 kudos
- 0 kudos
I was to able to find a workaround. Created separate jobs for those that need to be in different branch (testing tasks) and then ran all of them from a new job.
- 0 kudos
- 1850 Views
- 1 replies
- 0 kudos
Resolved! Setting a preset list of values in a task parameter in databricks job
I want to be able to have a user select from a preset list of values for a task parameter when they kick off a job with the "Run now with different parameters" option. In a notebook I am able to use dbutils.widgets.dropdown() to set the list of value...
- 1850 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately providing a job params dropdown list is not currently available, you can alway do a Run with different params, but the user will have to change them manually and not with a predefined list.
- 0 kudos
- 800 Views
- 1 replies
- 0 kudos
New icon for SQL Editor looks like a broken image
Hey - I may be showing my age here, but I felt compelled to point out that at a glance, the new icon for a SQL Editor tab in the Databricks UI looks an awful lot like a broken image link icon, from the days of Internet Explorer. This, subconsciously,...
- 800 Views
- 1 replies
- 0 kudos
- 0 kudos
Is this still showing broking image? Is this only happening in Explorer, if you try Chrome for example does it work?Can you share an screenshot of your workspace to better understand how it shows?
- 0 kudos
- 908 Views
- 1 replies
- 0 kudos
Databricks Initial Costs AWS
I have a new premium account. I set up a cost dashboard (see attached) after I create a new workspace using AWS Quickstart, where I see some costs. Why do I have this If I am not using Databricks at all? How can I save the costs?
- 908 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you seeing this data from the Usage tab in the Account console? Does it allow you to filter it by SKU?
- 0 kudos
- 1736 Views
- 2 replies
- 1 kudos
Resolved! Internal Error with MERGE Command in Spark SQL
I'm trying to perform a MERGE between two tables (customers and customers_update) using Spark SQL, but I’m encountering an internal error during the planning phase. The error message suggests it might be a bug in Spark or one of the plugins in use.He...
- 1736 Views
- 2 replies
- 1 kudos
- 1 kudos
The issue you encountered with the MERGE statement in Spark SQL, which was resolved by specifying the database and metastore, is likely related to how Spark handles table references during the planning phase. The internal error you faced suggests a b...
- 1 kudos
- 997 Views
- 1 replies
- 0 kudos
Ingress/Egress private endpoint
Hello , We have configured our Databricks environment with private endpoint connections injected into our VNET, which includes two subnets (public and private). We have disabled public IPs and are using Network Security Groups (NSGs) on the subnet, a...
- 997 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Fkebbati ,There always be some costs related to data transfer between those account. Let's have a look at private link pricing page. So it's expected, but MS likes to hide this kind of information
- 0 kudos
- 13788 Views
- 4 replies
- 2 kudos
Masking techniques for more PII columns
Hi Databricks Team,We would appreciate it if you could inform us about the situations when Column-Masking, Row-Level Filtering, and Attributed-Based Masking should be utilized, as well as the recommended technique for handling large data volumes cont...
- 13788 Views
- 4 replies
- 2 kudos
- 2 kudos
Agree with @Meghla-C , https://databricks.aha.io/ideas/ideas/DB-I-7941 this was the feature request and if you see the status, it is in preview.
- 2 kudos
- 769 Views
- 1 replies
- 1 kudos
Best way to find Databricks Certified Professionals?
Hi all! We have a few Databricks certified folks on our team, but we are looking for more! What is the best way to find a list of or know who is certified? We are looking for North America / Europe / LATAM / South America. I am part of the Microsoft ...
- 769 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @ladyleet ,There is the community of Databricks certified: Databricks Certified Credential Holder Directory.
- 1 kudos
- 684 Views
- 0 replies
- 0 kudos
Has Anyone Used Databricks for Financial Auditing and Compliance?
I’m exploring how Databricks can support financial auditing and compliance, especially with platforms like aauditing. Has anyone here used Databricks for data analysis or reporting in the context of auditing services? Any insights on workflows or bes...
- 684 Views
- 0 replies
- 0 kudos
- 1264 Views
- 1 replies
- 0 kudos
Amazon MSK integration with Databricks
Hello Everyone,I am a beginner in the world of Data bricks. I am trying to achieve a use case which involves : consuming messages from Amazon MSK and creating a Delta table in Data bricks.I need to get some insights on what all accesses are supposed ...
- 1264 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Amrit23 , Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream. https://docs.databricks.com/en/connect/streaming/kafka.html And for the Access Permissions: Amazon MSK: Ensure you have the nec...
- 0 kudos
- 1563 Views
- 1 replies
- 0 kudos
Resolved! Databricks SQL as alternative to Spark thrift server
We are currently using Spark as our SQL engine with Thrift Server but are evaluating Databricks Serverless SQL as a potential alternative. We have a few specific questions:Does Databricks Serverless SQL support custom Spark extensions?Can we configur...
- 1563 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SachinJanani , 1 - Databricks Serverless SQL does not support custom Spark extensions (Advanced Spark Configs/Libraries etc). This is because the serverless environment is designed to be highly optimized and managed by Databricks, which limits th...
- 0 kudos
- 4479 Views
- 4 replies
- 0 kudos
GDAL on Databricks Cluster Runtime 12.2 LTS
I need gdal in my course work.After reading this post, I used init script as follows to install gdal into runtime 12.2 LTS dbutils.fs.put("/databricks/scripts/gdal_install.sh",""" #!/bin/bash sudo add-apt-repository ppa:ubuntugis/ppa sudo apt-get up...
- 4479 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, in case anyone is still struggling here. I found I could not get the init script approach to work, but if I just run a shell command to install gdal at the start of my notebook it works fine. You might note, however, that this installs gdal versi...
- 0 kudos
- 1758 Views
- 1 replies
- 0 kudos
Problems with Big Data Solutions and Databricks- Any advice?
Hello everyone,I am currently facing several challenges related to big data solutions, particularly with the Databricks. As many of you may know, Databricks is a powerful platform for data engineering and analytics, but I have encountered some signif...
- 1758 Views
- 1 replies
- 0 kudos
- 0 kudos
Your problem statement is too generic. If your company is facing this, you can reach out to your SA; they will help you. If it's a personal project, then mention what you are trying in detail with cluster size, what you are trying to integrate with, ...
- 0 kudos
