Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
HiI'm currently looking into connecting to the SQL Warehouse through SDA/SDI. Does anyone have any experience doing so and can share some takeaways on how to implement it. We want to expose the Databricks tables to SAP. We're already doing this by us...
Hey,If you're exploring how to connect your SQL Warehouse to SAP or want to streamline the process of transferring data from SAP HANA into Databricks, our SAP HANA to Databricks Connector could be a valuable tool. This connector allows you to directl...
We have a managed Hive Metastore (HMS) table, and we would like to convert it into an external table and the location of that external hms table as mount points.
Hello.Is it possible to do masking on column data, etc. for the data provided by the cleanroom creatorI am wondering because I don't think Delta shaling allows masking of column data.
Yes, you are correct. Databricks cleanrooms are built on Delta Sharing, which is foundational to how data is securely shared in Databricks. Since Delta Sharing itself does not natively support column-level masking or row-level security, these feature...
Hello,While reviewing the Azure Databricks pricing page to check the cost for Job Compute, I came across a term I hadn’t seen before: "Jobs Light Compute."I suspect this refers to the now end-of-support Databricks Runtime known as Databricks Light:Da...
"Light" is deprecated, and you can't create a new compute with that type. Usually when deprecated products show up on pricing pages, someone is paying for extended support to Microsoft, but sometimes it means they didn't edit that page. You can alw...
I'm running a model using LGBM with Spark, within Pipeline, but when I log it to MlFlow using the log_model feature store function, I can't replicate the pipeline in prediction. When I execute:predict = fs.score_batch(logged_model, df_pred) display(...
Hi Everyone,can someone suggest me to select best native job monitoring tool available in Databricks for fulfill my need;we need to monitor the following:Number of failed jobs and its name : for last 24 hoursTable that are not getting dataLatest inge...
You can use the databricks API to collect all required information.. https://docs.databricks.com/api/workspace/jobs/listLoad the output to a delta table. Use the Databricks dashboards in displaying this data.. schedule the job for loading the databri...
Hello @Cert-Team @Certificate Team,Request Id# 00432042I encountered a pathetic experience while attempting my Databricks Certified Data Engineer Professional certification exam. This is a completely unethical process to harass the examinee and lose...
@Cert-Team I have similar issue. My exam got suspended without even starting. Similar to the snippet in this url https://kryterion.my.site.com/support/s/article/What-To-Do-When-An-Auto-Suspension-Occurs?language=en_US please advise how can I relaunch...
Hello Databricks Community,I am using Terraform to create a Databricks recipient for sharing data with a non-Databricks user. The recipient is set up with authentication_type = "TOKEN", which generates a temporary URL to download a credentials file....
Hi Team,Could you please help me to understand, 1)Why we need to migrate Unity catalog? if we are not migrating what benefits we will not get?2) How to migrate Unity catalog (What all are objects needs to migrate and any tool) ? Regards,Phanindra
@Phani1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platf...
Hello Phanindra,
Inside Databricks ecosystem, you can easily get end-to-end lineage. The data lineage will be build automatically as you ingest and transform data inside the lakehouse. Here is a technical view on how this is achieved:
On the other ha...
Hi Team,After successfully creating a group and populating it with members, I am facing the problem of the is_member() function always returning False, despite the user being part of the group. This issue is specific to my sandbox Databricks Account....
Can you post your code? I use the 'is_account_group_member' in my code as follows.
Also note there is a alternative to Dynamic Views that is preferred. It's named 'Row filtering and Column masking'. Go here to learn more:
https://docs.databricks...
I used below code to populate target table, it is working as expected but expect for surrogatekey column. After I inserted dummy entry -1 and did run merge code, it is generating the numbers in surrogatekey column like this 1,3,5,7,9 odd numbers, it ...
This is expected. Identity columns can have gaps, or per the documentation:
Values assigned by identity columns are unique and increment in the direction of the specified step, and in multiples of the specified step size, but are not guaranteed to b...
I am trying to send Databricks cluster logs to Grafana using an init_script where I define the system journals to consume. The issue I am facing is that I cannot get the driver logs, standard error, and output to reach Grafana. Is there something spe...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
