- 577 Views
- 3 replies
- 0 kudos
Databricks Lake House Data Clean Room Utilization
Hello.Is it possible to do masking on column data, etc. for the data provided by the cleanroom creatorI am wondering because I don't think Delta shaling allows masking of column data.
- 577 Views
- 3 replies
- 0 kudos
- 0 kudos
Yes, you are correct. Databricks cleanrooms are built on Delta Sharing, which is foundational to how data is securely shared in Databricks. Since Delta Sharing itself does not natively support column-level masking or row-level security, these feature...
- 0 kudos
- 543 Views
- 1 replies
- 0 kudos
About "Jobs Light Compute" Listed in the Azure Databricks Pricing Table
Hello,While reviewing the Azure Databricks pricing page to check the cost for Job Compute, I came across a term I hadn’t seen before: "Jobs Light Compute."I suspect this refers to the now end-of-support Databricks Runtime known as Databricks Light:Da...
- 543 Views
- 1 replies
- 0 kudos
- 0 kudos
"Light" is deprecated, and you can't create a new compute with that type. Usually when deprecated products show up on pricing pages, someone is paying for extended support to Microsoft, but sometimes it means they didn't edit that page. You can alw...
- 0 kudos
- 290 Views
- 0 replies
- 0 kudos
Problem when use synapse ml lightgbm with feature store
I'm running a model using LGBM with Spark, within Pipeline, but when I log it to MlFlow using the log_model feature store function, I can't replicate the pipeline in prediction. When I execute:predict = fs.score_batch(logged_model, df_pred) display(...
- 290 Views
- 0 replies
- 0 kudos
- 13357 Views
- 5 replies
- 3 kudos
Resolved! Databricks Monitoring
Hi Everyone,can someone suggest me to select best native job monitoring tool available in Databricks for fulfill my need;we need to monitor the following:Number of failed jobs and its name : for last 24 hoursTable that are not getting dataLatest inge...
- 13357 Views
- 5 replies
- 3 kudos
- 3 kudos
You can use the databricks API to collect all required information.. https://docs.databricks.com/api/workspace/jobs/listLoad the output to a delta table. Use the Databricks dashboards in displaying this data.. schedule the job for loading the databri...
- 3 kudos
- 2746 Views
- 5 replies
- 1 kudos
Resolved! DataBricks Certification Exam Got Suspended. Require Immediate support
Hello @Cert-Team @Certificate Team,Request Id# 00432042I encountered a pathetic experience while attempting my Databricks Certified Data Engineer Professional certification exam. This is a completely unethical process to harass the examinee and lose...
- 2746 Views
- 5 replies
- 1 kudos
- 1 kudos
@Cert-Team I have similar issue. My exam got suspended without even starting. Similar to the snippet in this url https://kryterion.my.site.com/support/s/article/What-To-Do-When-An-Auto-Suspension-Occurs?language=en_US please advise how can I relaunch...
- 1 kudos
- 8339 Views
- 8 replies
- 3 kudos
403 FORBIDDEN You don't have permission to access this page 2023-07-02 08:03:23 | Error 403 | https:
I am getting above message when trying to access the quiz for Generative AI Fundamentals.
- 8339 Views
- 8 replies
- 3 kudos
- 488 Views
- 3 replies
- 1 kudos
How to Set Custom Expiration Time for TOKEN Authentication in Databricks Recipient using Terraform?
Hello Databricks Community,I am using Terraform to create a Databricks recipient for sharing data with a non-Databricks user. The recipient is set up with authentication_type = "TOKEN", which generates a temporary URL to download a credentials file....
- 488 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Sudheer2, Could you please try enabling debug logs? TF_LOG=DEBUG DATABRICKS_DEBUG_TRUNCATE_BYTES=250000 terraform apply -no-color 2>&1 |tee tf-debug.log
- 1 kudos
- 681 Views
- 1 replies
- 0 kudos
Migration to new Tools
Has anyone else taken a migration journey from SAS to a different language/platform?? Love to hear about your journey…
- 681 Views
- 1 replies
- 0 kudos
- 0 kudos
@cannonball111 Yes, we have done using Travinto Technologies tool
- 0 kudos
- 2406 Views
- 1 replies
- 0 kudos
Unity catalog Migration
Hi Team,Could you please help me to understand, 1)Why we need to migrate Unity catalog? if we are not migrating what benefits we will not get?2) How to migrate Unity catalog (What all are objects needs to migrate and any tool) ? Regards,Phanindra
- 2406 Views
- 1 replies
- 0 kudos
- 0 kudos
@Phani1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platf...
- 0 kudos
- 1028 Views
- 1 replies
- 1 kudos
end-to-end lineage using Unity Catalog
Hi Team,what is the Possibilities of getting end-to-end lineage using Unity CatalogRegards,Phanindra
- 1028 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello Phanindra, Inside Databricks ecosystem, you can easily get end-to-end lineage. The data lineage will be build automatically as you ingest and transform data inside the lakehouse. Here is a technical view on how this is achieved: On the other ha...
- 1 kudos
- 1226 Views
- 1 replies
- 0 kudos
Data Masking issue
Hi Team,After successfully creating a group and populating it with members, I am facing the problem of the is_member() function always returning False, despite the user being part of the group. This issue is specific to my sandbox Databricks Account....
- 1226 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you post your code? I use the 'is_account_group_member' in my code as follows. Also note there is a alternative to Dynamic Views that is preferred. It's named 'Row filtering and Column masking'. Go here to learn more: https://docs.databricks...
- 0 kudos
- 1409 Views
- 1 replies
- 0 kudos
cross join issue generating surrogate keys in delta table
I used below code to populate target table, it is working as expected but expect for surrogatekey column. After I inserted dummy entry -1 and did run merge code, it is generating the numbers in surrogatekey column like this 1,3,5,7,9 odd numbers, it ...
- 1409 Views
- 1 replies
- 0 kudos
- 0 kudos
This is expected. Identity columns can have gaps, or per the documentation: Values assigned by identity columns are unique and increment in the direction of the specified step, and in multiples of the specified step size, but are not guaranteed to b...
- 0 kudos
- 483 Views
- 4 replies
- 2 kudos
Expose Databricks Cluster Logs to Grafana
I am trying to send Databricks cluster logs to Grafana using an init_script where I define the system journals to consume. The issue I am facing is that I cannot get the driver logs, standard error, and output to reach Grafana. Is there something spe...
- 483 Views
- 4 replies
- 2 kudos
- 4654 Views
- 3 replies
- 1 kudos
Spatial Queries
Hi,I'm trying to execute the following code:%sqlSELECT LSOA21CD, ST_X(ST_GeomFromWKB(Geom_Varbinary)) AS STX, ST_Y(ST_GeomFromWKB(Geom_Varbinary)) AS STYFROM ordnance_survey_lsoas_december_2021_population_weighted_centroidsWHERE LSOA21CD ...
- 4654 Views
- 3 replies
- 1 kudos
- 1 kudos
we also need this function without the private preview, any solutions?
- 1 kudos
- 891 Views
- 5 replies
- 2 kudos
Issue with Non-Bulk Inserts Using JDBC on Databricks Runtime 14.1
Hello team,I am experiencing an issue with insert operations on Databricks using the JDBC driver. In my SAS Viya system, the DatabricksJDBC42.jar driver version 2.6.40 is configured. I’ve noticed that, up to Databricks Runtime version 13.1, insert op...
- 891 Views
- 5 replies
- 2 kudos
- 2 kudos
Dear @Walter_C , After an initial test with a moderate number of rows, we believed it to be the optimal solution.However, upon increasing the row count beyond a million, we realized the dataset is divided into chunks of unclear size. Sometimes chunks...
- 2 kudos
