- 825 Views
- 1 replies
- 0 kudos
Mock user doesn't have the required privileges to access catalog `remorph` while running 'make test'
Utility : Remorph (Databricks)Issue : 'User `me@example.com` doesn't have required privileges :: ``to access catalog `remorph`' while running 'make test' cmdI am encountering an issue while running tests for Databricks Labs Remorph using 'make test'...
- 825 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rai00, Ensure that the mock user me@example.com has the necessary privileges at both the catalog and schema levels. The user needs specific privileges such as USE_SCHEMA and CREATE_VOLUME Use the WorkspaceClient to check the effective privilege...
- 0 kudos
- 3438 Views
- 2 replies
- 2 kudos
Resolved! Trigger Dashboard Update At The End of a Workflow
Heya I have a workflow that computes some data and writes to a delta table, and I have a dashboard that is based on the table. How can I trigger refresh on the dashboard once the workflow is finished? Thanks!
- 3438 Views
- 2 replies
- 2 kudos
- 2 kudos
How does one do this with the new dashboards? I only see the ability to do this with legacy dashoards.
- 2 kudos
- 13891 Views
- 11 replies
- 2 kudos
Why can't I delete experiments without deleting the notebook? Or better Organize experiments into folders?
My Databricks Experiments is cluttered with a whole lot of experiments. Many of them are notebooks which are showing there for some reason (even though they didn't have an MLflow run associated with it). I would like to delete the experiments, but it...
- 13891 Views
- 11 replies
- 2 kudos
- 2 kudos
Hey @Debayan @SparkMaster A bit late here, but I believe this is being caused by a click on the right side experiments icon. This may look like a meaningless click but it actually triggers a run.
- 2 kudos
- 8244 Views
- 1 replies
- 0 kudos
Resolved! Can we modify the constraint of a primary key in an existed table?
Hello Community,Is it possible to modify the schema of an existing table that currently has an ID column without any constraints? I would like to update the schema to make the ID column a primary key with auto-increment starting by the maximum id al...
- 8244 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @jeremy98 Based on some old article it looks it cannot be done:There are a few caveats you should keep in mind when adopting this new feature. Identity columns cannot be added to existing tables; the tables will need to be recreated with the new ...
- 0 kudos
- 6045 Views
- 4 replies
- 0 kudos
Resolved! Can we display key vault secret in Databricks notebook
I am using databricks notebook and Azure key vault.When I am using below function I am getting as output [REDACTED].'dbutils.secrets.get(scope_name,secret_name)' I want to know if there is any way to display the secret in databricks.
- 6045 Views
- 4 replies
- 0 kudos
- 0 kudos
@Shreyash_Gupta You can simply iterate over each letter of the secret and print it.Something like this:for letter in dbutils.secrets.get(scope_name,secret_name): print(letter)
- 0 kudos
- 6923 Views
- 5 replies
- 1 kudos
Resolved! I haven't received badge for completion
Hi,Today I completed the test for Lakehouse fundamentals by scored 85%, still I haven't received the badge through my email francis@intellectyx.comKindly let me know please !-Francis
- 6923 Views
- 5 replies
- 1 kudos
- 1 kudos
HI I completed the test for Databricks Certified Data Engineer Associate on 17 December 2024. still I haven't received the badge through my email sureshrocks.1984@hotmail.comKindly let me know please !SURESHK
- 1 kudos
- 564 Views
- 1 replies
- 0 kudos
Question about notifications and failed jobs
Hello, I had a question involving rerunning a job from a checkpoint using ‘Repair Run’. I have a job which failed and Im looking to rerun the stream from a checkpoint. My job uses notifications for file detection (cloudFiles.useNotifications). My que...
- 564 Views
- 1 replies
- 0 kudos
- 0 kudos
When rerunning your job from a checkpoint using Repair Run with cloudFiles.useNotifications, only unprocessed messages in the queue (representing new or failed-to-process files) will be consumed. Files or events already recorded in the checkpoint wil...
- 0 kudos
- 1362 Views
- 2 replies
- 1 kudos
Resolved! Any way to ignore DLT tables in pipeline
Hello,In our testing environment we would like to be able to only update the DLT tables we are testing for our pipeline. This would help speed up the testing. We currently have the pipeline code being generated dynamically based on how many tables th...
- 1362 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @eballinger. To address your requirement of updating only specific Delta Live Tables (DLT) in your testing environment without removing the others, you can leverage the @dlt.table decorator and the temporary parameter in your Python code. This app...
- 1 kudos
- 1377 Views
- 4 replies
- 0 kudos
"Databricks Bundle Deploy -t prod" command deletes log of historical runs
I'm using Databricks Asset Bundles with Azure DevOps CI/CD for workflow deployment. While the initial deployment to production works fine, I encounter an issue when updating the workflow in the development environment and redeploying it to production...
- 1377 Views
- 4 replies
- 0 kudos
- 0 kudos
When you re-deploy you job, do you augment the version? (e.g., 4.3.0 -> 4.3.1)I have been through this, when I change a definition in the databricks.yml, for example when changing the bundle name, because it detects as a new workflow.Can you explain ...
- 0 kudos
- 1783 Views
- 4 replies
- 0 kudos
Random failure in the loop in pyspark
Hi,I'm encountering an issue in a pyspark code, where I'm calculating certain information monthly in a loop. The flow is pretty much as:Read input and create/read intermediate parquet files,Upsert records in intermediate parquet files with the monthl...
- 1783 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you show some code to get the gist of what the code does? Are the parquet files accessed as a catalog table? Could it be that some other job makes changes to input tables?
- 0 kudos
- 1468 Views
- 1 replies
- 1 kudos
Resolved! Check for row level security and column masking
Hi All,We have sensitive tables and have applied row level security and column masking. I would like to build into our job a check to make sure these tables still have the row filters and column masks applied. This would help ensure these security fi...
- 1468 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @eballinger. Have you tried using DESCRIBE TABLE EXTENDED on the table, that will give you details about filters applied to a table.
- 1 kudos
- 2489 Views
- 2 replies
- 0 kudos
Integrating Azure Log Analytics with Delta Live Tables Pipelines and Job Clusters
Hi,I'm setting up a Delta Live Tables (DLT) pipeline for my medallion architecture. I’m interested in tracking, ingesting, and analyzing the log files in Azure Log Analytics. However, I haven’t found much information on how to configure this setup.Sp...
- 2489 Views
- 2 replies
- 0 kudos
- 0 kudos
"message": " File <command-68719476741>, line 10\n log_analytics_pkey = dbutils.secrets.get(scope=\"ScopeLogAnalyticsPKey\", key=\"LogAnalyticsPKey\")\n ^\nSyntaxError: invalid syntax\n", "error_class": "_UNCLASSIFIED_PYTHON_COMMAND_ERROR" It seems ...
- 0 kudos
- 18153 Views
- 4 replies
- 2 kudos
how to check table size by partition?
I want to check the size of the delta table by partition.As you can see, only the size of the table can be checked, but not by partition.
- 18153 Views
- 4 replies
- 2 kudos
- 2 kudos
The previous two answers did not work for me (DBX 15.4).I found a hacky way using the delta log: find latest (group of) checkpoint (parquet) file(s) in delta log and use it as source prefix `000000000000xxxxxxx.checkpoint`:SELECT partition_column_1,...
- 2 kudos
- 2708 Views
- 7 replies
- 0 kudos
Enable predictive optimization for my account
I want to enable predictive optimization for my account for but i can not see the option as mentioned in documentation Access the accounts console.Navigate to Settings, then Feature enablement.Select Enabled next to Predictive optimization.I am metas...
- 2708 Views
- 7 replies
- 0 kudos
- 0 kudos
Got it, after reviewing your workspace I can notice that it is GCP workspace located in europe-west3, as of now Predictive Optimization is not supported on GCP side, there are few regions that are set to be enabled by end of Q4 but unfortunately the ...
- 0 kudos
- 907 Views
- 1 replies
- 0 kudos
Pyspark and SQL Warehouse
If i write pyspark code and i need to get data in powerbi will it be possible to merge data into one semantic model? For instance the pipeline were developed using SQL so its directly compatible with SQL Warehouse
- 907 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, it is possible to merge data into one semantic model in Power BI when using PySpark code to get data. Databricks supports integration with Power BI, allowing you to create a unified semantic model. You can develop your data pipeline using PySpar...
- 0 kudos
| User | Count |
|---|---|
| 1636 | |
| 791 | |
| 517 | |
| 349 | |
| 287 |
