- 2183 Views
- 4 replies
- 2 kudos
init_script breaks Notebooks
Hi everyoneWe would like to use our private company Python repository for installing Python libraries with pip install.To achieve this, I created a simple script which sets the index-url configuration of pip to our private repoI set this script as a...
- 2183 Views
- 4 replies
- 2 kudos
- 2 kudos
Did you also try cloning the cluster or using other cluster for the testing? The metastore down is normally a Hive Metastore issue, should not be impacting here, but you could check for more details on the error on the log4j under Driver logs.
- 2 kudos
- 28767 Views
- 6 replies
- 1 kudos
Resolved! Performance issue with pyspark udf function calling rest api
Hi All,I am facing some performance issue with one of pyspark udf function that post data to REST API(uses cosmos db backend to store the data).Please find the details below: # The spark dataframe(df) contains near about 30-40k data. # I am using pyt...
- 28767 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Sanjoy Sen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...
- 1 kudos
- 1568 Views
- 1 replies
- 1 kudos
Resolved! Sort after update on DWH
Running query on serverless DWH:UPDATEcatalog.schema.tableSETcol_tmp = CAST(col as DECIMAL(30, 15))In query profiling, it has some sort and shuffle stages in graph.Table has partition by partition_date columnSome details in sort node mentions that so...
- 1568 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @shusharin_anton, The sort and shuffle stages in your query profile are likely triggered by the need to redistribute and order the data based on the partition_date column. This behavior can be attributed to the way Spark handles data partitioning ...
- 1 kudos
- 1137 Views
- 1 replies
- 0 kudos
Mock user doesn't have the required privileges to access catalog `remorph` while running 'make test'
Utility : Remorph (Databricks)Issue : 'User `me@example.com` doesn't have required privileges :: ``to access catalog `remorph`' while running 'make test' cmdI am encountering an issue while running tests for Databricks Labs Remorph using 'make test'...
- 1137 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rai00, Ensure that the mock user me@example.com has the necessary privileges at both the catalog and schema levels. The user needs specific privileges such as USE_SCHEMA and CREATE_VOLUME Use the WorkspaceClient to check the effective privilege...
- 0 kudos
- 4008 Views
- 2 replies
- 2 kudos
Resolved! Trigger Dashboard Update At The End of a Workflow
Heya I have a workflow that computes some data and writes to a delta table, and I have a dashboard that is based on the table. How can I trigger refresh on the dashboard once the workflow is finished? Thanks!
- 4008 Views
- 2 replies
- 2 kudos
- 2 kudos
How does one do this with the new dashboards? I only see the ability to do this with legacy dashoards.
- 2 kudos
- 16304 Views
- 11 replies
- 2 kudos
Why can't I delete experiments without deleting the notebook? Or better Organize experiments into folders?
My Databricks Experiments is cluttered with a whole lot of experiments. Many of them are notebooks which are showing there for some reason (even though they didn't have an MLflow run associated with it). I would like to delete the experiments, but it...
- 16304 Views
- 11 replies
- 2 kudos
- 2 kudos
Hey @Debayan @SparkMaster A bit late here, but I believe this is being caused by a click on the right side experiments icon. This may look like a meaningless click but it actually triggers a run.
- 2 kudos
- 8888 Views
- 1 replies
- 0 kudos
Resolved! Can we modify the constraint of a primary key in an existed table?
Hello Community,Is it possible to modify the schema of an existing table that currently has an ID column without any constraints? I would like to update the schema to make the ID column a primary key with auto-increment starting by the maximum id al...
- 8888 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @jeremy98 Based on some old article it looks it cannot be done:There are a few caveats you should keep in mind when adopting this new feature. Identity columns cannot be added to existing tables; the tables will need to be recreated with the new ...
- 0 kudos
- 9270 Views
- 4 replies
- 0 kudos
Resolved! Can we display key vault secret in Databricks notebook
I am using databricks notebook and Azure key vault.When I am using below function I am getting as output [REDACTED].'dbutils.secrets.get(scope_name,secret_name)' I want to know if there is any way to display the secret in databricks.
- 9270 Views
- 4 replies
- 0 kudos
- 0 kudos
@Shreyash_Gupta You can simply iterate over each letter of the secret and print it.Something like this:for letter in dbutils.secrets.get(scope_name,secret_name): print(letter)
- 0 kudos
- 896 Views
- 1 replies
- 0 kudos
Question about notifications and failed jobs
Hello, I had a question involving rerunning a job from a checkpoint using ‘Repair Run’. I have a job which failed and Im looking to rerun the stream from a checkpoint. My job uses notifications for file detection (cloudFiles.useNotifications). My que...
- 896 Views
- 1 replies
- 0 kudos
- 0 kudos
When rerunning your job from a checkpoint using Repair Run with cloudFiles.useNotifications, only unprocessed messages in the queue (representing new or failed-to-process files) will be consumed. Files or events already recorded in the checkpoint wil...
- 0 kudos
- 2156 Views
- 2 replies
- 1 kudos
Resolved! Any way to ignore DLT tables in pipeline
Hello,In our testing environment we would like to be able to only update the DLT tables we are testing for our pipeline. This would help speed up the testing. We currently have the pipeline code being generated dynamically based on how many tables th...
- 2156 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @eballinger. To address your requirement of updating only specific Delta Live Tables (DLT) in your testing environment without removing the others, you can leverage the @dlt.table decorator and the temporary parameter in your Python code. This app...
- 1 kudos
- 2113 Views
- 4 replies
- 0 kudos
"Databricks Bundle Deploy -t prod" command deletes log of historical runs
I'm using Databricks Asset Bundles with Azure DevOps CI/CD for workflow deployment. While the initial deployment to production works fine, I encounter an issue when updating the workflow in the development environment and redeploying it to production...
- 2113 Views
- 4 replies
- 0 kudos
- 0 kudos
When you re-deploy you job, do you augment the version? (e.g., 4.3.0 -> 4.3.1)I have been through this, when I change a definition in the databricks.yml, for example when changing the bundle name, because it detects as a new workflow.Can you explain ...
- 0 kudos
- 2924 Views
- 4 replies
- 0 kudos
Random failure in the loop in pyspark
Hi,I'm encountering an issue in a pyspark code, where I'm calculating certain information monthly in a loop. The flow is pretty much as:Read input and create/read intermediate parquet files,Upsert records in intermediate parquet files with the monthl...
- 2924 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you show some code to get the gist of what the code does? Are the parquet files accessed as a catalog table? Could it be that some other job makes changes to input tables?
- 0 kudos
- 2459 Views
- 1 replies
- 1 kudos
Resolved! Check for row level security and column masking
Hi All,We have sensitive tables and have applied row level security and column masking. I would like to build into our job a check to make sure these tables still have the row filters and column masks applied. This would help ensure these security fi...
- 2459 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @eballinger. Have you tried using DESCRIBE TABLE EXTENDED on the table, that will give you details about filters applied to a table.
- 1 kudos
- 3626 Views
- 2 replies
- 0 kudos
Integrating Azure Log Analytics with Delta Live Tables Pipelines and Job Clusters
Hi,I'm setting up a Delta Live Tables (DLT) pipeline for my medallion architecture. I’m interested in tracking, ingesting, and analyzing the log files in Azure Log Analytics. However, I haven’t found much information on how to configure this setup.Sp...
- 3626 Views
- 2 replies
- 0 kudos
- 0 kudos
"message": " File <command-68719476741>, line 10\n log_analytics_pkey = dbutils.secrets.get(scope=\"ScopeLogAnalyticsPKey\", key=\"LogAnalyticsPKey\")\n ^\nSyntaxError: invalid syntax\n", "error_class": "_UNCLASSIFIED_PYTHON_COMMAND_ERROR" It seems ...
- 0 kudos
- 19582 Views
- 4 replies
- 2 kudos
how to check table size by partition?
I want to check the size of the delta table by partition.As you can see, only the size of the table can be checked, but not by partition.
- 19582 Views
- 4 replies
- 2 kudos
- 2 kudos
The previous two answers did not work for me (DBX 15.4).I found a hacky way using the delta log: find latest (group of) checkpoint (parquet) file(s) in delta log and use it as source prefix `000000000000xxxxxxx.checkpoint`:SELECT partition_column_1,...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
