Is it possible somehow to restrict creation of the serving endpoints to specific users? I want to grant Workspace access under the Entitlements of the specific group, but I want not to allow users of this group create serving endpoints.The only way I...
Hi everyone,I regularly (once a week) run the analyze table compute statistics command on all my tables in Databricks to keep statistics up to date for query optimization.In the Spark table UI catalog, I can see some statistics metadata like spark.st...
Hello @Sainath368! sql.statistics.createdAt reflects the epoch time when statistics were created. Unfortunately, there's no direct command available to check when the statistics were last updated. As a workaround, you can manually set the current tim...
Hi, In a python file using dbutils.notebook.run() I'm running specific notebook.The notebook is failling but i'm getting a general error log instead the real specific log.When I'm running the notebook directly - I'm getting the specific error log.gen...
@Vinay_M_RBTW, when trying to run a job using Databricks API, I encounter the same issue (general "FAILED: Workload failed"):from databricks.sdk import WorkspaceClient client = WorkspaceClient() run = client.jobs.run_now(job_id) error message:state_...
We have created databricks jobs and each has multiple tasks. Each task is 24/7 running streaming with checkpoint enabled. We want it to be stateful when cancel and run the job but it seems like, when we cancel the job run it kill the parent process a...
If the “reporting” layer is essentially micro-batching over bounded backlogs, run it with availableNow (or a scheduled batch job) so each run is naturally bounded and exits cleanly on its own, no manual cancel. This greatly reduces chances of partial...
My Databricks job is completing successful but my data is not written into the target table, source path is correct, each n every thing is correct, but I am not sure y data is not written into the delta table.
hi @Srajole ,There are a bunch of possibilities as to why the data is not being written into the table -You’re writing to a path different from the table’s storage location, or using a write mode that doesn’t replace data as expected.spark.sql("DESCR...
Hello there!I am writing this open message to know how you guys are using the computes in your work cases.Currently, in my company, we have multiple compute instances that can be differentiated into two main types:Clusters with a large instance for b...
Hello @juanjomendez96 ,to my best knowledge and experience autoscaled shared cluster (using smaller instances) works good for most 2nd-case scenario (clusters for ad-hoc/development team usage).This approach allows You to reuse the resources across t...
I'm trying to use the SAP JDBC driver to read data in my Spark application which I deploy via asset bundles with job computes.I was able to install the SAP JDBC Driver on a general purpose cluster by adding the jar (com.sap.cloud.db.jdbc:ngdbc:2.25.9...
Hi @VicS ,To add a Maven package to a job task definition , in libraries, specify a maven mapping for each Maven package to be installed. For each mapping, specify the following: resources: jobs: my_job: # ... tasks: - task_...
Python 3.10.12 I am trying to get these filter results: example:If "column1" = '2024' and column2 in ('DE','SC') then 'value1" else 'value2'If "column1" = "2023" and column2 in ('DE,'SC')then "value3 else "value4"if the row/record does not fit the cr...
Hi @abueno .I'm assuming you're asking how to do this in pyspark. You can use when and otherwise conditional functions to achieve your expected result:from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringT...
I am trying to setup a delta sharing Recipient using OIDC Federation with the Issuer URL being cognito idp endpoint.Are there any examples, other than EntraID, for the values of Subject Claim/Subject/Audiences in the OIDC Policy for Cognito or Google...
We managed to figure how to make machine to machine authentication to work.when you setup cognito pool for m2m scenario you add App Clientand then set App Client as both `sub` and Audience in databricks recepient OIDC Policy:2. Set `aud` claim to the...
Can we connect Salesforce with Databricks.I want to do both push and pull activity using Databricks and Salesforce.Do There any challenge while using ODBC?Please help me with detailed approach.Thanks a ton.
Have resolved it Step 1: Create a Service PrincipalLog in to your Databricks Workspace and navigate to the Admin Settings page by clicking your email in the bottom-left corner and selecting "Admin Settings".Go to the Identity and access tab and click...
Hi there,I'd appreciate some assistance with troubleshooting what is supposed to be a (somewhat) simple use of autoloader. Below are some screenshots highlighting my issue:When I attempt to create the dataframe via spark.readStream.format("cloudFiles...
Hi @ChristianRRL This is a common issue with Spark Structured Streaming and the display() function.The error occurs because you're trying to display a streaming DataFrame, which requires special handling. Here are several solutions:1. Use writeStrea...
Has anyone successfully connected to a DB2 database on ZOS from a Databricks cluster using a JDBC connection?I also need to specify an SSL certificate path and not sure if I need to use an init script on the cluster to do so.Any examples would be ver...
Hi @Karl , Greetings! I've outlined the steps below to connect from Databricks to IBM DB2 using JDBC:Step 1: Obtain the DB2 JDBC Driver Visit the IBM website to download the appropriate JDBC driver for DB2 on z/OS.Reference Document: IBM DB2 JDBC Dr...
we are on a trial databricks premium workspace (fully managed by databricks)we are trying to test Delta Sharing feature, where we are sharing UC table with a recipient using python client (outside of databricks). We are using `delta-sharing` python l...
Yes - you are right, works on trial - i was able to setup S3 as an external location and configured the UC schema to use this S3 ext location, and then was able to query data from an external python client.
Hi, I have datbricks on AWS, I created some secrets in AWS Secrets Manger, I would need to create the scopes based on AWS secrets manager.When I use Azure's Key Vault, when creating the scope, it uses the option -scope-backend-type AZURE_KEYVAULT, bu...
Hi @thiagoawstest Step 1: Create Secret ScopeYou can create a secret scope using the Databricks REST API as shown below: python import requests import json # Define the endpoint and headers url = "https://<databricks-instance>/api/2.0/secrets/scope...
Hi There,I have a table of 160 GB with partition applied on country and yearmonth columns, I maintain a previous history of 6 years and replace the partitions (latest 2 months) to add the new data.I use overwrite mode to replace the effected partitio...
Hi @SusmithaBadam, based on your use case, partitioned tables are performing better because they work kind of like labeled folders. When you group by, it can quickly go to the exact folder instead of scanning everything, so it’s much faster.Liquid cl...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
