- 10107 Views
- 3 replies
- 1 kudos
stop autoloader with continuous trigger programatically
Hi,I am running autoloader with continuous trigger. How can I stop this trigger during some specific time, only if no data pending and current batch process is complete. How to check how many records pending in queue and current state.Regards,Sanjay
- 10107 Views
- 3 replies
- 1 kudos
- 25282 Views
- 2 replies
- 1 kudos
Resolved! Deny assignment modification to allow attach/detach of disks in azure databricks
Our application does storage autoscaling on Azure. We would like to deploy our solution with Azure databricks. But even though the service principal associated with our application has the necessary roles and permissions to attach/detach a disk from ...
- 25282 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your replyIs there any way databricks provides to bypass the deny assignment for specific apps? I noticed in the deny assignment unity-catalog-access-connector has been provided exlusion under the excludePrincipals section. is there a w...
- 1 kudos
- 1568 Views
- 1 replies
- 0 kudos
Create persistent Scala UDF
I've created a UDF in Scala (using Databricks notebook) which checks if the value of a field in a table is greater than 0, then sum those values. UDF is working as expected. The problem is, that this UDF will be used by multiple SQLs across different...
- 1568 Views
- 1 replies
- 0 kudos
- 0 kudos
In the child notebook:def my_udf(x): return x + 1 spark.udf.register("my_udf", my_udf) dbutils.notebook.exit(my_udf)In the master notebook:child_udf = dbutils.notebook.run("PathToChildnotebook", timeout_seconds=600) spark.udf.register("my_udf",...
- 0 kudos
- 3867 Views
- 3 replies
- 1 kudos
[UNBOUND_SQL_PARAMETER] When running Placeholder query
I am using the databricks-sql-go library version 1.5.2. I am trying to run a query with placeholders of type '?'The query looks like params, args := databricksParams(values)sql := fmt.Sprintf(`SELECT COUNT(*) FROM %s.%s WHERE %s IN (%s)`, schema, tab...
- 3867 Views
- 3 replies
- 1 kudos
- 1 kudos
Can you print out an example after the Sprintf substitutions?It seems you generated a query with a named parameter: ":_58" But args (which should be a Map) does not have a key named "_58".
- 1 kudos
- 2400 Views
- 1 replies
- 0 kudos
DLT: Only STREAMING tables can have multiple queries.
I am trying to to do a one-time back-fill on a DLT table following the example here: dlt.table() def test(): # providing a starting version return (spark.readStream.format("delta") .option("readChangeFeed", "true") .option("...
- 2400 Views
- 1 replies
- 0 kudos
- 0 kudos
I should also add that when I drop the `backfill` function, validation happens successfully and we get the following pipeline DAG:
- 0 kudos
- 13366 Views
- 1 replies
- 1 kudos
Introducing AI Model Sharing with Databricks!
Today, we're excited to announce that AI model sharing is available in both Databricks Delta Sharing and on the Databricks Marketplace. With Delta Sharing you can now easily share and serve AI models securely within your organization or externally ac...
- 13366 Views
- 1 replies
- 1 kudos
- 1 kudos
I'm eager to dive in and leverage these new features to elevate my AI game with Databricks.This is Johnson from KBS Technologies.Thanks for your update.
- 1 kudos
- 3826 Views
- 3 replies
- 0 kudos
Cannot create a repo because the parent path does not exist
I tried following this docWork With Large Monorepos With Sparse Checkout Support in Databricks Repos | Databricks BlogWhen I hook it up to my repos using Azure DevOps Services and check mark Sparse checkout mode, I pass in a subdirectory in my Cone p...
- 3826 Views
- 3 replies
- 0 kudos
- 11094 Views
- 2 replies
- 0 kudos
Resolved! Show Existing Header From CSV I External Table
Hello, is there a way to load csv data into an external table without the _c0, _c1 columns showing?
- 11094 Views
- 2 replies
- 0 kudos
- 4127 Views
- 3 replies
- 0 kudos
Resolved! Unable to load csv data with correct header values in External tables
Hello, is there a way to load "CSV" data into an external table without the _c0, _c1 columns showing?I've tried using the options within the sql statement that does not appear to work.Which results in this table
- 4127 Views
- 3 replies
- 0 kudos
- 0 kudos
you need set "USING data_source"https://community.databricks.com/t5/data-engineering/create-external-table-using-multiple-paths-locations/td-p/44042
- 0 kudos
- 1971 Views
- 1 replies
- 0 kudos
What is the REST API payload for "not sending alert when alert is back to normal"?
Greeting,I am using REST API to create & schedule sql alert by default these alerts will send notifications when they are back to normal which I don't want, In the UI I have the option to uncheck the box (shown in the picture) but I can't find any do...
- 1971 Views
- 1 replies
- 0 kudos
- 0 kudos
@Lambda Did you refer this documentation: https://docs.databricks.com/api/workspace/alerts/create
- 0 kudos
- 1965 Views
- 1 replies
- 0 kudos
Spark VCore for Databrick vCPU
Hi Team,What is the equivalent Spark VCore for Databrick vCPU? , like for below example for DS3 v2 , vCPU=4 and RAM = 14.00 GiB, would like to know equivalent Spark VCore for DS3 v2 as in Azure Databricks Pricing | Microsoft AzureRegards,Phanindra
- 1965 Views
- 1 replies
- 0 kudos
- 0 kudos
@Phani1 In Databricks, the vCPU count is equivalent to the number of Spark vCores. This means that if the DS3 v2 instance has 4 vCPU, it would also have 4 Spark vCores. Please note that the Spark VCore count is based on the vCPU count of the underlyi...
- 0 kudos
- 1508 Views
- 1 replies
- 0 kudos
Can't enter 'edit mode' using shortcut
Hi,With the recently added AI Assistant to databricks notebooks, I'm having issues entering the 'edit mode' of a notebook cell. Previously, I could simply press the 'Enter' key in order to do this but this no longer works.Is anyone else having the sa...
- 1508 Views
- 1 replies
- 0 kudos
- 0 kudos
@AtomicBoy99, it seems like the fix is in place. Can you confirm if it is working well now?
- 0 kudos
- 4687 Views
- 2 replies
- 0 kudos
Python Logging cant save log in DBFS
Hi! I am trying to integrate logging into my project. Got the library and logs to work but cant log the file into DBFS directly.Have any of you been able to save and append the log file directly to dbfs? From what i came across online the best way to...
- 4687 Views
- 2 replies
- 0 kudos
- 0 kudos
you can use azure_storage_loggingSet Python Logging to Azure Blob, but Can not Find Log File there - Stack Overflow
- 0 kudos
- 6687 Views
- 2 replies
- 1 kudos
Databricks notebook how to stop truncating numbers when export the query result to csv
I use Databricks notebook to query databases and export / download result to csv. I just accidentally close a pop-up window asking if need to truncate the numbers, I accidentally chose yes and don't ask again. Now all my long digit numbers are trunca...
- 6687 Views
- 2 replies
- 1 kudos
- 1 kudos
@Jasonh202222 - Kindly check the below navigation path user settings -> Account settings -> Display -> Download and Export. Under Download and Export, Enable the checkbox - "Prompt for formatting large numbers when downloading or exporting" and cl...
- 1 kudos
- 6655 Views
- 2 replies
- 1 kudos
Resolved! Difference between username and account_id
I have a web app that can read files from a person's cloud-based drive (e.g., OneDrive, Google Drive, Dropbox). The app gets access to the files using OAuth2. The app only ever has access to the files for that user. Part of the configuration requir...
- 6655 Views
- 2 replies
- 1 kudos
- 1 kudos
The provided links were helpful. The take-away* usernames are "globally" unique to an individual; the username is the person's email.* a username can be associated with up to 50 accounts; account_ids track the resources available to the user.This cl...
- 1 kudos
