- 8636 Views
- 5 replies
- 7 kudos
Resolved! Databricks Autoloader is getting stuck and does not pass to the next batch
I have a simple job scheduled every 5 min. Basically it listens to cloudfiles on storage account and writes them into delta table, extremely simple. The code is something like this:df = (spark .readStream .format("cloudFiles") .option('cloudFil...
- 8636 Views
- 5 replies
- 7 kudos
- 7 kudos
I had the same issue: files would randomly not be loaded.Setting `.option("cloudFiles.useIncrementalListing", False)` Seemed to do the trick!
- 7 kudos
- 4827 Views
- 3 replies
- 4 kudos
Resolved! Spot instance in Azure Databricks
When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run In order to identify the spot instances we got for any...
- 4827 Views
- 3 replies
- 4 kudos
- 20374 Views
- 4 replies
- 26 kudos
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace �� Private link In your storage accounts please go to “Networ...
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace Private linkIn your storage accounts please go to “Networking” -> “Private endpoint connections” and click Add Private Endpoint.It is important to add private links in ...
- 20374 Views
- 4 replies
- 26 kudos
- 5582 Views
- 7 replies
- 4 kudos
need help with Azure Databricks questions on CTE and SQL syntax within notebooks
Hi amazing community folks,Feel free to share your experience or knowledge regarding below questions:-1.) Can we pass a CTE sql statement into spark jdbc? i tried to do it i couldn't but i can pass normal sql (Select * from ) and it works. i heard th...
- 5582 Views
- 7 replies
- 4 kudos
- 4 kudos
Not a comparison, but there is a DB-SQL cheatsheet at https://www.databricks.com/sites/default/files/2023-09/databricks-sql-cheatsheet.pdf/
- 4 kudos
- 5998 Views
- 5 replies
- 1 kudos
Resolved! How to use Databricks Query History API (REST API)
I have setup authentication using this page https://docs.databricks.com/sql/api/authentication.html and run curl -n -X GET https://<databricks-instance>.cloud.databricks.com/api/2.0/sql/history/queriesTo get history of all sql endpoint queries, but I...
- 5998 Views
- 5 replies
- 1 kudos
- 1 kudos
Here's how to query with databricks-sdk-py (working code). I had a frustrating time doing it with vanilla python + requests/urllib and couldn't figure it out. import datetime import os from databricks.sdk import WorkspaceClient from databricks.sdk.se...
- 1 kudos
- 14141 Views
- 9 replies
- 6 kudos
Resolved! dropping a managed table does not remove the underlying files
the documentation states that "drop table":Deletes the table and removes the directory associated with the table from the file system if the table is not EXTERNAL table. An exception is thrown if the table does not exist.In case of an external table...
- 14141 Views
- 9 replies
- 6 kudos
- 6 kudos
Hi,There is a way to force delete files after drop the table and don't wait 30 days to see size in S3 decrease?Tables that I dropped related to the dev and staging, I don't want to keep there files for 30 days
- 6 kudos
- 2441 Views
- 4 replies
- 2 kudos
User entry via portal/UI
Hello Experts,We have data bricks on Azure. We need to provide a user interface to users so that some customizing table entries can be entered by end users which in turn are saved in the Delta table. Is there any feature in Databricks or tools that w...
- 2441 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Viren, I have the same use case on my side and looking for some tips. I was thinking Power Apps and Power Automate with API calls for a job that will run the process but happy to check on other solutions as well !Regards
- 2 kudos
- 3883 Views
- 6 replies
- 7 kudos
Azure DevOps Repo - Invalid Git Credentials
I have a Repo in Databricks connected to Azure DevOps Repositories.The repo has been working fine for almost a month, until last week. Now when I try to open the Git settings in Databricks, I am getting "Invalid Git Credentials". Nothing has change...
- 3883 Views
- 6 replies
- 7 kudos
- 18152 Views
- 8 replies
- 2 kudos
Connecting to Snowflake using an SSO user from Azure Databricks
Hi all,I'm just reaching out to see if anyone has information or can point me in a useful direction. I need to connect to Snowflake from Azure Databricks using the connector: https://learn.microsoft.com/en-us/azure/databricks/external-data/snowflakeT...
- 18152 Views
- 8 replies
- 2 kudos
- 2 kudos
we ended up using device flow oauth because, as noted above, it is not possible to launch a browser on the Databricks cluster from a notebook so you cannot use "externalBrowser" flow. It gives you a url and a code and you open the url in a new tab an...
- 2 kudos
- 7500 Views
- 5 replies
- 4 kudos
Import module never ends on random branches
Hello,Since a week ago, our notebook are stuck in running on the firsts cells which import python module from our github repository which is cloned in databricks.The cells stays in running state and when we try to manually cancel the jobs in databric...
- 7500 Views
- 5 replies
- 4 kudos
- 4892 Views
- 3 replies
- 2 kudos
Resolved! Configure multiple source paths for auto loader
I am currently using two streams to monitor data in two different containers on an Azure storage account. Is there any way to configure an autoloader to read from two different locations? The schemas of the files are identical.
- 4892 Views
- 3 replies
- 2 kudos
- 2 kudos
@Morten Stakkeland :Yes, it's possible to configure an autoloader to read from multiple locations.You can define multiple CloudFiles sources for the autoloader, each pointing to a different container in the same storage account. In your case, since ...
- 2 kudos
- 10657 Views
- 4 replies
- 6 kudos
Resolved! Redirect error in access to web app in Azure Databricks with private front endpoint
I have created a workspace with private endpoint in Azure following this guide:https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-linkOnce I have created the private link of type browser_authent...
- 10657 Views
- 4 replies
- 6 kudos
- 6 kudos
You don't need a CNAME record.Go to your private link resource in Azure and click on Settings > DNS Configuration. Make sure you have created private link A records for all the FQDNs listed under 'Custom DNS records'. You have most likely missed one ...
- 6 kudos
- 4065 Views
- 3 replies
- 2 kudos
Resolved! How to mount Azure Blob Storage with OAuth2?
We already know that we can mount Azure Data Lake Gen2 with OAuth2 using this:configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", ...
- 4065 Views
- 3 replies
- 2 kudos
- 2 kudos
Try replacing wasbs with abfss and dfs with blob in the URI, should work!
- 2 kudos
- 12172 Views
- 4 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 12172 Views
- 4 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
- 0 kudos
- 7319 Views
- 11 replies
- 2 kudos
Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection
HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. Is it by design? Is it a limitation when using Privat...
- 7319 Views
- 11 replies
- 2 kudos
- 2 kudos
Fixed, go to Profile -> Compute-> SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.
- 2 kudos