We already know that we can mount Azure Data Lake Gen2 with OAuth2 using this:configs = {"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
...
Is there any update on this feature request? OAuth still seems not to be working with Azure Blob Storage... Configuration works fine for ADLS gen 2, but for Azure Blob Storage still only SAS and Account key seems to be working.
Hi,I am trying to install the following four dependency packages in order to install "slu-openGIS/postmastr" directly from GitHub:unitssftigristidycensusWhen attempting to install "units", I received the following configuration error: %r
install.pack...
Hi, I am using debezium server to send data from Postgres to a Kafka topic (in fact, Azure EventHub). My question is, what are the best practices and recommendations to save raw data and then implement a medallion architecture?For clarification, I wa...
Hello,I am trying to run the Classroom-Setup from the course files notebook (ASP 1.1 - Databricks Platform)(Course:Apache Spark™ Programming with DatabricksID: E-P0W7ZV)Instructions: "Setup:Run classroom setup to mount Databricks training datasets an...
Hi @rhevarr, Thank you for posting your concern on Community!
To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours.
The latest version of Databricks-jdbc available through Maven (2.6.36) now has these three vulnerabilities:https://www.cve.org/CVERecord?id=CVE-2021-36090https://www.cve.org/CVERecord?id=CVE-2023-6378https://www.cve.org/CVERecord?id=CVE-2023-6481All ...
Hi @karthik-kobai,
Thank you for bringing this to my attention! Let’s address the vulnerabilities in the Databricks JDBC driver.
The current version of the Databricks JDBC driver you mentioned is 2.6.361. It appears that this version has dependen...
When trying to use `dbutils.fs.cp` in the #databricks-connect #databricks-connect context to upload files to Azure Datalake Gen2 I get a malformed URI errorI have used the code provided here:https://learn.microsoft.com/en-gb/azure/databricks/dev-tool...
Hi @KrzysztofPrzyso, It appears that you’re encountering an issue with relative paths in absolute URIs when using dbutils.fs.cp in the context of Databricks Connect to upload files to Azure Data Lake Gen2.
Let’s break down the problem and explore po...
Hi There,I'm currently going through Module 4 of the Data Engineering Associate pathway, specifically lesson 4.1 - DLT UI Walkthrough. We are instructed to specify the Cluster Policy as 'DBAcademy DLT' when configuring the pipeline. However, this opt...
Hi @subrat.,
Thank you for posting your concern on Community!
To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours.
I wish to run aws CLI command in databricks, is there a way i can achieve the same, to be more specific i would like to run:aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-...
Hi @yatharth ,
Certainly! You can use the Databricks CLI to run commands and automate tasks within your Databricks environment. The Databricks CLI provides a set of commands that allow you to interact with Databricks workspaces, clusters, librari...
Flow 'user_silver' has FAILED fatally. An error occurred because we detected an update or delete to one or more rows in the source table. Streaming tables may only use append-only streaming sources. If you expect to delete or update rows to the sourc...
Hey @rt-slowth Just checking in if the provided solution was helpful to you. If yes, please accept this as a Best Solution so that this thread can be considered closed.
I'm envisioning a live streaming pipeline.The bronze, or data ingestion, is being fetched using the directory listing mode of the autoloader.I'm not using File Notification Mode because I detect about 2-300 data changes per hour.I'm thinking about im...
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
Currently I have access to Dynamics CRM backend server via AAD, and I can query tables via XRM tool. I am trying to connect to Dynamics CRM backend server in Databricks, and I am not sure how the connection needs to be set up or if any other access n...
Hi Dilorom - did you ever solve your issue? I'm trying to connect to Microsoft Dynamics Business Central and get an error when I run %pip install dynamics365bc.ERROR: Could not find a version that satisfies the requirement dynamics365bc (from version...
Delta solves the large number of small file problems using the below operations available for a Delta table. Optimize writes helps to optimizes the write operation by adding an additional shuffle step and reducing the number of output files. By defau...
Hi Team,We are wondering if there is a recommended way to install the chromium browser and chrome driver on Databricks Runtime 10.4 and above ?I have been through the site and have come across several links to this effect, but they all seem to be ins...
Look into Playwrite instead of Selenium. I went through the same process y'all went through here (ended up writing a init script to install the drivers etc)This is all done for you in playwright. Refer to this post - I hope it helps!!https://communit...
I'm trying to create an external location from notebook, and I got this kind of error[PARSE_SYNTAX_ERROR] Syntax error at or near 'LOCATION'(line 1, pos 16)
== SQL ==
CREATE EXTERNAL LOCATION IF NOT EXISTS test_location URL 's3://test-bronze/db/tes...
Hey everyone,I'm facing an issue with retrieving data from a volume or table that contains a string with a symbol, for example, 'databricks+'. Whenever I try to retrieve this data, I encounter a syntax error. Can anyone help me resolve this issue?
Hello @Debayan please how can i collect metrics provided by clusters metrics for databricks runtime 13.1 or latest using shell bash script. Cordially, Aubert EMAKO
