cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Ian_P
by New Contributor II
  • 2853 Views
  • 5 replies
  • 1 kudos

Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode): py4j.security.Py4JSecurityException: Method public java.la...

Ian_P_0-1690531566535.png
Data Engineering
Databricks
spark
Unity Catalog
  • 2853 Views
  • 5 replies
  • 1 kudos
Latest Reply
Ian_P
New Contributor II
  • 1 kudos

@Ayushi_Suthar @Yulei After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster. RegardsIan

  • 1 kudos
4 More Replies
prasad95
by New Contributor III
  • 1276 Views
  • 4 replies
  • 2 kudos
Data Engineering
Delta Lake
  • 1276 Views
  • 4 replies
  • 2 kudos
Latest Reply
saikumar246
New Contributor III
  • 2 kudos

Hi, @prasad95 Thank you for sharing your concern here.  In addition to the @Kaniz comments you can follow below To capture Change Data (CDC) from DynamoDB Streams and write it into a Delta table in Databricks: 1. Connect to DynamoDB Streams and read ...

  • 2 kudos
3 More Replies
User16752245312
by New Contributor III
  • 1478 Views
  • 3 replies
  • 2 kudos
  • 1478 Views
  • 3 replies
  • 2 kudos
Latest Reply
Aria
New Contributor III
  • 2 kudos

We are using azure.I dont see an option for deployment name. Secondly, we have already deployed all our workspaces and wants to have user friendly URLs.Like some changes in DNS server or proxy URLs.

  • 2 kudos
2 More Replies
asingamaneni
by New Contributor II
  • 637 Views
  • 0 replies
  • 0 kudos

Brickflow - An opinionated python framework to help build and deploy Databricks workflows at scale

Hello, Databricks Community, I’m thrilled to introduce “Brickflow,” our innovative solution designed to streamline the development and deployment of workflows at scale, leveraging straightforward Python coding. This tool has already achieved a signif...

  • 637 Views
  • 0 replies
  • 0 kudos
Kaizen
by Contributor III
  • 755 Views
  • 3 replies
  • 0 kudos

Copy Local file using a Shared Cluster

Hi, I am saving some files locally on my cluster and moving them after my job. These are log files of my process so I cant directly reference a DBFS location. However the dbutils.fs.cp command does not work on the shared cluster. This does however wo...

  • 755 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaizen
Contributor III
  • 0 kudos

For reference when doing this on a single user (personal) cluster - the file is store in:/databricks/driver/logs.txt Which has no issue accessing and copying to dbfs after using the dbutil commands

  • 0 kudos
2 More Replies
jwilliam
by Contributor
  • 2085 Views
  • 3 replies
  • 2 kudos

Resolved! How to mount Azure Blob Storage with OAuth2?

We already know that we can mount Azure Data Lake Gen2 with OAuth2 using this:configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", ...

  • 2085 Views
  • 3 replies
  • 2 kudos
Latest Reply
mathijs-fish
New Contributor III
  • 2 kudos

Is there any update on this feature request? OAuth still seems not to be working with Azure Blob Storage... Configuration works fine for ADLS gen 2, but for Azure Blob Storage still only SAS and Account key seems to be working.

  • 2 kudos
2 More Replies
thethirtyfour
by New Contributor III
  • 1062 Views
  • 2 replies
  • 1 kudos

Resolved! Install R Package "sf"

Hi,I am trying to install the following four dependency packages in order to install "slu-openGIS/postmastr" directly from GitHub:unitssftigristidycensusWhen attempting to install "units", I received the following configuration error: %r install.pack...

  • 1062 Views
  • 2 replies
  • 1 kudos
Latest Reply
thethirtyfour
New Contributor III
  • 1 kudos

Thank you!

  • 1 kudos
1 More Replies
jcozar
by Contributor
  • 4509 Views
  • 5 replies
  • 2 kudos

Resolved! CDC and raw data

Hi, I am using debezium server to send data from Postgres to a Kafka topic (in fact, Azure EventHub). My question is, what are the best practices and recommendations to save raw data and then implement a medallion architecture?For clarification, I wa...

  • 4509 Views
  • 5 replies
  • 2 kudos
Latest Reply
jcozar
Contributor
  • 2 kudos

Thank you very much @Palash01 ! It has been really helpful!

  • 2 kudos
4 More Replies
rhevarr
by New Contributor II
  • 387 Views
  • 1 replies
  • 0 kudos

Course: Apache Spark Programming with Databricks ID: E-P0W7ZV // Issue Classroom-Setup

Hello,I am trying to run the Classroom-Setup from the course files notebook (ASP 1.1 - Databricks Platform)(Course:Apache Spark™ Programming with DatabricksID: E-P0W7ZV)Instructions: "Setup:Run classroom setup to mount Databricks training datasets an...

Data Engineering
academy
Course
Databricks
spark
  • 387 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @rhevarr, Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours.

  • 0 kudos
karthik-kobai
by New Contributor II
  • 449 Views
  • 1 replies
  • 0 kudos

Databricks-jdbc and vulnerabilities CVE-2021-36090 CVE-2023-6378 CVE-2023-6481

The latest version of Databricks-jdbc available through Maven (2.6.36) now has these three vulnerabilities:https://www.cve.org/CVERecord?id=CVE-2021-36090https://www.cve.org/CVERecord?id=CVE-2023-6378https://www.cve.org/CVERecord?id=CVE-2023-6481All ...

  • 449 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @karthik-kobai,  Thank you for bringing this to my attention! Let’s address the vulnerabilities in the Databricks JDBC driver. The current version of the Databricks JDBC driver you mentioned is 2.6.361. It appears that this version has dependen...

  • 0 kudos
subrat
by New Contributor
  • 806 Views
  • 1 replies
  • 0 kudos

Missing 'DBAcademy DLT' as a Cluster Policy when creating Delta Live Tables pipeline

Hi There,I'm currently going through Module 4 of the Data Engineering Associate pathway, specifically lesson 4.1 - DLT UI Walkthrough. We are instructed to specify the Cluster Policy as 'DBAcademy DLT' when configuring the pipeline. However, this opt...

  • 806 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @subrat.,  Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours.

  • 0 kudos
rt-slowth
by Contributor
  • 1249 Views
  • 5 replies
  • 0 kudos

Error : . If you expect to delete or update rows to the source table in the future.......

Flow 'user_silver' has FAILED fatally. An error occurred because we detected an update or delete to one or more rows in the source table. Streaming tables may only use append-only streaming sources. If you expect to delete or update rows to the sourc...

  • 1249 Views
  • 5 replies
  • 0 kudos
Latest Reply
Palash01
Contributor III
  • 0 kudos

Hey @rt-slowth Just checking in if the provided solution was helpful to you. If yes, please accept this as a Best Solution so that this thread can be considered closed.

  • 0 kudos
4 More Replies
rt-slowth
by Contributor
  • 1979 Views
  • 6 replies
  • 0 kudos

Questions about the design of bronze, silver, and gold for live streaming pipelines

I'm envisioning a live streaming pipeline.The bronze, or data ingestion, is being fetched using the directory listing mode of the autoloader.I'm not using File Notification Mode because I detect about 2-300 data changes per hour.I'm thinking about im...

Data Engineering
Delta Live Table
spark
  • 1979 Views
  • 6 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
5 More Replies
Dilorom
by New Contributor
  • 1901 Views
  • 1 replies
  • 0 kudos

How to connect to Dynamics CRM server in Databricks.

Currently I have access to Dynamics CRM backend server via AAD, and I can query tables via XRM tool. I am trying to connect to Dynamics CRM backend server in Databricks, and I am not sure how the connection needs to be set up or if any other access n...

  • 1901 Views
  • 1 replies
  • 0 kudos
Latest Reply
sheridan06
New Contributor III
  • 0 kudos

Hi Dilorom - did you ever solve your issue? I'm trying to connect to Microsoft Dynamics Business Central and get an error when I run %pip install dynamics365bc.ERROR: Could not find a version that satisfies the requirement dynamics365bc (from version...

  • 0 kudos
User16869510359
by Esteemed Contributor
  • 5063 Views
  • 2 replies
  • 0 kudos

Resolved! How does Delta solve the large number of small file problems?

Delta creates more small files during merge and updates operations.

  • 5063 Views
  • 2 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Delta solves the large number of small file problems using the below operations available for a Delta table. Optimize writes helps to optimizes the write operation by adding an additional shuffle step and reducing the number of output files. By defau...

  • 0 kudos
1 More Replies
Labels