cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Direo
by Contributor II
  • 144 Views
  • 1 replies
  • 0 kudos

Can Databricks federation policy support cross-cloud authentication?

Hi!I'm exploring options for workload identity federation and have a question about cross-cloud scenarios.Current Setup:Azure Databricks workspaceWorkloads running in GCP (planning to use GKE/Kubernetes)Need to authenticate GCP-based workloads to Azu...

  • 144 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Yes, Databricks federation policy can support cross-cloud authentication, allowing the use of external identity providers (IdPs) that may reside in different clouds. This includes scenarios where tokens issued by trusted IdPs—such as those for servic...

  • 0 kudos
sfishel18
by New Contributor II
  • 430 Views
  • 5 replies
  • 0 kudos

GEOMETRY column type breaks all access to table from Spark

Hello, I have a Databricks table with a column using the new GEOMETRY type. When I try to access this table from a Spark workload, I am not able to describe the table or operate on any of its columns. My Spark config is the following, per the Databri...

  • 430 Views
  • 5 replies
  • 0 kudos
Latest Reply
sfishel18
New Contributor II
  • 0 kudos

Hi @NandiniN,To my understanding, there is no way to use the Databricks SDK as a catalog implementation in an external Spark workload, correct?The table I'm using is a Delta table I created from a Databricks notebook. The rows represent some areas-of...

  • 0 kudos
4 More Replies
shanisolomonron
by New Contributor
  • 253 Views
  • 4 replies
  • 1 kudos

Merge, Update and deletes using deletion vectors

Hi,When deletion vectors are enabled on a Delta table, is there a guarantee that MERGE, UPDATE, or DELETE operations will not rewrite unmodified data, but rather use deletion vectors to soft delete the original file?For example, suppose the table cur...

  • 253 Views
  • 4 replies
  • 1 kudos
Latest Reply
shanisolomonron
New Contributor
  • 1 kudos

Thanks @K_Anudeep.Could you clarify the condition in which deletion vectors might be enabled, but I might not see the above sequence of transactions, but rather see:Remove a.parquetAdd a new file (e.g. new.parquet) containing the non-deleted + update...

  • 1 kudos
3 More Replies
wschoi
by New Contributor III
  • 13116 Views
  • 15 replies
  • 13 kudos

How to fix plots and image color rendering on Notebooks?

I am currently running dark mode for my Databricks Notebooks, and am using the "new UI" released a few days ago (May 2023) and the "New notebook editor."Currently all plots (like matplotlib) are showing wrong colors. For example, denoting:```... p...

  • 13116 Views
  • 15 replies
  • 13 kudos
Latest Reply
dbx_acbz
New Contributor
  • 13 kudos

Hi there! Is there any update on this issue?

  • 13 kudos
14 More Replies
rgrosskopf
by New Contributor II
  • 7120 Views
  • 3 replies
  • 1 kudos

How to access secrets in Hashicorp Vault from Databricks notebooks?

I see in this blog post that Databricks supports Hashicorp Vault for secrets storage but I've been unable to find any additional details on how that would work. Specifically, how would I authenticate to Vault from within a Databricks notebook?

  • 7120 Views
  • 3 replies
  • 1 kudos
Latest Reply
srvmsr
New Contributor
  • 1 kudos

If you looking for gcp databricks and hashicorp vault integration, please find the blog post describing the setup .https://www.linkedin.com/pulse/gcp-databricks-hashicorp-vault-integration-tufan-mishra-kveie 

  • 1 kudos
2 More Replies
Pat
by Esteemed Contributor
  • 1393 Views
  • 1 replies
  • 1 kudos

Pipeline Tags in development mode

Hey,I couldn't find this documented anywhere, but I have been deploying Databricks Workflows (now called Jobs, I believe) with Pipelines using DABs. I have a 1 set of configuration, so there is no place for human error here.When I deploy bundle in DE...

  • 1393 Views
  • 1 replies
  • 1 kudos
Latest Reply
sarahbhord
Databricks Employee
  • 1 kudos

Hey Pat - thanks for reaching out. There’s no official intent for tag visibility to differ between environments when the config and deployments are consistent. Is the CLI version the same for prod and dev workspaces? Can you make sure that all of the...

  • 1 kudos
Sagar_0607
by New Contributor
  • 48 Views
  • 1 replies
  • 1 kudos

Need the output of a task in Databricks job in JSON format

Where can I see the logs in JSON format of the output produced by a task in Databricks jobs?

  • 48 Views
  • 1 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @Sagar_0607 ,You can use following REST API endpoint, which let's you retrieve the output and metadata of a single task run:Get the output for a single run | Jobs API | REST API reference | Databricks on AWSWith this endpoint, when a notebook task...

  • 1 kudos
santosh-santosh
by New Contributor II
  • 416 Views
  • 8 replies
  • 0 kudos

Create external tables with properties set in delta log and no collation

There is an external delta lake table that need to be mounted on to the unity catalogIt has some properties configured in the _delta_log folder alreadyWhen try to create table using CREATE TABLE catalog_name.schema_name.table_name USING DELTA LOCATIO...

  • 416 Views
  • 8 replies
  • 0 kudos
Latest Reply
ManojkMohan
Valued Contributor III
  • 0 kudos

@santosh-santosh Did you execute the steps part by part i shared in the DM ? Step 0: Define your external tables, Step 1: Inspect external Delta table schema & properties Success Check:Ensure all expected properties  are captured  Step 2: Inspect Uni...

  • 0 kudos
7 More Replies
eballinger
by Contributor
  • 1187 Views
  • 6 replies
  • 0 kudos

Email notification to end users

Is there a way a way we can notify all of our databricks end users by email when there is a issue? We currently have our jobs setup to notify the technical team when a job workflow fails. That part works fine.But we would like the ability to maybe us...

  • 1187 Views
  • 6 replies
  • 0 kudos
Latest Reply
AnanthuR
New Contributor
  • 0 kudos

Hello,I have a similar doubt!I’m running a data pipeline on Databricks, and at the end of the pipeline, I generate some results. I’d like to notify the relevant people automatically via email (or another messaging method) with a summary of the result...

  • 0 kudos
5 More Replies
Hanfo2back
by New Contributor
  • 233 Views
  • 5 replies
  • 4 kudos

Resolved! DLT Pipeline Failed to create new KafkaAdminClient SQLSTATE: XXKST:

I encountered the error: No LoginModule found for org.apache.kafka.common.security.scram.ScramLoginModule while consuming data from Kafka using a Databricks pipeline. The pipeline had been running smoothly before, but the error appeared on September ...

  • 233 Views
  • 5 replies
  • 4 kudos
Latest Reply
Advika
Databricks Employee
  • 4 kudos

Hello @Hanfo2back! Can you please try changing SASL login string to use kafkashaded.org.apache.kafka.common.security.scram.ScramLoginModule instead of org.apache.kafka.common.security.scram.ScramLoginModule.

  • 4 kudos
4 More Replies
Jpeterson
by New Contributor III
  • 4814 Views
  • 7 replies
  • 4 kudos

Databricks SQL Warehouse, Tableau and spark.driver.maxResultSize error

I'm attempting to create a tableau extract on tableau server with a connection to databricks large sql warehouse. The extract process fails due to spark.driver.maxResultSize error.Using a databricks interactive cluster in the data science & engineer...

  • 4814 Views
  • 7 replies
  • 4 kudos
Latest Reply
Varina1312
New Contributor
  • 4 kudos

I've totally been there with Tableau extracts choking on Databricks! That error is a real pain. I haven't worked directly with SQL Warehouse config, but your interactive cluster workaround makes sense. As for reducing data size, could you try aggrega...

  • 4 kudos
6 More Replies
jin2631816
by New Contributor II
  • 627 Views
  • 5 replies
  • 0 kudos

[Free Edition] Outbound internet suddenly blocked - Error: HTTPSConnectionPool(host='www.google.com'

Hi guys,I'm using the new Databricks Free Edition, and I'm seeing what looks like a sudden change in outbound internet access policy.Yesterday morning, I was able to access external APIs and test simple internet calls using Python and %sh commands in...

  • 627 Views
  • 5 replies
  • 0 kudos
Latest Reply
WiliamRosa
New Contributor III
  • 0 kudos

Hi @test_user_12, @jin2631816,If it helps, here’s the official documentation with the limitations of the Free Edition:https://docs.databricks.com/aws/en/getting-started/free-edition-limitations

  • 0 kudos
4 More Replies
Bedoonraj
by New Contributor
  • 256 Views
  • 3 replies
  • 0 kudos

TEMPORARILY_UNAVAILABLE: The service at /api/2.1/unity-catalog/tables is taking too long to process

I'm using DBT to run a model in databricks. I have view model, which holds 2 months of data (~2 million). There is no wide dependency transformation. All are case when statements. Total column no is 234. Till yesterday view was running fine. but toda...

  • 256 Views
  • 3 replies
  • 0 kudos
Latest Reply
WiliamRosa
New Contributor III
  • 0 kudos

Hi @Bedoonraj,I tested the API call and it worked fine. I also confirmed, just like Khaja_Zaffer mentioned, that there’s no instability with this service. I’d suggest checking your cluster settings and, if possible, trying the test on a different com...

  • 0 kudos
2 More Replies
BMex
by New Contributor III
  • 428 Views
  • 3 replies
  • 2 kudos

Resolved! Issue with Databricks Jobs: SQLSTATE: XXKST

Hi,we have our Databricks Jobs deployed via DABs, and they have been running fine for a while now (approximately 1 month since we migrated from ADF). However, since yesterday, we are getting a weird issue while writing. See error below:[STREAM_FAILED...

Data Engineering
Databricks
databricks-sql
jobs
spark
sqlstate
  • 428 Views
  • 3 replies
  • 2 kudos
Latest Reply
WiliamRosa
New Contributor III
  • 2 kudos

Hi @BMex,The link I shared with a similar issue contains some solutions — did any of them work for you?

  • 2 kudos
2 More Replies
ManojkMohan
by Valued Contributor III
  • 270 Views
  • 1 replies
  • 0 kudos

Databricks ro Salesforce | Unity Catalog Query

AskCan we get a UC catalog (like prod or genie) in free edition of data bricks ?Problem i am solving:Structuring Data in Databricks before sending customer, account data to salesforceissue:cannot see workspace-local tables (workspace.default.structur...

image (1).png image (2).png
  • 270 Views
  • 1 replies
  • 0 kudos
Latest Reply
RogerThatttt
New Contributor II
  • 0 kudos

root cause of not seeing your workspace-local tables (workspace.default.structured_pdf_table) is the unavailability of a Unity Catalog or Delta Sharing connector configuration in your Free Edition workspace. To resolve this, you typically need admin ...

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels