cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
ArturOA
by • New Contributor III
  • 184 Views
  • 3 replies
  • 0 kudos

Attaching to Serverless from Azure Data Factory via Service Principal

Hi,We have issues trying to run Databricks notebooks orchestrated with Azure Data Factory. We have been doing this for a while now without any issues when we use Job Clusters, Existing General Purpose Clusters, or Cluster Pools. We use an Azure Data ...

ArturOA_0-1729677593083.png
Data Engineering
Reply
  • 184 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @ArturOA ,Maybe you forget to give permission to ADF MSI to this serverless warehouse? Check how's your permission tab looks like. 

  • 0 kudos
2 More Replies
OliverCadman
by • New Contributor III
  • 14369 Views
  • 7 replies
  • 6 kudos

'File not found' error when executing %run magic command

I'm just walking through a simple exercise presented in the Databricks Platform Lab notebook, in which I'm executing a remote notebook from within using the %run command. The remote notebook resides in the same directory as the Platform Lab notebook,...

Data Engineering
%file_not_found
%magic_commands
%run
Reply
  • 14369 Views
  • 7 replies
  • 6 kudos
View Replies
Latest Reply
ArturOA
New Contributor III
  • 6 kudos

I have seen this error popping up when you define a Python file without the header: # Databricks notebook source Databricks has a hard time running the file as a nested notebook and you can get some weird errors.   

  • 6 kudos
6 More Replies
Brad
by • Contributor
  • 221 Views
  • 3 replies
  • 0 kudos

What is "ExecuteGrpcResponseSender: Deadline reached, shutting down stream"

 Hi, I have a delta table which is loaded by structured streaming job. When I tried to read this delta table and do a MERGE with foreachBatch, I found sometimes there is a big interval between streaming starts and MERGE starting to run and seems spar...

Data Engineering
Reply
  • 221 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

It may not necessarily be a bug, but some tuning due to architectural differences. What the message says is: The system was processing a gRPC operation identified by opId=5ef071b7-xxx, and it set a deadline for that operation (likely 120 seconds).The...

  • 0 kudos
2 More Replies
597581
by • New Contributor III
  • 769 Views
  • 19 replies
  • 20 kudos

Run selected text shortcut not working

The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...

Data Engineering
Reply
  • 769 Views
  • 19 replies
  • 20 kudos
View Replies
Latest Reply
CharP
New Contributor
  • 20 kudos

it seems that is has been fixed now...

  • 20 kudos
18 More Replies
dtb_usr
by • New Contributor
  • 177 Views
  • 1 replies
  • 1 kudos

Creating a private connection with Google Sheets

How do I ingest sensitive data from a googlesheet doc to Databricks unity catalogue without making the googlesheet public.

Data Engineering
Reply
  • 177 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
agallard
New Contributor III
  • 1 kudos

Hi @dtb_usr,You can share the Google Sheet with the Service Account and use Google Sheets API ClientOpen the Google Sheet you want to access.Click on Share and add the email address of the service account (it will look something like your-service-acc...

  • 1 kudos
SethParker
by • New Contributor II
  • 31 Views
  • 1 replies
  • 0 kudos

SQL View Formatting in Catalog - Can you turn it off?

It appears as though Databricks now formats SQL View definitions when showing them in the Catalog.  Our solution is based on views, and we have comment tags in those views.  We format these views so that it is easy for us to find and update parts of ...

SethParker_0-1730757882698.png SethParker_1-1730757945105.png
Data Engineering
Reply
  • 31 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you please provide me with the exact steps you are following to get this sample from the Catalog page?

  • 0 kudos
radix
by • New Contributor II
  • 169 Views
  • 1 replies
  • 0 kudos

Databricks cluster pools with init scripts

Ability to submit a single job with cluster pools and init scriptsfor the following payload:  { "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": ...

Data Engineering
Reply
  • 169 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Are you still facing issues with the job run submit API endpoint?

  • 0 kudos
jonxu
by • New Contributor II
  • 84 Views
  • 2 replies
  • 1 kudos

Resolved! streaming vs batch, unbounded vs bound

Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?i.e., if I set one second as the frequency for batch processing, will it...

Data Engineering
Reply
  • 84 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jonxu
New Contributor II
  • 1 kudos

Many thanks for the clarification!

  • 1 kudos
1 More Replies
AlexSantiago
by • New Contributor II
  • 1903 Views
  • 6 replies
  • 1 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

Data Engineering
Reply
  • 1903 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
Scofeild618
Visitor
  • 1 kudos

When using the Spotify API and encountering authentication issues, particularly with token generation, it's crucial to follow the correct OAuth flow. Verify that your backend is properly set up for token handling and ensure that the raw input call ad...

  • 1 kudos
5 More Replies
rvo19941
by • New Contributor
  • 43 Views
  • 0 replies
  • 0 kudos

Auto Loader File Notification Mode not working with ADLS Gen2 and files written as a stream

Dear,I am working on a real-time use case and am therefore using Auto Loader with file notification to ingest json files from a Gen2 Azure Storage Account in real-time. Full refreshes of my table work fine but I noticed Auto Loader was not picking up...

Data Engineering
ADLS
Auto Loader
Event Subscription
File Notification
Queue Storage
Reply
  • 43 Views
  • 0 replies
  • 0 kudos
PremPrakash
by • New Contributor
  • 69 Views
  • 2 replies
  • 1 kudos

Resolved! Using instance profile for sns message publish with PassRole

Hi, I want to attach instance profile to compute and  publish message on SNS without using credentials. Is that possible? has anyone used it. Will Boto3 support it?

Data Engineering
Reply
  • 69 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
PremPrakash
New Contributor
  • 1 kudos

Yes, I have tried it, it is working. 

  • 1 kudos
1 More Replies
thisisadarshsin
by • New Contributor II
  • 2167 Views
  • 5 replies
  • 0 kudos

Permission issue in Fundamentals of the Databricks Lakehouse Platform Quiz

Hi ,I am getting this Error,when i am trying to give the exam ofFundamentals of the Databricks Lakehouse Platform.403FORBIDDENYou don't have permission to access this page2023-05-20 12:37:41 | Error 403 | https://customer-academy.databricks.com/I al...

Data Engineering
Reply
  • 2167 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
shubham5
New Contributor II
  • 0 kudos

Hi Team,Even I'm also getting the same error last 2 days while Taking the quiz and get your badge for Lakehouse Fundamentals.Error : You are not authorized to access https://customer-academy.databricks.com. Please select a platform you can access fro...

  • 0 kudos
4 More Replies
jorgemarmol
by • New Contributor II
  • 2455 Views
  • 5 replies
  • 0 kudos

Delta Live Tables: Too much time to do the "setting up"

Hello community!Recently I have been working in delta live table for a big project. My team and me have been studying so much and finally we have built a good pipeline with CDC that load 608 entities (and, therefore, 608 delta live tables and 608 mat...

jorgemarmol_0-1688633577282.png
Data Engineering
Reply
  • 2455 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
charl-p-botha
New Contributor II
  • 0 kudos

Same issue here. Medallion architecture in a test setup with 150 tables. SETTING_UP_TABLES takes 11 to 15 minutes, everything else together takes 10 minutes to do the incremental update (40K bronze rows, these fan out to asset tables, not much data!)...

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels