cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
597581
by • New Contributor III
  • 1621 Views
  • 22 replies
  • 26 kudos

Resolved! Run selected text shortcut not working

The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...

Data Engineering
Reply
  • 1621 Views
  • 22 replies
  • 26 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 26 kudos

Folks, could you please double-check now, issue should be fixed now. Thanks!

  • 26 kudos
21 More Replies
AtulMathur
by • New Contributor II
  • 215 Views
  • 2 replies
  • 1 kudos

Resolved! Comparing two SQL notebooks from different Environments

Hello Everyone,I am part of data testing team which is responsible to verify data trends and insights generated from different sources. There are multiple schemas and tables in our platform. We use SQL queries in notebooks to verify all enrichment, m...

Data Engineering
Reply
  • 215 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
AtulMathur
New Contributor II
  • 1 kudos

Thank you Walter. I did thought about it doing it one by one but then it was not coming out to be very efficient way. I  then found a way to do it in Python via iterating through a dataframe of table names.

  • 1 kudos
1 More Replies
ns_casper
by • New Contributor II
  • 648 Views
  • 3 replies
  • 0 kudos

Databricks Excel ODBC driver bug

Hello!I might have experienced a bug with the ODBC driver. We have an issue where given certain priviledges in databricks, the ODBC driver is unable to show any schemas/tables.When we click the 'expand' button on any catalog in the list (of which we ...

Data Engineering
Reply
  • 648 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
KTheJoker
Databricks Employee
  • 0 kudos

Hey @ns_casper when you list catalog/schemas in Excel, it fires off queries to Databricks SQL. Can you see those queries in the Query History? Are there any errors there? I suspect there is some issue with the underlying catalog that's preventing the...

  • 0 kudos
2 More Replies
JesseSchouten
by • New Contributor
  • 186 Views
  • 0 replies
  • 0 kudos

DLT issue - slow download speed in DLT clusters

Hi all,I'm encountering some issues within my DLT pipelines. Summarized: it takes a long time to install the cluster libraries and dependencies (using %pip installs) due to horribly slow download speeds.These are the symptoms:- From all purpose clust...

Data Engineering
Reply
  • 186 Views
  • 0 replies
  • 0 kudos
OliverCadman
by • New Contributor III
  • 14958 Views
  • 7 replies
  • 6 kudos

'File not found' error when executing %run magic command

I'm just walking through a simple exercise presented in the Databricks Platform Lab notebook, in which I'm executing a remote notebook from within using the %run command. The remote notebook resides in the same directory as the Platform Lab notebook,...

Data Engineering
%file_not_found
%magic_commands
%run
Reply
  • 14958 Views
  • 7 replies
  • 6 kudos
View Replies
Latest Reply
ArturOA
New Contributor III
  • 6 kudos

I have seen this error popping up when you define a Python file without the header: # Databricks notebook source Databricks has a hard time running the file as a nested notebook and you can get some weird errors.   

  • 6 kudos
6 More Replies
Brad
by • Contributor II
  • 320 Views
  • 3 replies
  • 0 kudos

What is "ExecuteGrpcResponseSender: Deadline reached, shutting down stream"

 Hi, I have a delta table which is loaded by structured streaming job. When I tried to read this delta table and do a MERGE with foreachBatch, I found sometimes there is a big interval between streaming starts and MERGE starting to run and seems spar...

Data Engineering
Reply
  • 320 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

It may not necessarily be a bug, but some tuning due to architectural differences. What the message says is: The system was processing a gRPC operation identified by opId=5ef071b7-xxx, and it set a deadline for that operation (likely 120 seconds).The...

  • 0 kudos
2 More Replies
dtb_usr
by • New Contributor II
  • 348 Views
  • 1 replies
  • 1 kudos

Creating a private connection with Google Sheets

How do I ingest sensitive data from a googlesheet doc to Databricks unity catalogue without making the googlesheet public.

Data Engineering
Reply
  • 348 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
agallard
New Contributor III
  • 1 kudos

Hi @dtb_usr,You can share the Google Sheet with the Service Account and use Google Sheets API ClientOpen the Google Sheet you want to access.Click on Share and add the email address of the service account (it will look something like your-service-acc...

  • 1 kudos
radix
by • New Contributor II
  • 224 Views
  • 1 replies
  • 0 kudos

Databricks cluster pools with init scripts

Ability to submit a single job with cluster pools and init scriptsfor the following payload:  { "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": ...

Data Engineering
Reply
  • 224 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Are you still facing issues with the job run submit API endpoint?

  • 0 kudos
jonxu
by • New Contributor II
  • 277 Views
  • 2 replies
  • 1 kudos

Resolved! streaming vs batch, unbounded vs bound

Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?i.e., if I set one second as the frequency for batch processing, will it...

Data Engineering
Reply
  • 277 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jonxu
New Contributor II
  • 1 kudos

Many thanks for the clarification!

  • 1 kudos
1 More Replies
PremPrakash
by • New Contributor II
  • 188 Views
  • 2 replies
  • 1 kudos

Resolved! Using instance profile for sns message publish with PassRole

Hi, I want to attach instance profile to compute and  publish message on SNS without using credentials. Is that possible? has anyone used it. Will Boto3 support it?

Data Engineering
Reply
  • 188 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
PremPrakash
New Contributor II
  • 1 kudos

Yes, I have tried it, it is working. 

  • 1 kudos
1 More Replies
stiaangerber
by • New Contributor III
  • 218 Views
  • 1 replies
  • 0 kudos

Simba ODBC for ARM-based Linux

HiIs there an ARM build of the Simba ODBC driver available for Linux? I've seen this thread (for Mac)https://community.databricks.com/t5/data-engineering/problems-connecting-simba-odbc-with-a-m1-macbook-pro/td-p/20566but it seems that there are only ...

Data Engineering
Reply
  • 218 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@stiaangerber unfortunately not. The odbc driver for linux is technically CPU agnostic, as long as it's amd64/x86_64, it should work, but we don't have one for ARM.

  • 0 kudos
sandeephenkel23
by • New Contributor III
  • 187 Views
  • 3 replies
  • 0 kudos

QuantileDiscretizer is not whiteliested erro!!

Dear Team,We observed that while attempting to use the following import:from pyspark.sql import functions as Ffrom pyspark.ml.feature import QuantileDiscretizerwe are encountering the following error:Py4JSecurityException: QuantileDiscretizer is not ...

Data Engineering
Reply
  • 187 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a...

  • 0 kudos
2 More Replies
DRock
by • New Contributor II
  • 411 Views
  • 5 replies
  • 0 kudos

Resolved! ODBC data source to connect to a Databricks catalog.database via MS Access Not Working

When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...

Data Engineering
Reply
  • 411 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Right, that indeed seems to be the problem, ideally MS Access should be fetching the default correct values, but only newer clients are. Instead of our original approach, could you please: 1.- Open the ODBC Driver DSN Setup. 2.- Go to Advanced Option...

  • 0 kudos
4 More Replies
jeft
by • New Contributor II
  • 120 Views
  • 1 replies
  • 0 kudos

mongodb ingest data into databricks error

spark = SparkSession.builder \.appName("MongoDBToDatabricks") \.config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:10.4.0") \.config("spark.mongodb.read.connection.uri", mongodb_uri) \.config("spark.mongodb.write.connection.u...

Data Engineering
Reply
  • 120 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jeft
New Contributor II
  • 0 kudos

 it shows maximun number of retires has been exceeded. i have no idea why it couldnt connect as i able to telnet to mongoDB

  • 0 kudos
RoelofvS
by • New Contributor II
  • 420 Views
  • 5 replies
  • 0 kudos

Schema evolution in Autoloader not evolving beyond version 0

I am working through the current version of the standard AutoLoader demo, i.e.    dbdemos.install('auto-loader')I.e. data gets read into a dataframe, but never written to a target table.Notebook is "01-Auto-loader-schema-evolution-Ingestion"Compute i...

Data Engineering
Reply
  • 420 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
RoelofvS
New Contributor II
  • 0 kudos

Hello @Brahmareddy,I have tried the above, without success.> enable detailed logging to trace schema evolution stepsPlease can you giude me with the steps or a URL? We are on AWS.Kind regards - Roelof 

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels