cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
597581
by • New Contributor III
  • 626 Views
  • 18 replies
  • 20 kudos

Run selected text shortcut not working

The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...

Data Engineering
Reply
  • 626 Views
  • 18 replies
  • 20 kudos
View Replies
Latest Reply
AmberT
Visitor
  • 20 kudos

I have the same issue in cells running R

  • 20 kudos
17 More Replies
SethParker
by • New Contributor II
  • 13 Views
  • 1 replies
  • 0 kudos

SQL View Formatting in Catalog - Can you turn it off?

It appears as though Databricks now formats SQL View definitions when showing them in the Catalog.  Our solution is based on views, and we have comment tags in those views.  We format these views so that it is easy for us to find and update parts of ...

SethParker_0-1730757882698.png SethParker_1-1730757945105.png
Data Engineering
Reply
  • 13 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you please provide me with the exact steps you are following to get this sample from the Catalog page?

  • 0 kudos
radix
by • New Contributor II
  • 161 Views
  • 1 replies
  • 0 kudos

Databricks cluster pools with init scripts

Ability to submit a single job with cluster pools and init scriptsfor the following payload:  { "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": ...

Data Engineering
Reply
  • 161 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Are you still facing issues with the job run submit API endpoint?

  • 0 kudos
jonxu
by • New Contributor II
  • 56 Views
  • 2 replies
  • 1 kudos

Resolved! streaming vs batch, unbounded vs bound

Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?i.e., if I set one second as the frequency for batch processing, will it...

Data Engineering
Reply
  • 56 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jonxu
New Contributor II
  • 1 kudos

Many thanks for the clarification!

  • 1 kudos
1 More Replies
AlexSantiago
by • New Contributor II
  • 1895 Views
  • 6 replies
  • 1 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

Data Engineering
Reply
  • 1895 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
Scofeild618
Visitor
  • 1 kudos

When using the Spotify API and encountering authentication issues, particularly with token generation, it's crucial to follow the correct OAuth flow. Verify that your backend is properly set up for token handling and ensure that the raw input call ad...

  • 1 kudos
5 More Replies
rvo19941
by • New Contributor
  • 25 Views
  • 0 replies
  • 0 kudos

Auto Loader File Notification Mode not working with ADLS Gen2 and files written as a stream

Dear,I am working on a real-time use case and am therefore using Auto Loader with file notification to ingest json files from a Gen2 Azure Storage Account in real-time. Full refreshes of my table work fine but I noticed Auto Loader was not picking up...

Data Engineering
ADLS
Auto Loader
Event Subscription
File Notification
Queue Storage
Reply
  • 25 Views
  • 0 replies
  • 0 kudos
PremPrakash
by • Visitor
  • 55 Views
  • 2 replies
  • 1 kudos

Resolved! Using instance profile for sns message publish with PassRole

Hi, I want to attach instance profile to compute and  publish message on SNS without using credentials. Is that possible? has anyone used it. Will Boto3 support it?

Data Engineering
Reply
  • 55 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
PremPrakash
Visitor
  • 1 kudos

Yes, I have tried it, it is working. 

  • 1 kudos
1 More Replies
thisisadarshsin
by • New Contributor II
  • 2165 Views
  • 5 replies
  • 0 kudos

Permission issue in Fundamentals of the Databricks Lakehouse Platform Quiz

Hi ,I am getting this Error,when i am trying to give the exam ofFundamentals of the Databricks Lakehouse Platform.403FORBIDDENYou don't have permission to access this page2023-05-20 12:37:41 | Error 403 | https://customer-academy.databricks.com/I al...

Data Engineering
Reply
  • 2165 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
shubham5
New Contributor II
  • 0 kudos

Hi Team,Even I'm also getting the same error last 2 days while Taking the quiz and get your badge for Lakehouse Fundamentals.Error : You are not authorized to access https://customer-academy.databricks.com. Please select a platform you can access fro...

  • 0 kudos
4 More Replies
jorgemarmol
by • New Contributor II
  • 2448 Views
  • 5 replies
  • 0 kudos

Delta Live Tables: Too much time to do the "setting up"

Hello community!Recently I have been working in delta live table for a big project. My team and me have been studying so much and finally we have built a good pipeline with CDC that load 608 entities (and, therefore, 608 delta live tables and 608 mat...

jorgemarmol_0-1688633577282.png
Data Engineering
Reply
  • 2448 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
charl-p-botha
New Contributor II
  • 0 kudos

Same issue here. Medallion architecture in a test setup with 150 tables. SETTING_UP_TABLES takes 11 to 15 minutes, everything else together takes 10 minutes to do the incremental update (40K bronze rows, these fan out to asset tables, not much data!)...

  • 0 kudos
4 More Replies
Brad
by • Contributor
  • 157 Views
  • 1 replies
  • 1 kudos

Resolved! How to disable all cache

Hi, I'm trying to test some SQL perf. I run below firstspark.conf.set('spark.databricks.io.cache.enabled', False) However, the 2nd run for the same query is still way faster than the first time run. Is there a way to make the query start from a clean...

Data Engineering
Reply
  • 157 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

Hi @Brad , It is not clear which cache storage is helping with running your query faster, so the most straightforward way is to reset the sparkContext. Alternatively, these are the three clear cache ways I can think from the top of my head: // Clear ...

  • 1 kudos
stiaangerber
by • New Contributor III
  • 168 Views
  • 1 replies
  • 0 kudos

Simba ODBC for ARM-based Linux

HiIs there an ARM build of the Simba ODBC driver available for Linux? I've seen this thread (for Mac)https://community.databricks.com/t5/data-engineering/problems-connecting-simba-odbc-with-a-m1-macbook-pro/td-p/20566but it seems that there are only ...

Data Engineering
Reply
  • 168 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@stiaangerber unfortunately not. The odbc driver for linux is technically CPU agnostic, as long as it's amd64/x86_64, it should work, but we don't have one for ARM.

  • 0 kudos
sandeephenkel23
by • New Contributor III
  • 90 Views
  • 3 replies
  • 0 kudos

QuantileDiscretizer is not whiteliested erro!!

Dear Team,We observed that while attempting to use the following import:from pyspark.sql import functions as Ffrom pyspark.ml.feature import QuantileDiscretizerwe are encountering the following error:Py4JSecurityException: QuantileDiscretizer is not ...

Data Engineering
Reply
  • 90 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a...

  • 0 kudos
2 More Replies
DRock
by • New Contributor II
  • 200 Views
  • 5 replies
  • 0 kudos

Resolved! ODBC data source to connect to a Databricks catalog.database via MS Access Not Working

When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...

Data Engineering
Reply
  • 200 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Right, that indeed seems to be the problem, ideally MS Access should be fetching the default correct values, but only newer clients are. Instead of our original approach, could you please: 1.- Open the ODBC Driver DSN Setup. 2.- Go to Advanced Option...

  • 0 kudos
4 More Replies
jeft
by • Visitor
  • 38 Views
  • 1 replies
  • 0 kudos

mongodb ingest data into databricks error

spark = SparkSession.builder \.appName("MongoDBToDatabricks") \.config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:10.4.0") \.config("spark.mongodb.read.connection.uri", mongodb_uri) \.config("spark.mongodb.write.connection.u...

Data Engineering
Reply
  • 38 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jeft
Visitor
  • 0 kudos

 it shows maximun number of retires has been exceeded. i have no idea why it couldnt connect as i able to telnet to mongoDB

  • 0 kudos
RoelofvS
by • New Contributor II
  • 223 Views
  • 5 replies
  • 0 kudos

Schema evolution in Autoloader not evolving beyond version 0

I am working through the current version of the standard AutoLoader demo, i.e.    dbdemos.install('auto-loader')I.e. data gets read into a dataframe, but never written to a target table.Notebook is "01-Auto-loader-schema-evolution-Ingestion"Compute i...

Data Engineering
Reply
  • 223 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
RoelofvS
New Contributor II
  • 0 kudos

Hello @Brahmareddy,I have tried the above, without success.> enable detailed logging to trace schema evolution stepsPlease can you giude me with the steps or a URL? We are on AWS.Kind regards - Roelof 

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels