- 539 Views
- 5 replies
- 0 kudos
Can't pass dynamic parameters to non-notebook Python job (spark_python_task)
I need to access the date of a given job running as a non-notebook Python job (spark_python_task). I want to pass a value from the cli when running it and being available to access the value in the scriptI tried the approaches in the attached image w...
- 539 Views
- 5 replies
- 0 kudos
- 0 kudos
Can you confirm if this solution applies to you https://community.databricks.com/t5/data-engineering/retrieve-job-level-parameters-in-spark-python-task-not-notebooks/td-p/75324 ?
- 0 kudos
- 1372 Views
- 1 replies
- 1 kudos
SQL Notebook Tab Spacing
My SQL notebooks in databricks suffer from having at least several different counts of spaces between tab marks. It makes it very difficult to maintain pretty code spacing. What sets the tab spacing in SQL language notebooks, and how is it set/adju...
- 1372 Views
- 1 replies
- 1 kudos
- 1 kudos
I’m experiencing the same issue with SQL cell indentation in Databricks notebooks. While editing, I’ve noticed that the indentation level can vary from one cell to another - sometimes it’s two spaces, other times it’s four. This inconsistency makes i...
- 1 kudos
- 2616 Views
- 22 replies
- 26 kudos
Resolved! Run selected text shortcut not working
The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...
- 2616 Views
- 22 replies
- 26 kudos
- 26 kudos
Folks, could you please double-check now, issue should be fixed now. Thanks!
- 26 kudos
- 465 Views
- 2 replies
- 1 kudos
Resolved! Comparing two SQL notebooks from different Environments
Hello Everyone,I am part of data testing team which is responsible to verify data trends and insights generated from different sources. There are multiple schemas and tables in our platform. We use SQL queries in notebooks to verify all enrichment, m...
- 465 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you Walter. I did thought about it doing it one by one but then it was not coming out to be very efficient way. I then found a way to do it in Python via iterating through a dataframe of table names.
- 1 kudos
- 16280 Views
- 7 replies
- 6 kudos
'File not found' error when executing %run magic command
I'm just walking through a simple exercise presented in the Databricks Platform Lab notebook, in which I'm executing a remote notebook from within using the %run command. The remote notebook resides in the same directory as the Platform Lab notebook,...
- 16280 Views
- 7 replies
- 6 kudos
- 6 kudos
I have seen this error popping up when you define a Python file without the header: # Databricks notebook source Databricks has a hard time running the file as a nested notebook and you can get some weird errors.
- 6 kudos
- 417 Views
- 3 replies
- 0 kudos
What is "ExecuteGrpcResponseSender: Deadline reached, shutting down stream"
Hi, I have a delta table which is loaded by structured streaming job. When I tried to read this delta table and do a MERGE with foreachBatch, I found sometimes there is a big interval between streaming starts and MERGE starting to run and seems spar...
- 417 Views
- 3 replies
- 0 kudos
- 0 kudos
It may not necessarily be a bug, but some tuning due to architectural differences. What the message says is: The system was processing a gRPC operation identified by opId=5ef071b7-xxx, and it set a deadline for that operation (likely 120 seconds).The...
- 0 kudos
- 750 Views
- 1 replies
- 1 kudos
Creating a private connection with Google Sheets
How do I ingest sensitive data from a googlesheet doc to Databricks unity catalogue without making the googlesheet public.
- 750 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @dtb_usr,You can share the Google Sheet with the Service Account and use Google Sheets API ClientOpen the Google Sheet you want to access.Click on Share and add the email address of the service account (it will look something like your-service-acc...
- 1 kudos
- 328 Views
- 1 replies
- 0 kudos
Databricks cluster pools with init scripts
Ability to submit a single job with cluster pools and init scriptsfor the following payload: { "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": ...
- 328 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you still facing issues with the job run submit API endpoint?
- 0 kudos
- 442 Views
- 2 replies
- 1 kudos
Resolved! streaming vs batch, unbounded vs bound
Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?i.e., if I set one second as the frequency for batch processing, will it...
- 442 Views
- 2 replies
- 1 kudos
- 353 Views
- 2 replies
- 1 kudos
Resolved! Using instance profile for sns message publish with PassRole
Hi, I want to attach instance profile to compute and publish message on SNS without using credentials. Is that possible? has anyone used it. Will Boto3 support it?
- 353 Views
- 2 replies
- 1 kudos
- 297 Views
- 1 replies
- 0 kudos
Simba ODBC for ARM-based Linux
HiIs there an ARM build of the Simba ODBC driver available for Linux? I've seen this thread (for Mac)https://community.databricks.com/t5/data-engineering/problems-connecting-simba-odbc-with-a-m1-macbook-pro/td-p/20566but it seems that there are only ...
- 297 Views
- 1 replies
- 0 kudos
- 0 kudos
@stiaangerber unfortunately not. The odbc driver for linux is technically CPU agnostic, as long as it's amd64/x86_64, it should work, but we don't have one for ARM.
- 0 kudos
- 364 Views
- 3 replies
- 0 kudos
QuantileDiscretizer is not whiteliested erro!!
Dear Team,We observed that while attempting to use the following import:from pyspark.sql import functions as Ffrom pyspark.ml.feature import QuantileDiscretizerwe are encountering the following error:Py4JSecurityException: QuantileDiscretizer is not ...
- 364 Views
- 3 replies
- 0 kudos
- 0 kudos
@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a...
- 0 kudos
- 749 Views
- 5 replies
- 0 kudos
Resolved! ODBC data source to connect to a Databricks catalog.database via MS Access Not Working
When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...
- 749 Views
- 5 replies
- 0 kudos
- 0 kudos
Right, that indeed seems to be the problem, ideally MS Access should be fetching the default correct values, but only newer clients are. Instead of our original approach, could you please: 1.- Open the ODBC Driver DSN Setup. 2.- Go to Advanced Option...
- 0 kudos
- 653 Views
- 5 replies
- 0 kudos
Schema evolution in Autoloader not evolving beyond version 0
I am working through the current version of the standard AutoLoader demo, i.e. dbdemos.install('auto-loader')I.e. data gets read into a dataframe, but never written to a target table.Notebook is "01-Auto-loader-schema-evolution-Ingestion"Compute i...
- 653 Views
- 5 replies
- 0 kudos
- 0 kudos
Hello @Brahmareddy,I have tried the above, without success.> enable detailed logging to trace schema evolution stepsPlease can you giude me with the steps or a URL? We are on AWS.Kind regards - Roelof
- 0 kudos
- 516 Views
- 1 replies
- 1 kudos
Enhanced Cost Management for Serverless Compute
Budget policies include tags that are applied to serverless compute activities incurred by users assigned. These tags are recorded in your billing records, allowing you to allocate specific serverless usage to designated budgets. For more information...
- 516 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for sharing.
- 1 kudos