- 46307 Views
- 5 replies
- 6 kudos
How/where can I see a list of my dbfs files?
When using the Community Edition, I'm trying to find a place in the UI where I can browse the files that I've uploaded to dbfs. How/where can I do that? When I try to view them from the Data sidebar I see nothing, yet I know they're there, as if I us...
- 46307 Views
- 5 replies
- 6 kudos
- 6 kudos
1. Enable it from the settings->advanced-> dbfs browser 2. Refresh the page3. You can browse by clicking Catalog->Browse DBFS
- 6 kudos
- 475 Views
- 4 replies
- 2 kudos
DLT Apply Changes
Hi,In DLT, how do we specify which columns we don't want to overwrite when using the “apply changes” operation in the DLT (in the attached example, we want to avoid overwriting the “created_time” column)?I am using this sample code dlt.apply_changes(...
- 475 Views
- 4 replies
- 2 kudos
- 2 kudos
I am facing a similar issue. Is there any solution to select which columns to update when using "apply changes" operation?
- 2 kudos
- 295 Views
- 3 replies
- 0 kudos
Create a simple Geospatial Table with Geography type column
Hello AllI am looking for guidance on creating an simple US states table with shape as Geography column type. I do not want to use Apache Sedona (due to cluster limitations). I am going to create an Node JS application which is going to query this ge...
- 295 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you for prompt response. Yes, I have gone through the blog. I followed the same process ; for examples, buildings table have geometry which stores as binary type. My question, how to store geometry in 'Geography' type. The GEOGRAPHY data type ...
- 0 kudos
- 441 Views
- 5 replies
- 0 kudos
Can't pass dynamic parameters to non-notebook Python job (spark_python_task)
I need to access the date of a given job running as a non-notebook Python job (spark_python_task). I want to pass a value from the cli when running it and being available to access the value in the scriptI tried the approaches in the attached image w...
- 441 Views
- 5 replies
- 0 kudos
- 0 kudos
Can you confirm if this solution applies to you https://community.databricks.com/t5/data-engineering/retrieve-job-level-parameters-in-spark-python-task-not-notebooks/td-p/75324 ?
- 0 kudos
- 984 Views
- 1 replies
- 1 kudos
SQL Notebook Tab Spacing
My SQL notebooks in databricks suffer from having at least several different counts of spaces between tab marks. It makes it very difficult to maintain pretty code spacing. What sets the tab spacing in SQL language notebooks, and how is it set/adju...
- 984 Views
- 1 replies
- 1 kudos
- 1 kudos
I’m experiencing the same issue with SQL cell indentation in Databricks notebooks. While editing, I’ve noticed that the indentation level can vary from one cell to another - sometimes it’s two spaces, other times it’s four. This inconsistency makes i...
- 1 kudos
- 2108 Views
- 22 replies
- 26 kudos
Resolved! Run selected text shortcut not working
The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...
- 2108 Views
- 22 replies
- 26 kudos
- 26 kudos
Folks, could you please double-check now, issue should be fixed now. Thanks!
- 26 kudos
- 330 Views
- 2 replies
- 1 kudos
Resolved! Comparing two SQL notebooks from different Environments
Hello Everyone,I am part of data testing team which is responsible to verify data trends and insights generated from different sources. There are multiple schemas and tables in our platform. We use SQL queries in notebooks to verify all enrichment, m...
- 330 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you Walter. I did thought about it doing it one by one but then it was not coming out to be very efficient way. I then found a way to do it in Python via iterating through a dataframe of table names.
- 1 kudos
- 15576 Views
- 7 replies
- 6 kudos
'File not found' error when executing %run magic command
I'm just walking through a simple exercise presented in the Databricks Platform Lab notebook, in which I'm executing a remote notebook from within using the %run command. The remote notebook resides in the same directory as the Platform Lab notebook,...
- 15576 Views
- 7 replies
- 6 kudos
- 6 kudos
I have seen this error popping up when you define a Python file without the header: # Databricks notebook source Databricks has a hard time running the file as a nested notebook and you can get some weird errors.
- 6 kudos
- 368 Views
- 3 replies
- 0 kudos
What is "ExecuteGrpcResponseSender: Deadline reached, shutting down stream"
Hi, I have a delta table which is loaded by structured streaming job. When I tried to read this delta table and do a MERGE with foreachBatch, I found sometimes there is a big interval between streaming starts and MERGE starting to run and seems spar...
- 368 Views
- 3 replies
- 0 kudos
- 0 kudos
It may not necessarily be a bug, but some tuning due to architectural differences. What the message says is: The system was processing a gRPC operation identified by opId=5ef071b7-xxx, and it set a deadline for that operation (likely 120 seconds).The...
- 0 kudos
- 528 Views
- 1 replies
- 1 kudos
Creating a private connection with Google Sheets
How do I ingest sensitive data from a googlesheet doc to Databricks unity catalogue without making the googlesheet public.
- 528 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @dtb_usr,You can share the Google Sheet with the Service Account and use Google Sheets API ClientOpen the Google Sheet you want to access.Click on Share and add the email address of the service account (it will look something like your-service-acc...
- 1 kudos
- 265 Views
- 1 replies
- 0 kudos
Databricks cluster pools with init scripts
Ability to submit a single job with cluster pools and init scriptsfor the following payload: { "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": ...
- 265 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you still facing issues with the job run submit API endpoint?
- 0 kudos
- 347 Views
- 2 replies
- 1 kudos
Resolved! streaming vs batch, unbounded vs bound
Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?i.e., if I set one second as the frequency for batch processing, will it...
- 347 Views
- 2 replies
- 1 kudos
- 270 Views
- 2 replies
- 1 kudos
Resolved! Using instance profile for sns message publish with PassRole
Hi, I want to attach instance profile to compute and publish message on SNS without using credentials. Is that possible? has anyone used it. Will Boto3 support it?
- 270 Views
- 2 replies
- 1 kudos
- 265 Views
- 1 replies
- 0 kudos
Simba ODBC for ARM-based Linux
HiIs there an ARM build of the Simba ODBC driver available for Linux? I've seen this thread (for Mac)https://community.databricks.com/t5/data-engineering/problems-connecting-simba-odbc-with-a-m1-macbook-pro/td-p/20566but it seems that there are only ...
- 265 Views
- 1 replies
- 0 kudos
- 0 kudos
@stiaangerber unfortunately not. The odbc driver for linux is technically CPU agnostic, as long as it's amd64/x86_64, it should work, but we don't have one for ARM.
- 0 kudos
- 312 Views
- 3 replies
- 0 kudos
QuantileDiscretizer is not whiteliested erro!!
Dear Team,We observed that while attempting to use the following import:from pyspark.sql import functions as Ffrom pyspark.ml.feature import QuantileDiscretizerwe are encountering the following error:Py4JSecurityException: QuantileDiscretizer is not ...
- 312 Views
- 3 replies
- 0 kudos
- 0 kudos
@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a...
- 0 kudos