- 1643 Views
- 2 replies
- 0 kudos
Python coding in notebook with a (long) token
I have written a python program (called by a trigger) that uses a token issued by a third party app (it's circa 400 bytes long including '.' and '-'). When I copy/paste this token into a Databricks notebook - curious formatting takes place and a coup...
- 1643 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Paul, You can use databricks secrets for preserving the integrity of the token.Here's the databricks doc for refernece : https://docs.databricks.com/aws/en/security/secrets
- 0 kudos
- 4454 Views
- 3 replies
- 0 kudos
Save output of show table extended to table?
I want to save the output of show table extended in catalogName like 'mysearchtext*';to a table.How do I do that?
- 4454 Views
- 3 replies
- 0 kudos
- 0 kudos
Use DESCRIBE EXTENDED customer AS JSON this returns as a json data . This you can load Applicable to databricks 16.2 and abovehttps://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-describe-table
- 0 kudos
- 5335 Views
- 2 replies
- 1 kudos
Missing Genie - Upload File Feature in Preview Section
Despite having admin privileges for both the workspace and Genie Workspace, we are unable to see the "Genie - Upload File" feature under the Preview section, even though the documentation indicates it should be available.We also attempted switching r...
- 5335 Views
- 2 replies
- 1 kudos
- 1 kudos
For more information around upload a file option please refer https://docs.databricks.com/aws/en/genie/file-uploadit supports csv and excel datasets as of now with condition that files must be smaller than 200 MB and contain fewer than 100 columns du...
- 1 kudos
- 1976 Views
- 4 replies
- 4 kudos
Resolved! using Azure Databricks vs using Databricks directly
Hi friends,A quick question regarding how data, workspace controls works while using "Azure Databricks". I am planning to use Azure Databricks that comes as part of my employer's Azure Subscriptions. I work for a Public sector organization, which is ...
- 1976 Views
- 4 replies
- 4 kudos
- 4769 Views
- 0 replies
- 0 kudos
Support for managed identity based authentication in python kafka client
We followed this document https://docs.databricks.com/aws/en/connect/streaming/kafka?language=Python#msk-aad to use Kafka client to read events from our event hub for a feature.As part of the SFI, the guidance is to move away from client secret and u...
- 4769 Views
- 0 replies
- 0 kudos
- 563 Views
- 1 replies
- 0 kudos
Right course for ML engineer
Hi I would like to learn databricks so that I could look for job opportunities as a ML engineer. I have background with python programming, computer vision (OpenCV) .not having much of experience with azure , aws so on.which course here is good with ...
- 563 Views
- 1 replies
- 0 kudos
- 0 kudos
Given your background in Python programming and computer vision but limited experience with cloud platforms, the best pathway to enter the job market as MLE using Databricks is to pursue the Databricks Certified Machine Learning Associate certificati...
- 0 kudos
- 3363 Views
- 1 replies
- 1 kudos
Resolved! Why does .collect() cause a shuffle while .show() does not?
I’m learning Spark using the book Spark: The Definitive Guide and came across some behavior I’m trying to understand.I am reading a csv_file which has 3 columns: DEST_COUNTRY_NAME, ORIGIN_COUNTRY_NAME, count. The dataset has a total of 256 rows.Here’...
- 3363 Views
- 1 replies
- 1 kudos
- 1 kudos
Q1: collect() moves all data to the driver, hence a shufle. show() just shows x records from the df, from a partition (or more partitions if x > partition size). No shuffling needed.For display purposes the results are of course gathered on the driv...
- 1 kudos
- 1827 Views
- 2 replies
- 2 kudos
Lazy evaluation in serverless vs all purpose compute ?
As you can see right now I am connected to serverless compute and when I give wrong path, spark does lazy evaluation and gives error on display. However, when I switch from serverless to my all purpose cluster I get the error when I create the df its...
- 1827 Views
- 2 replies
- 2 kudos
- 2 kudos
Based on the scenario, what https://community.databricks.com/t5/user/viewprofilepage/user-id/156441 saying is correct though the eager evaluation property is false in both cases and for All-Purpose clusters, Spark is checking the path immediately whe...
- 2 kudos
- 756 Views
- 1 replies
- 0 kudos
Unable to access external table created by DLT
I originally set the Storage location in my DLT as abfss://{container}@{storageaccount}.dfs.core.windows.net/...But when running the DLT I got the following error:So I decided to leave the above Storage location blank and define the path parameter in...
- 756 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Tommy , Thanks for your question. I would encourage you to verify once using a Pro SQL Warehouse temporarily instead of a Serverless SQL Warehouse given the compute differences between the two - Pro compute resides in your data plane, Serverless ...
- 0 kudos
- 2202 Views
- 2 replies
- 2 kudos
OCRmyPDF in Databricks
Hello,Do any of you have experience with using OCRmyPDF in Databricks? I have tried to install it in various was with different versions, but my notebook keep crashing with the error:The Python process exited with exit code 139 (SIGSEGV: Segmentation...
- 2202 Views
- 2 replies
- 2 kudos
- 2 kudos
Refer to this link too https://community.databricks.com/t5/data-engineering/pdf-parsing-in-notebook/td-p/14636
- 2 kudos
- 3279 Views
- 2 replies
- 0 kudos
Can I automate notebook tagging based on workspace folder structure?
Hi all,I’m currently organizing a growing number of notebooks in our Databricks workspace and trying to keep things manageable with proper tagging and metadata. One idea I had was to automatically apply tags to notebooks based on their folder structu...
- 3279 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @EllaClark, Yes, you can automate tagging of Databricks notebooks based on folder structure using the REST API and a script. Use the Workspace API to list notebook paths, extract folder names, and treat them as tags.If the API supports metadata up...
- 0 kudos
- 870 Views
- 1 replies
- 1 kudos
Resolved! Simple notebook sync
Hi, is there a simple way to sync a local notebook with a Databricks notebook? For example, is it possible to just connect to the Databricks kernel or something similar?I know there are IDE extensions for this, but unfortunately, they use the local d...
- 870 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Kabi, as of my knowledge databricks doesn’t support directly connecting to Databricks kernel. However, here are practical ways to sync your local notebook with Databricks:You can use Git to version control your notebooks. Clone your repo into Dat...
- 1 kudos
- 537 Views
- 1 replies
- 1 kudos
Databricks Dashboard ,passing Prompt Values from one page to another
HI Guys,I have a dashboard with main page where I have a base query and added a date time range widget and linked it to filter the base query , Now I have a Page 2 where i use a a different sumamrized query as a source , base query2 . I need this qu...
- 537 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Mani2105, I guess currently, Databricks dashboards don’t support sharing widget parameters like date range filters across pages. Each page is isolated, so filters must be recreated manually per page. Manual configuration remains the only way to m...
- 1 kudos
- 1124 Views
- 2 replies
- 1 kudos
Asinine bad word detection
Are you kidding me here--I couldn't post this reply because (see arrows because I can't say the words)? I've run afoul of this several times before, bad word detection was a solved problem in the 1990s and there is even a term for errors like this--...
- 1124 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Rjdudley! Thank you for bringing this to our attention. We understand how frustrating it can be to have your message incorrectly flagged, especially when you're contributing meaningfully. While our filters are in place to maintain a safe space...
- 1 kudos
- 1020 Views
- 5 replies
- 1 kudos
AI/BI Dashboard - Hide Column in Table Visualization, but not in exported data
How can I hide specific colum from a table visualization, but not in the exported data.I have over 200 columns in my query result and the ui freeze while I want to show it in a table visualization. So I want to hide specific columns, but if I export ...
- 1020 Views
- 5 replies
- 1 kudos
