cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Phani1
by Valued Contributor II
  • 8720 Views
  • 4 replies
  • 2 kudos

Masking techniques for more PII columns

Hi Databricks Team,We would appreciate it if you could inform us about the situations when Column-Masking, Row-Level Filtering, and Attributed-Based Masking should be utilized, as well as the recommended technique for handling large data volumes cont...

  • 8720 Views
  • 4 replies
  • 2 kudos
Latest Reply
NandiniN
Databricks Employee
  • 2 kudos

Agree with @Meghla-C , https://databricks.aha.io/ideas/ideas/DB-I-7941 this was the feature request and if you see the status, it is in preview.

  • 2 kudos
3 More Replies
NC
by New Contributor III
  • 3301 Views
  • 4 replies
  • 0 kudos

GDAL on Databricks Cluster Runtime 12.2 LTS

I need gdal in my course work.After reading this post, I used init script as follows to install gdal into runtime 12.2 LTS  dbutils.fs.put("/databricks/scripts/gdal_install.sh",""" #!/bin/bash sudo add-apt-repository ppa:ubuntugis/ppa sudo apt-get up...

  • 3301 Views
  • 4 replies
  • 0 kudos
Latest Reply
Matt_C
New Contributor II
  • 0 kudos

Hi, in case anyone is still struggling here. I found I could not get the init script approach to work, but if I just run a shell command to install gdal at the start of my notebook it works fine. You might note, however, that this installs gdal versi...

  • 0 kudos
3 More Replies
juliemoore
by New Contributor
  • 534 Views
  • 1 replies
  • 0 kudos

Problems with Big Data Solutions and Databricks- Any advice?

Hello everyone,I am currently facing several challenges related to big data solutions, particularly with the Databricks. As many of you may know, Databricks is a powerful platform for data engineering and analytics, but I have encountered some signif...

  • 534 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Your problem statement is too generic. If your company is facing this, you can reach out to your SA; they will help you. If it's a personal project, then mention what you are trying in detail with cluster size, what you are trying to integrate with, ...

  • 0 kudos
JolM
by New Contributor II
  • 595 Views
  • 1 replies
  • 1 kudos

Resolved! is there a way for us to see billing usage per catalog?

is there a way for us to see billing usage per catalog? I'm using 14days trial period for now.. would it be available in Premium?

  • 595 Views
  • 1 replies
  • 1 kudos
Latest Reply
gchandra
Databricks Employee
  • 1 kudos

Billing system table provides cost by notebook, jobs, and clusters. If catalog-to-job/cluster/notebook relation is maintained, then catalog-based usage can be determined. https://docs.databricks.com/en/admin/system-tables/billing.html

  • 1 kudos
hetrasol
by New Contributor III
  • 2209 Views
  • 7 replies
  • 0 kudos

Resolved! Unable to start browser for databricks certification

Hello, I  have registered for databricks certified data engineering associate exam. One of the requirements to give the exam is The exam is set for Sunday 6th October, 2024 but the browser installation (psi secure bridge browser) does not work. .Reac...

hetrasol_0-1728079492073.png
  • 2209 Views
  • 7 replies
  • 0 kudos
Latest Reply
TaiNguyen
New Contributor II
  • 0 kudos

Hi @hetrasol ,I'm a Windows user. After installation, I just got the Lockdown Browser OEM instead of the PSI browser, as you mentioned above. Can you help to instruct again on how to install these browsers

  • 0 kudos
6 More Replies
ff-paulo-barbos
by New Contributor
  • 1856 Views
  • 2 replies
  • 0 kudos

Spark Remote error when connecting to cluster

Hi, I am using the latest version of pyspark and I am trying to connect to a remote cluster with runtime 13.3.My doubts are:- Do i need databricks unity catalog enabled?- My cluster is already in a Shared policy in Access Mode, so what other configur...

  • 1856 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Is your workspace is already unity catalog enabled? Also, did you go through the considerations for enabling workspace for unity catalog? https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html#considerations-before-yo...

  • 0 kudos
1 More Replies
Thms317
by New Contributor III
  • 1442 Views
  • 2 replies
  • 2 kudos

Resolved! Cannot install wheel from Workspace in DLT

Hi all. I am no longer able to install my custom wheel in my DLT pipeline. No matter what configuration I try I cannot get it to work: parameterized or just hard-coding the path to the wheel. If I run the hard-coded cell with an all-purpose cluster t...

  • 1442 Views
  • 2 replies
  • 2 kudos
Latest Reply
Thms317
New Contributor III
  • 2 kudos

I managed to fix the issue. The problem was that my wheel was built for Databricks Runtime 14.3 LTS and I was using the PREVIEW channel rather than the CURRENT channel. At time of writing:CURRENT(default): Databricks Runtime 14.1 --> Python: 3.10.12P...

  • 2 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 382 Views
  • 1 replies
  • 0 kudos

Late file arrivals - Autoloader

 Hi All,I have a situation where I'm receiving various CSV files in a storage location.The issue I'm facing is that I'm using Databricks Autoloader, but some files might arrive later than expected. In this case, we need to notify the relevant team ab...

  • 382 Views
  • 1 replies
  • 0 kudos
Latest Reply
HaggMan
New Contributor III
  • 0 kudos

Well, Autoloader could work nicely with the notification event for arriving files. You could probably specify a window duration for your "on-time" arrivels and that could be your base check for on time. As files arrive they go to their window and whe...

  • 0 kudos
carolpeixinho
by New Contributor
  • 300 Views
  • 1 replies
  • 0 kudos

Sharing Opportunities with Databricks

Hi everyone,I would like to talk to someone that could set up a process of deals sharing with Databricks following the GDPR.Thanks,Carol.

  • 300 Views
  • 1 replies
  • 0 kudos
Latest Reply
SathyaSDE
Contributor
  • 0 kudos

Hi,Can you pls share some more details on what you are looking for ?If you are trying to share the data to/from Databricks, you can use Delta sharing , Clean rooms option - these provide data sharing options with strong security & governance.or if yo...

  • 0 kudos
Zoraida
by New Contributor
  • 421 Views
  • 1 replies
  • 0 kudos

Databricks Destiny with Fivetrans best practices

Hello! we are trying to use Fivetran for ingesting different sources into the data lake so we will have multiple connectors. We would like to know what are the recommendations when selecting the SQL warehouses. Since the new serverless SQL warehouses...

  • 421 Views
  • 1 replies
  • 0 kudos
Latest Reply
SathyaSDE
Contributor
  • 0 kudos

Hi,To understand about the Databricks SQL Serverless cost, you can see here - https://www.databricks.com/product/pricing/databricks-sqlIn terms of comparison, Databricks is said to be the most cost efficient & high performant in the market amongst it...

  • 0 kudos
Phani1
by Valued Contributor II
  • 297 Views
  • 1 replies
  • 0 kudos

code vulnerabilities, code smells, and bugs

Hi Team, is there a way in Databricks to check for code vulnerabilities, code smells, and bugs?Note :Databricks native functionality only  

  • 297 Views
  • 1 replies
  • 0 kudos
Latest Reply
SathyaSDE
Contributor
  • 0 kudos

Hi,As far as I am aware, for security scanning/monitoring at Databricks account level, we have belowSAT - https://github.com/databricks-industry-solutions/security-analysis-toolhttps://www.databricks.com/trust/trusthttps://learn.microsoft.com/en-us/a...

  • 0 kudos
Phani1
by Valued Contributor II
  • 275 Views
  • 1 replies
  • 0 kudos

CDC for Unstructured data

Hi All,how we can handle CDC for unstructured data in Databricks. What are some best practices we should follow to make this work effectively?Regards,Phani

  • 275 Views
  • 1 replies
  • 0 kudos
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @Phani1 ,Handling CDC for unstructured data—such as audio, images, or video files—in Databricks involves efficiently detecting and processing changes to these files as they occur.Here's how you can approach this:Use Databricks Autoloader: Autoload...

  • 0 kudos
mathijs-fish
by New Contributor III
  • 3576 Views
  • 5 replies
  • 3 kudos

Resolved! "with open" not working with Shared Access Cluster on mounted location

Hi All,For an application that we are building, we need a encoding detector/utf-8 enforcer. For this, we used the python library chardet in combination with "with open". We open a file from a mounted adls location (we use a legacy hive-metastore)When...

mathijsfish_1-1701785425743.png mathijsfish_2-1701785466668.png
Get Started Discussions
glob
Mount
os
with open
  • 3576 Views
  • 5 replies
  • 3 kudos
Latest Reply
nagND
New Contributor II
  • 3 kudos

Hi @mathijs-fish @Ayushi_Suthar  - I am having the same issue with shared cluster. I can see the list of PDF files on the mount using dbutils.fs.ls(mount_point), but when I am trying to read the PDF files using PyPDF, I am getting - FileNotFoundError...

  • 3 kudos
4 More Replies
brandocomando8
by New Contributor III
  • 2180 Views
  • 15 replies
  • 1 kudos

Sql Error on MultiNode cluster, but fine on SingleNode

If I run the following code on a cluster in SingleNode mode it works fine, but if I run the exact same cell on a MultiNode Cluster It throws:SparkConnectGrpcException: (java.sql.SQLTransientConnectionException) Could not connect to address=(host=HOST...

  • 2180 Views
  • 15 replies
  • 1 kudos
Latest Reply
gchandra
Databricks Employee
  • 1 kudos

data_security_mode": "NONE":  This is a non-Unity Catalog Cluster. No Governance enforced. "data_security_mode": "USER_ISOLATION": This is a UC Shared Compute cluster that has certain limitations when accessing Low-Level APIs, RDDs, and dbfs/data bri...

  • 1 kudos
14 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors