cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

102295members
52610posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Registration now open! Databricks Data + AI Summit 2024

Join tens of thousands of data leaders, engineers, scientists and architects from around the world at Moscone Center in San Francisco, June 10–13.  Explore the latest advances in Apache Spark™, Delta Lake, MLflow, LangChain, PyTorch, dbt, Prest...

  • 7055 Views
  • 1 replies
  • 4 kudos
02-12-2024
Meet DBRX, the New Standard for High-Quality LLMs

Get your first look at DBRX April 25, 2024 | 8 AM PT If you’re using off-the-shelf LLMs to build GenAI applications, you’re probably struggling with quality, privacy and governance issues. What you need is a way to cost-effectively build a custom LLM...

  • 2212 Views
  • 3 replies
  • 2 kudos
2 weeks ago
Meet the Community Team Virtually!

Prepare to enhance your socializing adventure! Date: April 18, 2024 Time: 9:00 - 9:30 AM IST  Location: Virtual Event (Link provided upon registration) What's in Store for You?  Exciting Icebreaker Activities Engaging Discussions Networking Oppo...

  • 5560 Views
  • 5 replies
  • 1 kudos
3 weeks ago
Data Warehousing in the Era of AI

AI has the power to address the data warehouse’s biggest challenges — performance, governance and usability — thanks to its deeper understanding of your data and how it’s used. This is data intelligence and it’s revolutionizing the way you query, man...

  • 2862 Views
  • 5 replies
  • 1 kudos
2 weeks ago

Community Activity

cosminsanda
by New Contributor II
  • 2 Views
  • 0 replies
  • 0 kudos

Unit Testing with the new Databricks Connect in Python

I would like to create a regular PySpark session in an isolated environment against which I can run my Spark based tests. I don't see how that's possible with the new Databricks Connect. I'm going in circles here, is it even possible?I don't want to ...

  • 2 Views
  • 0 replies
  • 0 kudos
Hogan
by Visitor
  • 56 Views
  • 1 replies
  • 0 kudos

Can browse external Storage, but can not create a Table from there - VNET, ADLSGen2

Hi there!Hope somebody here can help me. We have created a new Databricks Account on Azure with the ARM template for VNET injection.We have all the subnets etc., unitiy catalog active and the connector for databricks.I want now to create my first tab...

  • 56 Views
  • 1 replies
  • 0 kudos
Latest Reply
Hogan
Visitor
  • 0 kudos

Hi,To solve this problem, the following Microsoft documentation can be used to configure the NCC to enable the connection between the private Azure storage and the serverless resources.https://learn.microsoft.com/en-us/azure/databricks/security/netwo...

  • 0 kudos
sai_sathya
by New Contributor III
  • 110 Views
  • 6 replies
  • 1 kudos

DataFrame to CSV write has issues due to multiple commas inside an row value

Hi alliam working on a data containing JSON fields with embedded commas into CSV format. iam facing challenges due to the commas within the JSON being misinterpreted as column delimiters during the conversion process.i tried several methods to modify...

sai_sathya_0-1712850570456.png sai_sathya_1-1712850991923.png
  • 110 Views
  • 6 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hi Sai, I assume that the problem comes not from the PySpark, but from Excel. I tried to reproduce the error and didn't find the way - that a good thing, right ? Please try the following :    df.write.format("csv").save("/Volumes/<my_catalog_name>/<m...

  • 1 kudos
5 More Replies
Shreyash
by Visitor
  • 6 Views
  • 0 replies
  • 0 kudos

java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.DocumentAssembler

I am trying to serve a pyspark model using an endpoint. I was able to load and register the model normally. I could also load that model and perform inference but while serving the model, I am getting the following error: [94fffqts54] ERROR StatusLog...

Machine Learning
Model serving
sparknlp
  • 6 Views
  • 0 replies
  • 0 kudos
Kayla
by Contributor
  • 9 Views
  • 0 replies
  • 0 kudos

Errors When Using R on Unity Catalog Clusters

We are running into errors when running workflows with multiple jobs using the same notebook/different parameters. They are reading from tables we still have in hive_metastore, there's no Unity Catalog tables or functionality referenced anywhere. We'...

  • 9 Views
  • 0 replies
  • 0 kudos
JameDavi_51481
by New Contributor III
  • 1999 Views
  • 3 replies
  • 0 kudos

Can we add tags to Unity Catalog through Terraform?

We use Terraform to manage most of our infrastructure, and I would like to extend this to Unity Catalog. However, we are extensive users of tagging to categorize our datasets, and the only programmatic method I can find for adding tags is to use SQL ...

  • 1999 Views
  • 3 replies
  • 0 kudos
Latest Reply
TMD
New Contributor III
  • 0 kudos

Hello,Is there a plan for the Databricks TF provider support tagging in the near future?Thanks.

  • 0 kudos
2 More Replies
Nithya_r
by New Contributor II
  • 66 Views
  • 1 replies
  • 0 kudos

Access Delta sharing from Azure Data Factory

I recently got access to delta sharing and I am looking to access the data from the tables in share through ADF. I used linked services such as REST API and HTTP and successfully established connection using the credential file token and http path, h...

  • 66 Views
  • 1 replies
  • 0 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 0 kudos

Hey, I think you'll need to use a Databricks activity instead of Copy See : https://learn.microsoft.com/en-us/azure/data-factory/connector-overview#integrate-with-more-data-storeshttps://learn.microsoft.com/en-us/azure/data-factory/transform-data-dat...

  • 0 kudos
dilkushpatel
by Visitor
  • 21 Views
  • 0 replies
  • 0 kudos

Databricks connecting SQL Azure DW - Confused between Polybase and Copy Into

I see two articles on databricks documentationshttps://docs.databricks.com/en/archive/azure/synapse-polybase.html#language-pythonhttps://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#service-principal Polybase one is legacy o...

Data Engineering
azure
Copy
help
Polybase
Synapse
  • 21 Views
  • 0 replies
  • 0 kudos
phguk
by New Contributor II
  • 21 Views
  • 0 replies
  • 0 kudos

Why does use of Azure SSO require Databricks PAT enabled ?

My org uses Databricks and SSO. We are keen to disable the use of PAT but have noticed that when it's disabled, we're not able to use SSO. May I ask why does SSO have a dependency on PATs [arguably they are two distinct authentication methods] ?Also,...

  • 21 Views
  • 0 replies
  • 0 kudos
phguk
by New Contributor II
  • 1128 Views
  • 2 replies
  • 0 kudos

Adding NFS storage as external volume (Unity)

Can anyone share experience (or point me to another reference) that describes how to configure Azure Blob storage which has NFS enabled as an external volume to Databricks ?I've succeeded in adding SMB storage to Databricks but (if I understand prope...

  • 1128 Views
  • 2 replies
  • 0 kudos
Latest Reply
phguk
New Contributor II
  • 0 kudos

Apologies for the delay & many thanks for responding. Yes I've been able to mount my premium storage + NFS container as an external volume to Databricks.

  • 0 kudos
1 More Replies
Dp15
by New Contributor III
  • 206 Views
  • 2 replies
  • 1 kudos

Using UDF in an insert command

Hi,I am trying to use a UDF to get the last day of the month and use the boolean result of the function in an insert command. Please find herewith the function and the my query.function:import calendarfrom datetime import datetime, date, timedeltadef...

  • 206 Views
  • 2 replies
  • 1 kudos
Latest Reply
Dp15
New Contributor III
  • 1 kudos

Thank you @Kaniz for your detailed explanation

  • 1 kudos
1 More Replies
databird
by New Contributor II
  • 703 Views
  • 4 replies
  • 1 kudos

Redefine ETL strategy with pypskar approach

Hey everyone!I've some previous experience with Data Engineering, but totally new in Databricks and Delta Tables.Starting this thread hoping to ask some questions and asking for help on how to design a process.So I have essentially 2 delta tables (sa...

  • 703 Views
  • 4 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hi @databird , You can review the code of each demo by opening the content via "View the Notebooks" or by exploring the following repo : https://github.com/databricks-demos (you can try to search for "merge" to see all the occurrences, for example) T...

  • 1 kudos
3 More Replies
MathewDRitch
by Visitor
  • 57 Views
  • 2 replies
  • 0 kudos

Connecting from Databricks to Network Path

Hi All,Will appreciate if someone can help me with some references links on connecting from Databricks to external network path. I have Databricks on AWS and previously used to connect to files on external network path using Mount method. Now Databri...

  • 57 Views
  • 2 replies
  • 0 kudos
Latest Reply
MathewDRitch
  • 0 kudos

Currently we connect to the cloud storages as external storages using unity catalog. We have not yet connected to the on-premise network storages, which is currently the solution we are looking for. 

  • 0 kudos
1 More Replies
data-grassroots
by Visitor
  • 28 Views
  • 0 replies
  • 0 kudos

Ingesting Files - Same file name, modified content

We have a data feed with files whose filenames stays the same but the contents change over time (brand_a.csv, brand_b.csv, brand_c.csv ....).Copy Into seems to ignore the files when they change.If we set the Force flag to true and run it, we end up w...

  • 28 Views
  • 0 replies
  • 0 kudos
CDICSteph
by New Contributor
  • 665 Views
  • 3 replies
  • 0 kudos

permission denied listing external volume when using vscode databricks extension

hey, i'm using the Db extension for vscode (Databricks connect v2). When using dbutils to list an external volume defined in UC like so:   dbutils.fs.ls("/Volumes/dev/bronze/rawdatafiles/") i get this error: "databricks.sdk.errors.mapping.PermissionD...

  • 665 Views
  • 3 replies
  • 0 kudos
Latest Reply
lukasjh
Visitor
  • 0 kudos

We still face the problem (UC enabled shared cluster). Is there any resolution? @Kaniz  

  • 0 kudos
2 More Replies

Latest from our Blog

Attributing Costs in Databricks Model Serving

Databricks Model Serving provides a scalable, low-latency hosting service for AI models. It supports models ranging from small custom models to best-in-class large language models (LLMs). In this blog...

2362Views 1kudos

MLOps Gym - Unity Catalog Setup for MLOps

Unity Catalog (UC) is Databricks unified governance solution for all data and AI assets on the Data Intelligence Platform. UC is central to implementing MLOps on Databricks as it is where all your as...

2691Views 0kudos

Highly selective: SQL refined beyond the WHERE

Inuktitut, the language of the Inuit, has 50 words for snow and ice. That’s - as they say - fake news, but the point made is metaphorical: When something is important to a people, their language finds...

3322Views 3kudos