cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

104432members
55384posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Get Certified at Data & AI Summit and Earn this Exclusive Databricks Jacket

Agenda at a glance Join us for four days of deep learning around data, AI and LLM technologies. Monday 9:00 AM - 1:00 PM Training and Certification Dive deep into specific topics like data lakehouse architecture, Databricks SQL, MLflow, LLMs and mor...

  • 205 Views
  • 1 replies
  • 1 kudos
yesterday
Introducing the Databricks AI Fund

Databricks Ventures Launches New Fund to Extend our Ecosystem Leadership We launched Databricks Ventures in December 2021 as our strategic investment arm for funding innovative startups across the data, analytics and AI landscape — companies that sha...

  • 753 Views
  • 0 replies
  • 0 kudos
Wednesday
Announcing General Availability of Liquid Clustering

Out-of-the-box, self-tuning data layout that scales with your data We’re excited to announce the General Availability of Delta Lake Liquid Clustering in the Databricks Data Intelligence Platform. Liquid Clustering is an innovative data management tec...

  • 687 Views
  • 0 replies
  • 1 kudos
Wednesday
Supercharge Your Code Generation

We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing personalized code suggestions as you type. Directly integrated into the notebook and SQL ...

  • 963 Views
  • 1 replies
  • 2 kudos
a week ago

Community Activity

Nicu_Bordea
by New Contributor
  • 4 Views
  • 0 replies
  • 0 kudos

Create a community user group Romania

Hello,I would like to receive some support in creating a Community User Group in Romania, Cluj-Napoca. Our intention as a company is to become a partner and us to create a community: events, meetups so on.Looking for your help.Thank youNicu

  • 4 Views
  • 0 replies
  • 0 kudos
Christine
by Contributor
  • 37214 Views
  • 15 replies
  • 15 kudos

Resolved! Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster

I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...

  • 37214 Views
  • 15 replies
  • 15 kudos
Latest Reply
him_agg
Visitor
  • 15 kudos

I was having a similar issue in using .rdd.map()Solved it by adding two key value pairs in the spark config for the clusterspark.databricks.pyspark.enablePy4JSecurity falsespark.databricks.pyspark.trustedFilesystems org.apache.spark.api.java.JavaRDD ...

  • 15 kudos
14 More Replies
rashmiraj
by Visitor
  • 24 Views
  • 1 replies
  • 0 kudos

DataBricks Certification Exam Got Suspended. Require Immediate support

Hi Team,Today (29th May 2024), I began my Databricks assessment exam, but it was abruptly suspended by the proctor without any explanation. This was my first exam and it has been a disappointing experience.I started the exam calmly, but the proctor w...

  • 24 Views
  • 1 replies
  • 0 kudos
Latest Reply
Cert-Team
Honored Contributor III
  • 0 kudos

@rashmiraj I'm sorry to hear your exam was suspended. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours for a resolution. In the meantime, you can review the following documentation:Room requirementsBehavi...

  • 0 kudos
leolmz
by Visitor
  • 11 Views
  • 0 replies
  • 0 kudos

Download model artifact with CURL

Hi, I want to download a registered model by some downstream client and use a CURL or HTTP request method to do that (REST API seems to have no corresponding method for doing this). Will setting up an artifact store in S3 help?I am assuming for examp...

  • 11 Views
  • 0 replies
  • 0 kudos
Mits
by New Contributor II
  • 1230 Views
  • 4 replies
  • 3 kudos

Sending email alerts to non-databricks user

I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...

  • 1230 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Mitali Lad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 3 kudos
3 More Replies
Phani1
by Valued Contributor
  • 34 Views
  • 1 replies
  • 0 kudos

integrating Azure Databricks with AAD

Hi Team, Could you please provide the details/process for integrating Azure Databricks - Unity Catalog and AAD? Regards,Phani

  • 34 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Contributor III
  • 0 kudos

Hello @Phani1 ,These doc pages might be useful for you: Set up and manage Unity CatalogSync users and groups from Microsoft Entra ID 

  • 0 kudos
ismaelhenzel
by New Contributor III
  • 40 Views
  • 1 replies
  • 0 kudos

Upsert into a Delta Lake table with merge when using row masking function

I'm using databricks rls functions on my tables, and i need to make some merges into, but tables with rls functions does not support merge operations (https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#limitation...

ismaelhenzel_0-1716979371091.png
  • 40 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Contributor III
  • 0 kudos

Hi @ismaelhenzel, if you want to use the "MERGE INTO" sql command, you must turn-off rls. This is by design.

  • 0 kudos
SamarthJain
by New Contributor II
  • 3259 Views
  • 5 replies
  • 2 kudos

Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your...

Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your help here to understand what happens internally at the "Stream Initializing" phase of the Spark Streaming job tha...

  • 3259 Views
  • 5 replies
  • 2 kudos
Latest Reply
olivier_soucy
  • 2 kudos

I also had the same issue, but it seems only happening on DBR >= 15.0. Any idea why?

  • 2 kudos
4 More Replies
Mathias
by New Contributor II
  • 34 Views
  • 1 replies
  • 0 kudos

Delay rows coming into DLT pipeline

Backgroundand requirements: We are reading data from our factory and storing it in a DLT table called telemetry with columns sensorid, timestamp and value. We need to get rows where sensorid is “qrreader-x” and join with some other data from that sam...

  • 34 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Contributor III
  • 0 kudos

Hi @Mathias,  I'd say that watermarking might be a good solution for your use case. Please check Control late data threshold with multiple watermark policy in Structured Streaming.  If you want to dig-in further there's also: Spark Structured Streami...

  • 0 kudos
andre_rizzatti
by New Contributor II
  • 34 Views
  • 1 replies
  • 0 kudos

SQLWarehouse Case INsensitive

Good morning, is there any parameter or configuration that causes all my data to be consulted without case distinction? insensitive?

  • 34 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Contributor III
  • 0 kudos

Hello @andre_rizzatti,  At the current moment SQL Warehouses don't have this option. This feature is coming soon.  

  • 0 kudos
TWib
by New Contributor III
  • 1114 Views
  • 6 replies
  • 1 kudos

DatabricksSession broken for 15.1

This code fails with exception:[NOT_COLUMN_OR_STR] Argument `col` should be a Column or str, got Column.File <command-4420517954891674>, line 7 4 spark = DatabricksSession.builder.getOrCreate() 6 df = spark.read.table("samples.nyctaxi.trips") ---->...

  • 1114 Views
  • 6 replies
  • 1 kudos
Latest Reply
zerodarkzone
New Contributor II
  • 1 kudos

Hi, I'm having the same problem using the 14.3LTS runtime.The error just appeared yesterday. Before that, everything was working fine.

  • 1 kudos
5 More Replies
thiagoawstest
by New Contributor II
  • 35 Views
  • 0 replies
  • 0 kudos

migration Azure to AWS

Hello, I need to migrate from Databricks Azure to AWS, using tool-databricks-migration generates many errors, if I do it manually using databeicks-cli, what would be the best practice?Any tips, for example:-first migrate notebooks-second jobs-third u...

Data Engineering
AWS
migration
  • 35 Views
  • 0 replies
  • 0 kudos
Jinfei
by Contributor II
  • 3762 Views
  • 3 replies
  • 14 kudos

Apache Spark Programming course - can we enter full screen?

Not sure if this is the best place to provide course feedback, but I really want to give suggestions on the Apache Spark Programming course.I notice Apache Spark Programming course is a little different from others. It seems more advanced, but I can'...

image
  • 3762 Views
  • 3 replies
  • 14 kudos
Latest Reply
FeliciaWilliam
Contributor
  • 14 kudos

Very useful and informative Thanks to everyone

  • 14 kudos
2 More Replies
Sujitha
by Community Manager
  • 42137 Views
  • 53 replies
  • 111 kudos

🔔 Attention Databricks Academy Users: SSO Implementation Incoming! Secure Your Account Today!

On May 18th a new SSO user experience will launch in Customer and Partner Academy. In order to secure your account, please see the instructions below on how to update and verify that your data is correct and complete. Please ensure that you have two ...

Sujitha_0-1715596765217.png Sujitha_1-1715596782895.png Screenshot 2024-05-06 at 11.56.58 AM.png
  • 42137 Views
  • 53 replies
  • 111 kudos
Latest Reply
leocarvalho
  • 111 kudos

I can not access my account since the login change.How could I set my personal email?Are there another way to log in and set the personal email?Today I've open the second ticket to try solve my problem.The first one, stayed open for more than one wee...

  • 111 kudos
52 More Replies
aarshps
by Visitor
  • 28 Views
  • 0 replies
  • 0 kudos

Microsoft PNP Azure Log Analytics

There is this com.microsoft.pnp.Util package that is used inside my Scala notebook used for Azure Databricks to Azure Log Monitoring tables push for my application logs. When I updated my Databricks Runtime from 10.4 to 14 in my clusters, it starts t...

  • 28 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Top Kudoed Authors

Latest from our Blog

How to use Databricks Autoloader across AWS accounts

IntroductionCreate S3 bucket and cross-account instance profileUse Autoloader to create SNS-SQS across accountsManually create SNS-SQS for cross-account AutoloaderTest cross-account Autoloader connect...

88Views 2kudos

Grouped Pandas Optimization

In this short tutorial, we’ll implement an approach to making certain applyInPandas operations run many times faster. First, let's generate some dummy data for this example using Spark. For our exampl...

98Views 1kudos