cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

NicolasJ
by New Contributor
  • 3801 Views
  • 0 replies
  • 0 kudos

How to use Apache Sedona on Delta Live tables?

Hello,I am trying to run some geospatial transformations in Delta Live Table, using Apache Sedona.I tried defining a minimal example pipeline demonstrating the problem I encounter.First cell of my Notebook, I install apache-sedona Python package:%pip...

image
  • 3801 Views
  • 0 replies
  • 0 kudos
alejandrofm
by Valued Contributor
  • 2493 Views
  • 1 replies
  • 5 kudos

Resolved! How to set a global checkpoint for all notebooks?

I have several users doing data analysis on Databricks Spark notebooks, everything is smooth, now I want to make sure that the checkpointdir is configured on the cluster start, so every user doesn't had to set it on the Notebook (ending up in a lot o...

image
  • 2493 Views
  • 1 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

@Alejandro Martinez​ , For streaming jobs, there are, but others couldn't find them. Here are spark conf Configuration - Spark 3.2.1 Documentation (apache.org)spark.sql.streaming.checkpointLocation

  • 5 kudos
kiselevanton
by New Contributor II
  • 1183 Views
  • 0 replies
  • 2 kudos

Community edition, sign up issue

I am studying Databricks as part of Data Science with Databricks for Data Analysts Specialization. It requires me to sign up for community edition, but it simply doesn't work.I have tried to sign up 3 times with 3 different e-mails, 2 private and 1 c...

  • 1183 Views
  • 0 replies
  • 2 kudos
Anuj93
by New Contributor III
  • 1149 Views
  • 0 replies
  • 0 kudos

Mount Point Service Principle

Is there any way to find out the service principle/Azure Active Directory applicationused to mount a container?

  • 1149 Views
  • 0 replies
  • 0 kudos
JeNar
by New Contributor II
  • 1519 Views
  • 0 replies
  • 1 kudos

Email Verification Databricks Community Edition

I was try to make account in Databricks Community Edition, After finish my registration , i wait for email verification, But until now ..i didn't received email verification. And i try to make 2nd account with my another email. But it same, i didn't ...

  • 1519 Views
  • 0 replies
  • 1 kudos
Cameron_Afzal
by New Contributor II
  • 1160 Views
  • 1 replies
  • 0 kudos

I'm unable to create an account for Databricks Community Edition. I've tried multiple email addresses and browsers across multiple attempts. I...

I'm unable to create an account for Databricks Community Edition. I've tried multiple email addresses and browsers across multiple attempts. I fill out and submit the sign-up form but never receive the email and thus can't log in. Any advice? Are the...

  • 1160 Views
  • 1 replies
  • 0 kudos
Latest Reply
tipu
New Contributor II
  • 0 kudos

i have try the same thing it doesn't work. can someone please help us?

  • 0 kudos
LukaszJ
by Contributor III
  • 785 Views
  • 0 replies
  • 0 kudos

Real time query plotting

Hello,I have a table on Azure Databricks that I keep updating with the "A" notebook.And I want to real time plotting the query result from the table (let's say SELECT COUNT(name), name FROM my_schema.my_table GROUP BY name).I know about Azure Applica...

  • 785 Views
  • 0 replies
  • 0 kudos
LukaszJ
by Contributor III
  • 1718 Views
  • 2 replies
  • 1 kudos

Table access control cluster with R language

Hello,I want to have a high concurrency cluster with table access control and I want to use R language on it.I know that the documentation says that R and Scala is not available with table access control.But maybe you have some tricks or best practic...

  • 1718 Views
  • 2 replies
  • 1 kudos
Latest Reply
Aashita
Databricks Employee
  • 1 kudos

@Łukasz Jaremek​, Currently it is only available in Python and SQL.

  • 1 kudos
1 More Replies
prasadvaze
by Valued Contributor II
  • 7469 Views
  • 3 replies
  • 5 kudos

Resolved! Limit on number of result rows displayed on databricks SQL UI

Databricks SQL UI currently limits the query results display to 64000 rows. When will this limit go away? Using SSMS I get 40MM rows results in the UI and my users won't switch to databricks SQL for this reason

  • 7469 Views
  • 3 replies
  • 5 kudos
Latest Reply
User16765136105
New Contributor III
  • 5 kudos

Hi @prasad vaze​ - We do have a feature in the works that will increase this limit. If you reach out to your Databricks contact they can give you more details regarding dates and the preview.

  • 5 kudos
2 More Replies
samrachmiletter
by New Contributor III
  • 3315 Views
  • 2 replies
  • 5 kudos

Resolved! Is it possible to set order of precedence of spark SQL extensions?

I have the iceberg SQL extension installed, but running commands such as MERGE INTO result in the error pyspark.sql.utils.AnalysisException: MERGE destination only supports Delta sources.this seems to be due to using Delta's MERGE command as opposed ...

  • 3315 Views
  • 2 replies
  • 5 kudos
Latest Reply
samrachmiletter
New Contributor III
  • 5 kudos

This does help. I tried going through the DataFrameReader as well but ran into the same error, so it seems it is indeed not possible. Thank you @Hubert Dudek​!

  • 5 kudos
1 More Replies
laus
by New Contributor III
  • 35772 Views
  • 3 replies
  • 2 kudos

Resolved! How to solve Py4JJavaError: An error occurred while calling o5082.csv. : org.apache.spark.SparkException: Job aborted. when writing to csv

Hi ,I get the error: Py4JJavaError: An error occurred while calling o5082.csv.: org.apache.spark.SparkException: Job aborted. when writing to csv.Screenshot below with detail error.Any idea how to solve it?Thanks!

Screenshot 2022-03-31 at 17.33.26
  • 35772 Views
  • 3 replies
  • 2 kudos
Latest Reply
Noopur_Nigam
Databricks Employee
  • 2 kudos

Please try output.coalesce(1).write.option("header","true").format("csv").save("path")It seems to be same to https://community.databricks.com/s/topic/0TO3f000000CjVqGAK/py4jjavaerror

  • 2 kudos
2 More Replies
Hila_DG
by New Contributor II
  • 3378 Views
  • 5 replies
  • 4 kudos

Resolved! How to proactively monitor the use of the cache for driver node?

The problem:We have a dataframe which is based on the query:SELECT * FROM Very_Big_TableThis table returns over 4 GB of data, and when we try to push the data to Power BI we get the error message:ODBC: ERROR [HY000] [Microsoft][Hardy] (35) Error from...

  • 3378 Views
  • 5 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hey @Hila Galapo​ Hope everything is going good. Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 4 kudos
4 More Replies
findinpath
by Contributor
  • 4808 Views
  • 7 replies
  • 4 kudos

Resolved! Please share Databricks JDBC Driver on Maven Central

Can you please share the Databricks JDBC Driver on Maven Central ?I see it available on : https://databricks.com/spark/jdbc-drivers-download . However I can’t find it on Maven Central to make use of it in automated tests connecting to Databricks infr...

  • 4808 Views
  • 7 replies
  • 4 kudos
Latest Reply
findinpath
Contributor
  • 4 kudos

Thank you for the assistance and for releasing the jdbc driver to Maven Central.I consider the issue closed.

  • 4 kudos
6 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels