cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Mirko
by Contributor
  • 3713 Views
  • 6 replies
  • 1 kudos

Resolved! Group vs User rights

I have a small question: How does the combination of group and user rights work? Is it like in azure, that if i have for example databricks sql acces threw a (databricks) group i am member of, but in my personal account databricks sql is not enabled...

  • 3713 Views
  • 6 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Mirko Ludewig​ - Good morning (or evening depending on where you hail), would you be happy to mark whichever answer resolved the problem for you as best? That helps other members find the solutions more quickly.

  • 1 kudos
5 More Replies
KKDataEngineer
by New Contributor III
  • 1083 Views
  • 0 replies
  • 2 kudos

Spark Structred Streaming, An Aggregation DF with Watermark in Append mode to Delta table is not writing the most recent aggregation to the Delta table even after crossing the water mark boundary. This is causing dataloss

Team,  I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark. I am reading a stream from events hub ( Extract) Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED...

  • 1083 Views
  • 0 replies
  • 2 kudos
cristianc
by Contributor
  • 4275 Views
  • 9 replies
  • 0 kudos

Resolved! Query AWS Redshift from Databricks SQL

Greetings,In the documentation for Databricks SQL it states that it supports JDBC connections, however when connecting to AWS Redshift via the built in PostgreSQL driver ("CREATE TABLE sample USING JDBC" and "jdbc://postgresql:/..." URI) I'm getting ...

  • 4275 Views
  • 9 replies
  • 0 kudos
Latest Reply
cristianc
Contributor
  • 0 kudos

@Bilal Aslam​ anytime! Is there a place where customers could follow the timeline when such features are introduced?

  • 0 kudos
8 More Replies
Axel_Schwanke
by Contributor
  • 4728 Views
  • 7 replies
  • 3 kudos

Resolved! Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

I have a simple SparkSQL Select statementoffers_df = (spark.sql(""" SELECT * FROM delta.`{}` """.format(TABLE_LOCATION)))It runs under DBR 9.0 and previous. When changing the DBR to 9.1 ...10.1 I get an exceptionorg.apache.spark.SparkException: Una...

  • 4728 Views
  • 7 replies
  • 3 kudos
Latest Reply
Axel_Schwanke
Contributor
  • 3 kudos

Retest in DBR 10.3 beta SUCCESSFUL.Problem does not occur in DBR 10.3 beta

  • 3 kudos
6 More Replies
Anonymous
by Not applicable
  • 22407 Views
  • 4 replies
  • 4 kudos

Resolved! Spark is not able to resolve the columns correctly when joins data frames

Hello all, I m using pyspark ( python 3.8) over spark3.0 on Databricks. When running this DataFrame join:next_df = days_currencies_matrix.alias('a').join( data_to_merge.alias('b') , [ days_currencies_matrix.dt == data_to_merge.RATE_DATE, days...

  • 22407 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Alessio Palma​ - Howdy! My name is Piper, and I'm a moderator for the community. Would you be happy to mark whichever answer solved your issue so other members may find the solution more quickly?

  • 4 kudos
3 More Replies
Bilal1
by New Contributor III
  • 3580 Views
  • 6 replies
  • 4 kudos

Resolved! SQL Analytics: Is it possible to configure the default number format

HiWhen querying an integer value, the default format is '0.0' which results in an integer value 202111, displayed as 202,111. I can resolve the issue by updating the visualisation or using formatnumber in my query, however I would like to set a defa...

  • 3580 Views
  • 6 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Bilal Haniff​ - Would you be happy to mark whichever answer helped you the most as best? That helps others find solutions more quickly.

  • 4 kudos
5 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 1370 Views
  • 1 replies
  • 15 kudos

Resolved! Write to Azure Delta Lake - optimization request

Databricks/Delta team could optimize some commands which writes to Azure Blob Storage as Azure display that message:

image
  • 1370 Views
  • 1 replies
  • 15 kudos
Latest Reply
Anonymous
Not applicable
  • 15 kudos

Hey there. Thank you for your suggestion. I'll pass this up to the team.

  • 15 kudos
wyzer
by Contributor II
  • 3022 Views
  • 2 replies
  • 4 kudos

Resolved! How to show the properties of the folders/files from DBFS ?

Hello,How to show the properties of the folders/files from DBFS ?Currently i am using this command :display(dbutils.fs.ls("dbfs:/"))But it only shows :pathnamesizeHow to show these properties ? : CreatedBy (Name)CreatedOn (Date)ModifiedBy (Name)Modi...

  • 3022 Views
  • 2 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

Only one idea is to use %sh magic command but there is no name (just root)

  • 4 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 13289 Views
  • 3 replies
  • 26 kudos

How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace �� Private link In your storage accounts please go to “Networ...

How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace Private linkIn your storage accounts please go to “Networking” -> “Private endpoint connections” and click Add Private Endpoint.It is important to add private links in ...

image.png image.png image.png image.png
  • 13289 Views
  • 3 replies
  • 26 kudos
Latest Reply
Anonymous
Not applicable
  • 26 kudos

@Hubert Dudek​ - Have I told you lately that you're the best!?!

  • 26 kudos
2 More Replies
HGH_Vallarta
by New Contributor
  • 666 Views
  • 0 replies
  • 0 kudos

Not having a standard height becomes quite a challenge for many growing adolescents, which is said to be a condition of idiopathic short stature. So, ...

Not having a standard height becomes quite a challenge for many growing adolescents, which is said to be a condition of idiopathic short stature. So, while we can think it will be “ok” with time, we can’t ignore the elephant in the room. To diagnose ...

  • 666 Views
  • 0 replies
  • 0 kudos
Cordis_Technolo
by New Contributor
  • 810 Views
  • 0 replies
  • 0 kudos

Inventory Management System  Handling the inventory and keeping track of everything is the most difficult, yet important aspect of leading a business...

Inventory Management System Handling the inventory and keeping track of everything is the most difficult, yet important aspect of leading a business. Inventory never stays the same. Due to sales, return, purchases, etc your business’ inventory is alw...

  • 810 Views
  • 0 replies
  • 0 kudos
venkyv
by New Contributor II
  • 2028 Views
  • 1 replies
  • 3 kudos

Resolved! Can I use Databricks to join data from S3 and Postgres using SQL?

Hello, I'm very much new to Databricks and I'm finding it hard if it's right solution for our needs.Requirement:We have multiple data sources spread across AWS S3 and Postgres. We need a common SQL endpoint that can be used to write queries to join d...

  • 2028 Views
  • 1 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Yes you can. You can ETL to data lake storage register your tables to metastore and register your SELECT with JOINS as VIEW or even better create additionally jobs and store your JOINED table. From BI you can connect to databricks sql or to data lake...

  • 3 kudos
bluetail
by Contributor
  • 2539 Views
  • 4 replies
  • 2 kudos

Resolved! Value Labels fail to display in Databricks notebook but they are displayed ok in Jupyter

import matplotlib.pyplot as pltimport seaborn as snsimport pandas as pdimport numpy as npprob = np.random.rand(7) + 0.1prob /= prob.sum()df = pd.DataFrame({'department': np.random.choice(['helium', 'neon', 'argon', 'krypton', 'xenon', 'radon', 'ogane...

  • 2539 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Maria Bruevich​ - Do either of these answers help? If yes, would you be happy to mark one as best so that other members can find the solution more quickly?

  • 2 kudos
3 More Replies
kolangareth
by New Contributor III
  • 4977 Views
  • 9 replies
  • 3 kudos

Resolved! to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS

I am trying to do a dynamic partition overwrite on delta table using replaceWhere option. This was working fine until I upgraded the DB runtime to 9.1 LTS from 8.3.x. I am concatenating 'year', 'month' and 'day' columns and then using to_date functio...

  • 4977 Views
  • 9 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Prasanth Kolangareth​ - Does Hubert's answer resolve the problem for you? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

  • 3 kudos
8 More Replies
guruv
by New Contributor III
  • 5244 Views
  • 4 replies
  • 1 kudos

Resolved! Saprk UI not showing any running tasks

HI,I am running a Notebook job calling a JAR code (application code implmented in C#). in the Spark UI page for almost 2 hrs, it'w not showing any tasks and even the CPU usage is below 20%, memory usage is very small. Before this 2 hr window it shows...

  • 5244 Views
  • 4 replies
  • 1 kudos
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

If I understood the issue correctly .

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels