cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

rami1
by New Contributor II
  • 895 Views
  • 0 replies
  • 0 kudos

Data bricks Write Performance

I have a requirement to replay ingestion from landing data and build silver table. I am trying to write delta file from raw Avro files based in landing zone. The raw files are located in folder based on date. I am currently using streaming to read d...

  • 895 Views
  • 0 replies
  • 0 kudos
TyronZerafa
by New Contributor II
  • 1739 Views
  • 0 replies
  • 2 kudos

Integrating with Prometheus

How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true

  • 1739 Views
  • 0 replies
  • 2 kudos
AbhishekBreeks
by New Contributor II
  • 902 Views
  • 0 replies
  • 0 kudos

Host a Star Schema Data Warehouse on Azure Databricks

Hello, Is it a good idea to Host a Schema Data Warehouse on Azure Databricks database itself. Usually we use Azure Databricks to Prep the data and then Host it on Azure Sql Database. However question is can we not Host the data on Azure Databricks i...

  • 902 Views
  • 0 replies
  • 0 kudos
WhatIsHappening
by New Contributor
  • 888 Views
  • 0 replies
  • 0 kudos

Pandas Forward Fill Based on Keyword

Hello! I am trying to forward fill a column in a Pandas dataframe based on a keyword. I have come up with: pdf_df['EEName_TEST'] = pdf_df['EEName_TEST'].str.contains('Name:').ffill() This gives me a boolean result but I still can't figure out what ...

  • 888 Views
  • 0 replies
  • 0 kudos
stramzik
by New Contributor II
  • 1543 Views
  • 1 replies
  • 1 kudos

Unable to mount datalake gen1 to databricks

I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...

  • 1543 Views
  • 1 replies
  • 1 kudos
Latest Reply
stramzik
New Contributor II
  • 1 kudos

bumping up the thread

  • 1 kudos
lycenok
by New Contributor II
  • 955 Views
  • 0 replies
  • 0 kudos

display function eats consecutive spaces

When using display, more than 1 spaces in strings are ignored. Can we change that behaviour? Are there any options for display functions? code example: display( spark.createDataFrame( [ ( 'a a' , 'a a' ) ], [ 'string_column', 'string_column_2' ] )...

  • 955 Views
  • 0 replies
  • 0 kudos
Charbel
by New Contributor II
  • 1466 Views
  • 0 replies
  • 1 kudos

Delta table is not writing data read from kafka

Guys, could you help me? I'm reading 5 kafka threads through a list and saving the data in a Delta table The execution will be 1x a day, it seems that everything is working but I noticed that when I read the topic and it has no message, it still gen...

0693f000007OoRrAAK
  • 1466 Views
  • 0 replies
  • 1 kudos
RiyazAli
by Valued Contributor II
  • 1270 Views
  • 0 replies
  • 1 kudos

Unable to subset the data using SparkR, using piping convention to execute the commands

I'm operating on some data that looks like the image attached. the command that I'm performing is :<code>library(magrittr) subsetting the data for MAC-OS & sorting by event-timestamp. acDF <- eventsDF %>% SparkR::select("device", "event_timestamp...

0693f000007OoRwAAK
  • 1270 Views
  • 0 replies
  • 1 kudos
akj2784
by New Contributor II
  • 20601 Views
  • 11 replies
  • 1 kudos

How to connect PostgreSQL from Databricks

I am trying to connect PostgreSQL from Azure Databricks. I am using the below code to connect. jdbcHostname = "Test" jdbcPort = 1234 jdbcDatabase = "Test1" jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase) Conn...

  • 20601 Views
  • 11 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Javier De La Torre do you really need two-way SSL (verify-full)? In most cases one way SSL (sslmode=require) should be enough. @akj2784​  When you say "Connection was successful", where do you mean you established a successful connection? You might...

  • 1 kudos
10 More Replies
Josh21
by New Contributor II
  • 992 Views
  • 0 replies
  • 1 kudos

2012-12-30 has year of both 2012 and 2013 sql

I am trying to obtain the month and year in the format of "MM-YYY", then "YYY" to get a values such as 12-2012. I noticed an error where a timestamp of 2012-12-30T00:00:00.000+0000 results in both 12-2013 and 2013. This is an error, since 2012-12-30...

  • 992 Views
  • 0 replies
  • 1 kudos
alecdavis47
by New Contributor
  • 692 Views
  • 0 replies
  • 0 kudos

databricks-connect without using cluster

For those of you who use databricks-connect probably know that it’s a great tool to use the power of spark/databricks, while executing/debugging code (and having proper git integration) from your favorite IDE. However, when you want to test somethin...

  • 692 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels