cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

stramzik
by New Contributor II
  • 1127 Views
  • 1 replies
  • 1 kudos

Unable to mount datalake gen1 to databricks

I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...

  • 1127 Views
  • 1 replies
  • 1 kudos
Latest Reply
stramzik
New Contributor II
  • 1 kudos

bumping up the thread

  • 1 kudos
akj2784
by New Contributor II
  • 17028 Views
  • 11 replies
  • 1 kudos

How to connect PostgreSQL from Databricks

I am trying to connect PostgreSQL from Azure Databricks. I am using the below code to connect. jdbcHostname = "Test" jdbcPort = 1234 jdbcDatabase = "Test1" jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase) Conn...

  • 17028 Views
  • 11 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Javier De La Torre do you really need two-way SSL (verify-full)? In most cases one way SSL (sslmode=require) should be enough. @akj2784​  When you say "Connection was successful", where do you mean you established a successful connection? You might...

  • 1 kudos
10 More Replies
alecdavis47
by New Contributor
  • 466 Views
  • 0 replies
  • 0 kudos

databricks-connect without using cluster

For those of you who use databricks-connect probably know that it’s a great tool to use the power of spark/databricks, while executing/debugging code (and having proper git integration) from your favorite IDE. However, when you want to test somethin...

  • 466 Views
  • 0 replies
  • 0 kudos
nolanreilly
by New Contributor
  • 520 Views
  • 0 replies
  • 0 kudos

Impossible to read a custom pipeline? (Scala)

I have created a custom transformer to be used in a ml pipeline. I was able to write the pipeline to storage by extending the transformer class with DefaultParamsWritable. Reading the pipeline back in however, does not seem possible in Scala. I have...

  • 520 Views
  • 0 replies
  • 0 kudos
hravilla
by New Contributor
  • 2519 Views
  • 1 replies
  • 0 kudos

Upload file to DBFS fails with error code 0

When trying to upload to DBFS from local machine getting error as "Error occurred when processing file ... : Server responded with 0 code" DBR 7.3 LTSSpark 3.0.1 Scala 2.12 Uploading the file using the "upload" in the Databricks cloud console, the c...

  • 2519 Views
  • 1 replies
  • 0 kudos
Latest Reply
PramodNaik
New Contributor II
  • 0 kudos

Even I am facing the same issue with GCP databricks. I am able to upload files with smaller size. When i tried with 3MB file, databricks chokes. I get the above error. I tried with aws databricks, it works good even for bigger size files.

  • 0 kudos
aemorina
by New Contributor
  • 2009 Views
  • 0 replies
  • 0 kudos

Can Databricks Connect work on a cluster with port 443?

I am attempting to use Databricks Connect with a cluster in Azure Government with a port of 443 but get the following error when running databricks-connect test.The port you specified is either being used already or invalid. Port: The port that Data...

  • 2009 Views
  • 0 replies
  • 0 kudos
daniil_terentye
by New Contributor III
  • 1751 Views
  • 3 replies
  • 0 kudos

EXISTS statement works incorrectly

Hi everybody. Looks like EXISTS statement works incorrectly. If i execute the following statement in SQL Server it returns one row, as it should WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY...

  • 1751 Views
  • 3 replies
  • 0 kudos
Latest Reply
daniil_terentye
New Contributor III
  • 0 kudos

In newer versions of spark it's possible to use ANTI JOIN and SEMI JOIN It looks this way:WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY' AS name ), b AS ( SELECT 'a@b.com' AS user_username, 'Super Co...

  • 0 kudos
2 More Replies
nickmaco
by New Contributor II
  • 852 Views
  • 1 replies
  • 0 kudos

Databricks - autostart from jdbc query

Hi team, New to Databricks and trying to understand if there is a "True" auto-start capability with Databricks. We are evaluating Databricks Delta lake as an alternative cloud based datawarehouse solution but the biggest problem I see is the inabili...

  • 852 Views
  • 1 replies
  • 0 kudos
Latest Reply
nickmaco
New Contributor II
  • 0 kudos

Just adding on to this. Using DBeaver as a client and using a singlenode cluster and a pool of idling VM, it was possible to get the autostart time of the cluster down to 35 seconds, + 17 seconds for the query time on top to show the first 200 rows ...

  • 0 kudos
vishavgupta988
by New Contributor
  • 4007 Views
  • 2 replies
  • 0 kudos

How to set font-size of values in each cell of dataframe?

I am working on pandas and python.After processing a particular dataframe in my program , I am appending that dataframe below an existing Excel file. Now problem is my excel has font size of 11 pt but dataframe has font size of 12 pt.I want to set f...

  • 4007 Views
  • 2 replies
  • 0 kudos
Latest Reply
DominicFHelms
New Contributor II
  • 0 kudos

I like sharp fonts.

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels