cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

nolanreilly
by New Contributor
  • 521 Views
  • 0 replies
  • 0 kudos

Impossible to read a custom pipeline? (Scala)

I have created a custom transformer to be used in a ml pipeline. I was able to write the pipeline to storage by extending the transformer class with DefaultParamsWritable. Reading the pipeline back in however, does not seem possible in Scala. I have...

  • 521 Views
  • 0 replies
  • 0 kudos
hravilla
by New Contributor
  • 2528 Views
  • 1 replies
  • 0 kudos

Upload file to DBFS fails with error code 0

When trying to upload to DBFS from local machine getting error as "Error occurred when processing file ... : Server responded with 0 code" DBR 7.3 LTSSpark 3.0.1 Scala 2.12 Uploading the file using the "upload" in the Databricks cloud console, the c...

  • 2528 Views
  • 1 replies
  • 0 kudos
Latest Reply
PramodNaik
New Contributor II
  • 0 kudos

Even I am facing the same issue with GCP databricks. I am able to upload files with smaller size. When i tried with 3MB file, databricks chokes. I get the above error. I tried with aws databricks, it works good even for bigger size files.

  • 0 kudos
aemorina
by New Contributor
  • 2016 Views
  • 0 replies
  • 0 kudos

Can Databricks Connect work on a cluster with port 443?

I am attempting to use Databricks Connect with a cluster in Azure Government with a port of 443 but get the following error when running databricks-connect test.The port you specified is either being used already or invalid. Port: The port that Data...

  • 2016 Views
  • 0 replies
  • 0 kudos
daniil_terentye
by New Contributor III
  • 1763 Views
  • 3 replies
  • 0 kudos

EXISTS statement works incorrectly

Hi everybody. Looks like EXISTS statement works incorrectly. If i execute the following statement in SQL Server it returns one row, as it should WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY...

  • 1763 Views
  • 3 replies
  • 0 kudos
Latest Reply
daniil_terentye
New Contributor III
  • 0 kudos

In newer versions of spark it's possible to use ANTI JOIN and SEMI JOIN It looks this way:WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY' AS name ), b AS ( SELECT 'a@b.com' AS user_username, 'Super Co...

  • 0 kudos
2 More Replies
nickmaco
by New Contributor II
  • 862 Views
  • 1 replies
  • 0 kudos

Databricks - autostart from jdbc query

Hi team, New to Databricks and trying to understand if there is a "True" auto-start capability with Databricks. We are evaluating Databricks Delta lake as an alternative cloud based datawarehouse solution but the biggest problem I see is the inabili...

  • 862 Views
  • 1 replies
  • 0 kudos
Latest Reply
nickmaco
New Contributor II
  • 0 kudos

Just adding on to this. Using DBeaver as a client and using a singlenode cluster and a pool of idling VM, it was possible to get the autostart time of the cluster down to 35 seconds, + 17 seconds for the query time on top to show the first 200 rows ...

  • 0 kudos
vishavgupta988
by New Contributor
  • 4013 Views
  • 2 replies
  • 0 kudos

How to set font-size of values in each cell of dataframe?

I am working on pandas and python.After processing a particular dataframe in my program , I am appending that dataframe below an existing Excel file. Now problem is my excel has font size of 11 pt but dataframe has font size of 12 pt.I want to set f...

  • 4013 Views
  • 2 replies
  • 0 kudos
Latest Reply
DominicFHelms
New Contributor II
  • 0 kudos

I like sharp fonts.

  • 0 kudos
1 More Replies
okmich
by New Contributor II
  • 1462 Views
  • 0 replies
  • 1 kudos

S3 connection reset error :: Removing Spark Config on Cluster

Hi guys, I am running a production pipeline (Databricks Runtime 7.3 LTS) that keeps failing for some delta file reads with the error: 21/07/19 09:56:02 ERROR Executor: Exception in task 36.1 in stage 2.0 (TID 58) com.databricks.sql.io.FileReadExcept...

  • 1462 Views
  • 0 replies
  • 1 kudos
talegari
by New Contributor
  • 419 Views
  • 0 replies
  • 0 kudos

sparkR.session() from web terminal

Question: sparkR.session() gives an error when run on web terminal, while it runs in a notebook. What parameters should be provided to create a spark session on web terminal? PS: I am trying to run a .R file using Rscript call on terminal instead ...

  • 419 Views
  • 0 replies
  • 0 kudos
DanSiegel
by New Contributor
  • 636 Views
  • 0 replies
  • 0 kudos

Access an external table from another workspace

What's the best way to add an external table so another cluster/workspace can access an existing external table on S3? I need to redeploy my workspace into a new VPC, so I am not expecting any collisions of the warehouses. Is it as simple as adding ...

  • 636 Views
  • 0 replies
  • 0 kudos
CalvinCalvert_
by New Contributor
  • 538 Views
  • 0 replies
  • 0 kudos

How does FSCK work and does it have any negative effects on subsequent notebook executions?

In my environment, there are 3 groups of notebooks that run on their own schedules, however they all use the same underlying transaction logs (auditlogs, as we call them) in S3. From time to time, various notebooks from each of the 3 groups fail wit...

  • 538 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels