cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

krishnakash
by New Contributor II
  • 3710 Views
  • 1 replies
  • 1 kudos

How to provide custom class extending SparkPlugin/ExecutorPlugin in Databricks 7.3?

How to properly configure the jar containing the class and spark plugin in Databricks?During DBR 7.3 cluster creation, I tried by setting the spark.plugins, spark.driver.extraClassPath and spark.executor.extraClassPath Spark configs by copying the ja...

  • 3710 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello @Krishna Kashiv​ - I don't know if we've met yet. My name is Piper and I'm a community moderator here. Thank you for your new question. It looks thorough! Let's give it a while to see what our members have to say. Otherwise, we will circle back...

  • 1 kudos
sriwin
by New Contributor
  • 2873 Views
  • 1 replies
  • 0 kudos

Create gpg file and save to AWS s3 storage in scala

Hi - Could you please help me on how can I create a scala notebook to perform the below tasksEncrypt a text file using the gpgUpload the file to amazon s3 storageverify the file exists in amazon s3decrypt the encrypted file to verify no issuesApprec...

  • 2873 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello! My name is Piper and I'm a community moderator for Databricks. Thanks for your question. Let's give it a bit more to see what our members have to say. If not, we'll circle back around.

  • 0 kudos
cconnell
by Contributor II
  • 6715 Views
  • 11 replies
  • 7 kudos

Resolved! What is the proper way to import the new pyspark.pandas library?

I am moving an existing, working pandas program into Databricks. I want to use the new pyspark.pandas library, and change my code as little as possible. It appears that I should do the following:1) Add from pyspark import pandas as ps at the top2) Ch...

  • 6715 Views
  • 11 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Make sure to use the 10.0 Runtime which includes Spark 3.2

  • 7 kudos
10 More Replies
IgnacioCastinei
by New Contributor III
  • 10415 Views
  • 6 replies
  • 2 kudos

CLI Command <databricks fs cp> Not Uploading Files to DBFS

Hi all, So far I have been successfully using the CLI interface to upload files from my local machine to DBFS/FileStore/tables. Specifically, I have been using my terminal and the following command: databricks fs cp -r <MyLocalDataset> dbfs:/FileStor...

  • 10415 Views
  • 6 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

hi @Ignacio Castineiras​ ,If Arjun.kr's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?Please let us know if you still are having this issue.

  • 2 kudos
5 More Replies
ExtreemTactical
by New Contributor
  • 518 Views
  • 0 replies
  • 0 kudos

1.   DIFFERENT TYPES OF TACTICAL GEAR 1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a hug...

1.   DIFFERENT TYPES OF TACTICAL GEAR1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a huge group of features and capacities. Packs and pockets are made of climate-safe material planned to ke...

  • 518 Views
  • 0 replies
  • 0 kudos
Adrien
by New Contributor
  • 1872 Views
  • 1 replies
  • 0 kudos

Creating a table like in SQL with Spark

Hi !I'm working on a project at my company on Databricks using Scala and Spark. I'm new to Spark and Databricks and so I would like to know how to create a table on specific location (on the Delta Lake of my company). In SQL + some Delta features, I ...

  • 1872 Views
  • 1 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Adrien MERAT​ ,I would like to share the following documentation that will provide examples on how to create Delta tables:Create Delta table linkDelta data types link

  • 0 kudos
vasu_sethia
by New Contributor II
  • 3029 Views
  • 8 replies
  • 0 kudos

Spark adding NUL

Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is ​adding NUL character af the front of this line and at the end,...

  • 3029 Views
  • 8 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Hard to tell without having the code, but it might be the separator for the csv? You do have comma's in the string, and comma is the default separator for csv.

  • 0 kudos
7 More Replies
afshinR
by New Contributor III
  • 3529 Views
  • 4 replies
  • 3 kudos

Hi, I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text...

Hi,I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text area of my form back in to the code cell of the notebook.Example:displayHTML ("""<form><textarea> u...

  • 3529 Views
  • 4 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

Hi @afshin riahi​ ,Did Dan's response helped you to solve your question? if it did, can you mark it as best answer? I will help to move the post to the top so other can quickly find the solution.

  • 3 kudos
3 More Replies
cig0
by New Contributor II
  • 4809 Views
  • 5 replies
  • 2 kudos

Resolved! AWS VPC peering connection: can't make Databricks VPC reach our services on the accepter VPC

Hi,We followed this document (https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html) describing how to establish a connection between two (or more) VPC in AWS, but so far we haven't been able to communicate with t...

  • 4809 Views
  • 5 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Hi @Martin Cigorraga​ ,If Huaming's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?

  • 2 kudos
4 More Replies
Ayman
by New Contributor
  • 5052 Views
  • 3 replies
  • 0 kudos
  • 5052 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Ayman Alneser​ ,Did Huaming.lu's response worked for you? if it did, could you marked as the best solution so that other can quickly find it in the future.

  • 0 kudos
2 More Replies
TJS
by New Contributor II
  • 16523 Views
  • 6 replies
  • 5 kudos

Resolved! Can you help with this error please? Issue when using a new high concurrency cluster

Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...

  • 16523 Views
  • 6 replies
  • 5 kudos
Latest Reply
Pradeep54
Databricks Employee
  • 5 kudos

@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false

  • 5 kudos
5 More Replies
William_Scardua
by Valued Contributor
  • 10669 Views
  • 8 replies
  • 2 kudos

Resolved! How many hours I can estimate to trainning in a Databricks Academy Self-Placed Trainning platform ?

I done the Data Engineering Profissional and others training in a Self-Placed Trainning (https://www.linkedin.com/posts/wscardua_data-engineering-professional-activity-6851487238774108160-IsTE) . How many hours can I estimate for this training (and o...

  • 10669 Views
  • 8 replies
  • 2 kudos
Latest Reply
William_Scardua
Valued Contributor
  • 2 kudos

Can anyone help ?

  • 2 kudos
7 More Replies
Anonymous
by Not applicable
  • 1931 Views
  • 2 replies
  • 4 kudos

Multi-task Job Run starting point

Hi community!I would like to know if it is possible to start a Multi-task Job Run from and specific task. The use case is as follows:I have a 17 tasks JobA task in the middle, let's say a task after 2 dependencies, failsI found the error and now it i...

  • 1931 Views
  • 2 replies
  • 4 kudos
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 4 kudos

+1 to what @Dan Zafar​  said. We're working **** ** this. Looking forward to bring this to you in the near future.

  • 4 kudos
1 More Replies
alexraj84
by New Contributor
  • 12079 Views
  • 2 replies
  • 0 kudos

How to read a fixed length file in Spark using DataFrame API and SCALA

I have a fixed length file ( a sample is shown below) and I want to read this file using DataFrames API in Spark using SCALA(not python or java). Using DataFrames API there are ways to read textFile, json file and so on but not sure if there is a wa...

  • 12079 Views
  • 2 replies
  • 0 kudos
Latest Reply
Nagendra
New Contributor II
  • 0 kudos

Find the below solution which can be used. Let us consider this is the data in the file. EMP ID   First Name              Last Name                       1Chris                   M                                                     2John            ...

  • 0 kudos
1 More Replies
aditya_raj_data
by New Contributor II
  • 6893 Views
  • 4 replies
  • 2 kudos

Hosting python application on Azure Databricks and exposing it's rest APIs

Hello, I am trying to host my application on Databricks and I want to expose rest APIs of my application to be accessed from postman but I am unable to find any documentation on how to do this. I tried to write simple flask "hello world" code to try ...

  • 6893 Views
  • 4 replies
  • 2 kudos
Latest Reply
Manoj
Contributor II
  • 2 kudos

I did this using Azure web app and exposed the APIs , was able to access that in Post Man and Data bricks. Not used python app on data bricks

  • 2 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels