cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ExtreemTactical
by New Contributor
  • 805 Views
  • 0 replies
  • 0 kudos

1.   DIFFERENT TYPES OF TACTICAL GEAR 1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a hug...

1.   DIFFERENT TYPES OF TACTICAL GEAR1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a huge group of features and capacities. Packs and pockets are made of climate-safe material planned to ke...

  • 805 Views
  • 0 replies
  • 0 kudos
Adrien
by New Contributor
  • 3981 Views
  • 1 replies
  • 0 kudos

Creating a table like in SQL with Spark

Hi !I'm working on a project at my company on Databricks using Scala and Spark. I'm new to Spark and Databricks and so I would like to know how to create a table on specific location (on the Delta Lake of my company). In SQL + some Delta features, I ...

  • 3981 Views
  • 1 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Adrien MERAT​ ,I would like to share the following documentation that will provide examples on how to create Delta tables:Create Delta table linkDelta data types link

  • 0 kudos
vasu_sethia
by New Contributor II
  • 4981 Views
  • 8 replies
  • 0 kudos

Spark adding NUL

Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is ​adding NUL character af the front of this line and at the end,...

  • 4981 Views
  • 8 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Hard to tell without having the code, but it might be the separator for the csv? You do have comma's in the string, and comma is the default separator for csv.

  • 0 kudos
7 More Replies
afshinR
by New Contributor III
  • 5769 Views
  • 4 replies
  • 3 kudos

Hi, I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text...

Hi,I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text area of my form back in to the code cell of the notebook.Example:displayHTML ("""<form><textarea> u...

  • 5769 Views
  • 4 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

Hi @afshin riahi​ ,Did Dan's response helped you to solve your question? if it did, can you mark it as best answer? I will help to move the post to the top so other can quickly find the solution.

  • 3 kudos
3 More Replies
cig0
by New Contributor II
  • 7016 Views
  • 5 replies
  • 2 kudos

Resolved! AWS VPC peering connection: can't make Databricks VPC reach our services on the accepter VPC

Hi,We followed this document (https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html) describing how to establish a connection between two (or more) VPC in AWS, but so far we haven't been able to communicate with t...

  • 7016 Views
  • 5 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Hi @Martin Cigorraga​ ,If Huaming's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?

  • 2 kudos
4 More Replies
Ayman
by New Contributor
  • 6836 Views
  • 3 replies
  • 0 kudos
  • 6836 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Ayman Alneser​ ,Did Huaming.lu's response worked for you? if it did, could you marked as the best solution so that other can quickly find it in the future.

  • 0 kudos
2 More Replies
TJS
by New Contributor II
  • 19232 Views
  • 6 replies
  • 5 kudos

Resolved! Can you help with this error please? Issue when using a new high concurrency cluster

Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...

  • 19232 Views
  • 6 replies
  • 5 kudos
Latest Reply
Pradeep54
Databricks Employee
  • 5 kudos

@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false

  • 5 kudos
5 More Replies
William_Scardua
by Valued Contributor
  • 19115 Views
  • 8 replies
  • 2 kudos

Resolved! How many hours I can estimate to trainning in a Databricks Academy Self-Placed Trainning platform ?

I done the Data Engineering Profissional and others training in a Self-Placed Trainning (https://www.linkedin.com/posts/wscardua_data-engineering-professional-activity-6851487238774108160-IsTE) . How many hours can I estimate for this training (and o...

  • 19115 Views
  • 8 replies
  • 2 kudos
Latest Reply
William_Scardua
Valued Contributor
  • 2 kudos

Can anyone help ?

  • 2 kudos
7 More Replies
Anonymous
by Not applicable
  • 2740 Views
  • 2 replies
  • 4 kudos

Multi-task Job Run starting point

Hi community!I would like to know if it is possible to start a Multi-task Job Run from and specific task. The use case is as follows:I have a 17 tasks JobA task in the middle, let's say a task after 2 dependencies, failsI found the error and now it i...

  • 2740 Views
  • 2 replies
  • 4 kudos
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 4 kudos

+1 to what @Dan Zafar​  said. We're working **** ** this. Looking forward to bring this to you in the near future.

  • 4 kudos
1 More Replies
alexraj84
by New Contributor
  • 14865 Views
  • 2 replies
  • 0 kudos

How to read a fixed length file in Spark using DataFrame API and SCALA

I have a fixed length file ( a sample is shown below) and I want to read this file using DataFrames API in Spark using SCALA(not python or java). Using DataFrames API there are ways to read textFile, json file and so on but not sure if there is a wa...

  • 14865 Views
  • 2 replies
  • 0 kudos
Latest Reply
Nagendra
Databricks Partner
  • 0 kudos

Find the below solution which can be used. Let us consider this is the data in the file. EMP ID   First Name              Last Name                       1Chris                   M                                                     2John            ...

  • 0 kudos
1 More Replies
aditya_raj_data
by New Contributor II
  • 9099 Views
  • 4 replies
  • 2 kudos

Hosting python application on Azure Databricks and exposing it's rest APIs

Hello,   I am trying to host my application on Databricks and I want to expose rest APIs of my application to be accessed from postman but I am unable to find any documentation on how to do this. I tried to write simple flask "hello world" code to tr...

  • 9099 Views
  • 4 replies
  • 2 kudos
Latest Reply
Manoj
Contributor II
  • 2 kudos

I did this using Azure web app and exposed the APIs , was able to access that in Post Man and Data bricks. Not used python app on data bricks

  • 2 kudos
3 More Replies
User16753725182
by Databricks Employee
  • 3870 Views
  • 1 replies
  • 0 kudos

How to setup a private git repository in my workspace?

How to setup a private git repository in my workspace?

  • 3870 Views
  • 1 replies
  • 0 kudos
Latest Reply
atulsahu
New Contributor II
  • 0 kudos

As a platform engineer, I would go to the admin console and click on "workspace settings" and start by looking into the below settings. Repos: true, so that Repos integration is possibleThe next two settings, are important to make the overall experi...

  • 0 kudos
Rnmj
by New Contributor III
  • 17193 Views
  • 3 replies
  • 6 kudos

ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

I am trying to run a python code where a json file is flattened to pipe separated file . The code works with smaller files but for huge files of 2.4 GB I get below error:ConnectException: Connection refused (Connection refused)Error while obtaining a...

  • 17193 Views
  • 3 replies
  • 6 kudos
Latest Reply
Rnmj
New Contributor III
  • 6 kudos

Hi @Jose Gonzalez​ , @Werner Stinckens​  @Kaniz Fatma​ ,Thanks for your response .Appreciate a lot. The issue was in the code, it was a python /panda code running on Spark. Due to this only driver node was being used. i did validate this by increasin...

  • 6 kudos
2 More Replies
krishnakash
by New Contributor II
  • 5976 Views
  • 4 replies
  • 4 kudos

Resolved! Is there any way of determining last stage of SparkSQL Application Execution?

I have created custom UDF's that generate logs. These logs can be flushed by calling another API exposed which is exposed by an internal layer. However I want to call this API just after the execution of the UDF comes to an end. Is there any way of d...

  • 5976 Views
  • 4 replies
  • 4 kudos
Latest Reply
User16763506586
Databricks Employee
  • 4 kudos

@Krishna Kashiv​ May be ExecutorPlugin.java can help. It has all the methods you might required. Let me know if it works or not.You need to implement this interface org.apache.spark.api.plugin.SparkPluginand expose it as spark.plugins = com.abc.Imp...

  • 4 kudos
3 More Replies
Braxx
by Contributor II
  • 3027 Views
  • 1 replies
  • 3 kudos

Retry api request if fails

I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place? Here is an api request:d...

  • 3027 Views
  • 1 replies
  • 3 kudos
Latest Reply
Manoj
Contributor II
  • 3 kudos

@Bartosz Wachocki​ ,Use timeout, retry interval ,recursion and exception handling pseudo code belowtimeout = 300def exec_query(query,timeout): try: df = spark.createDataFrame(sf.bulk.MyTable.query(query)) except: if timeout > 0 : sleep(60) exec_que...

  • 3 kudos
Labels