- 8680 Views
- 11 replies
- 7 kudos
Resolved! What is the proper way to import the new pyspark.pandas library?
I am moving an existing, working pandas program into Databricks. I want to use the new pyspark.pandas library, and change my code as little as possible. It appears that I should do the following:1) Add from pyspark import pandas as ps at the top2) Ch...
- 8680 Views
- 11 replies
- 7 kudos
- 13096 Views
- 6 replies
- 2 kudos
CLI Command <databricks fs cp> Not Uploading Files to DBFS
Hi all, So far I have been successfully using the CLI interface to upload files from my local machine to DBFS/FileStore/tables. Specifically, I have been using my terminal and the following command: databricks fs cp -r <MyLocalDataset> dbfs:/FileStor...
- 13096 Views
- 6 replies
- 2 kudos
- 2 kudos
hi @Ignacio Castineiras​ ,If Arjun.kr's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?Please let us know if you still are having this issue.
- 2 kudos
- 729 Views
- 0 replies
- 0 kudos
1.   DIFFERENT TYPES OF TACTICAL GEAR 1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a hug...
1. DIFFERENT TYPES OF TACTICAL GEAR1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a huge group of features and capacities. Packs and pockets are made of climate-safe material planned to ke...
- 729 Views
- 0 replies
- 0 kudos
- 3193 Views
- 1 replies
- 0 kudos
Creating a table like in SQL with Spark
Hi !I'm working on a project at my company on Databricks using Scala and Spark. I'm new to Spark and Databricks and so I would like to know how to create a table on specific location (on the Delta Lake of my company). In SQL + some Delta features, I ...
- 3193 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Adrien MERAT​ ,I would like to share the following documentation that will provide examples on how to create Delta tables:Create Delta table linkDelta data types link
- 0 kudos
- 4340 Views
- 8 replies
- 0 kudos
Spark adding NUL
Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is ​adding NUL character af the front of this line and at the end,...
- 4340 Views
- 8 replies
- 0 kudos
- 0 kudos
Hard to tell without having the code, but it might be the separator for the csv? You do have comma's in the string, and comma is the default separator for csv.
- 0 kudos
- 5389 Views
- 4 replies
- 3 kudos
Hi, I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text...
Hi,I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text area of my form back in to the code cell of the notebook.Example:displayHTML ("""<form><textarea> u...
- 5389 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @afshin riahi​ ,Did Dan's response helped you to solve your question? if it did, can you mark it as best answer? I will help to move the post to the top so other can quickly find the solution.
- 3 kudos
- 6431 Views
- 5 replies
- 2 kudos
Resolved! AWS VPC peering connection: can't make Databricks VPC reach our services on the accepter VPC
Hi,We followed this document (https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html) describing how to establish a connection between two (or more) VPC in AWS, but so far we haven't been able to communicate with t...
- 6431 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Martin Cigorraga​ ,If Huaming's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?
- 2 kudos
- 6406 Views
- 3 replies
- 0 kudos
- 6406 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ayman Alneser​ ,Did Huaming.lu's response worked for you? if it did, could you marked as the best solution so that other can quickly find it in the future.
- 0 kudos
- 18556 Views
- 6 replies
- 5 kudos
Resolved! Can you help with this error please? Issue when using a new high concurrency cluster
Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...
- 18556 Views
- 6 replies
- 5 kudos
- 5 kudos
@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false
- 5 kudos
- 15834 Views
- 8 replies
- 2 kudos
Resolved! How many hours I can estimate to trainning in a Databricks Academy Self-Placed Trainning platform ?
I done the Data Engineering Profissional and others training in a Self-Placed Trainning (https://www.linkedin.com/posts/wscardua_data-engineering-professional-activity-6851487238774108160-IsTE) . How many hours can I estimate for this training (and o...
- 15834 Views
- 8 replies
- 2 kudos
- 2562 Views
- 2 replies
- 4 kudos
Multi-task Job Run starting point
Hi community!I would like to know if it is possible to start a Multi-task Job Run from and specific task. The use case is as follows:I have a 17 tasks JobA task in the middle, let's say a task after 2 dependencies, failsI found the error and now it i...
- 2562 Views
- 2 replies
- 4 kudos
- 4 kudos
+1 to what @Dan Zafar​ said. We're working **** ** this. Looking forward to bring this to you in the near future.
- 4 kudos
- 14218 Views
- 2 replies
- 0 kudos
How to read a fixed length file in Spark using DataFrame API and SCALA
I have a fixed length file ( a sample is shown below) and I want to read this file using DataFrames API in Spark using SCALA(not python or java). Using DataFrames API there are ways to read textFile, json file and so on but not sure if there is a wa...
- 14218 Views
- 2 replies
- 0 kudos
- 0 kudos
Find the below solution which can be used. Let us consider this is the data in the file. EMP ID First Name Last Name 1Chris M 2John ...
- 0 kudos
- 8623 Views
- 4 replies
- 2 kudos
Hosting python application on Azure Databricks and exposing it's rest APIs
Hello, I am trying to host my application on Databricks and I want to expose rest APIs of my application to be accessed from postman but I am unable to find any documentation on how to do this. I tried to write simple flask "hello world" code to tr...
- 8623 Views
- 4 replies
- 2 kudos
- 2 kudos
I did this using Azure web app and exposed the APIs , was able to access that in Post Man and Data bricks. Not used python app on data bricks
- 2 kudos
- 3496 Views
- 1 replies
- 0 kudos
How to setup a private git repository in my workspace?
How to setup a private git repository in my workspace?
- 3496 Views
- 1 replies
- 0 kudos
- 0 kudos
As a platform engineer, I would go to the admin console and click on "workspace settings" and start by looking into the below settings. Repos: true, so that Repos integration is possibleThe next two settings, are important to make the overall experi...
- 0 kudos
- 15898 Views
- 3 replies
- 6 kudos
ConnectException: Connection refused (Connection refused) This is often caused by an OOM error
I am trying to run a python code where a json file is flattened to pipe separated file . The code works with smaller files but for huge files of 2.4 GB I get below error:ConnectException: Connection refused (Connection refused)Error while obtaining a...
- 15898 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Jose Gonzalez​ , @Werner Stinckens​ @Kaniz Fatma​ ,Thanks for your response .Appreciate a lot. The issue was in the code, it was a python /panda code running on Spark. Due to this only driver node was being used. i did validate this by increasin...
- 6 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
