- 20147 Views
- 6 replies
- 5 kudos
Resolved! Can you help with this error please? Issue when using a new high concurrency cluster
Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...
- 20147 Views
- 6 replies
- 5 kudos
- 5 kudos
@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false
- 5 kudos
- 22583 Views
- 8 replies
- 2 kudos
Resolved! How many hours I can estimate to trainning in a Databricks Academy Self-Placed Trainning platform ?
I done the Data Engineering Profissional and others training in a Self-Placed Trainning (https://www.linkedin.com/posts/wscardua_data-engineering-professional-activity-6851487238774108160-IsTE) . How many hours can I estimate for this training (and o...
- 22583 Views
- 8 replies
- 2 kudos
- 3013 Views
- 2 replies
- 4 kudos
Multi-task Job Run starting point
Hi community!I would like to know if it is possible to start a Multi-task Job Run from and specific task. The use case is as follows:I have a 17 tasks JobA task in the middle, let's say a task after 2 dependencies, failsI found the error and now it i...
- 3013 Views
- 2 replies
- 4 kudos
- 4 kudos
+1 to what @Dan Zafar​ said. We're working **** ** this. Looking forward to bring this to you in the near future.
- 4 kudos
- 15520 Views
- 2 replies
- 0 kudos
How to read a fixed length file in Spark using DataFrame API and SCALA
I have a fixed length file ( a sample is shown below) and I want to read this file using DataFrames API in Spark using SCALA(not python or java). Using DataFrames API there are ways to read textFile, json file and so on but not sure if there is a wa...
- 15520 Views
- 2 replies
- 0 kudos
- 0 kudos
Find the below solution which can be used. Let us consider this is the data in the file. EMP ID First Name Last Name 1Chris M 2John ...
- 0 kudos
- 9691 Views
- 4 replies
- 2 kudos
Hosting python application on Azure Databricks and exposing it's rest APIs
Hello, I am trying to host my application on Databricks and I want to expose rest APIs of my application to be accessed from postman but I am unable to find any documentation on how to do this. I tried to write simple flask "hello world" code to tr...
- 9691 Views
- 4 replies
- 2 kudos
- 2 kudos
I did this using Azure web app and exposed the APIs , was able to access that in Post Man and Data bricks. Not used python app on data bricks
- 2 kudos
- 4347 Views
- 1 replies
- 0 kudos
How to setup a private git repository in my workspace?
How to setup a private git repository in my workspace?
- 4347 Views
- 1 replies
- 0 kudos
- 0 kudos
As a platform engineer, I would go to the admin console and click on "workspace settings" and start by looking into the below settings. Repos: true, so that Repos integration is possibleThe next two settings, are important to make the overall experi...
- 0 kudos
- 18442 Views
- 3 replies
- 6 kudos
ConnectException: Connection refused (Connection refused) This is often caused by an OOM error
I am trying to run a python code where a json file is flattened to pipe separated file . The code works with smaller files but for huge files of 2.4 GB I get below error:ConnectException: Connection refused (Connection refused)Error while obtaining a...
- 18442 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Jose Gonzalez​ , @Werner Stinckens​ @Kaniz Fatma​ ,Thanks for your response .Appreciate a lot. The issue was in the code, it was a python /panda code running on Spark. Due to this only driver node was being used. i did validate this by increasin...
- 6 kudos
- 7970 Views
- 4 replies
- 4 kudos
Resolved! Is there any way of determining last stage of SparkSQL Application Execution?
I have created custom UDF's that generate logs. These logs can be flushed by calling another API exposed which is exposed by an internal layer. However I want to call this API just after the execution of the UDF comes to an end. Is there any way of d...
- 7970 Views
- 4 replies
- 4 kudos
- 4 kudos
@Krishna Kashiv​ May be ExecutorPlugin.java can help. It has all the methods you might required. Let me know if it works or not.You need to implement this interface org.apache.spark.api.plugin.SparkPluginand expose it as spark.plugins = com.abc.Imp...
- 4 kudos
- 3287 Views
- 1 replies
- 3 kudos
Retry api request if fails
I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place? Here is an api request:d...
- 3287 Views
- 1 replies
- 3 kudos
- 3 kudos
@Bartosz Wachocki​ ,Use timeout, retry interval ,recursion and exception handling pseudo code belowtimeout = 300def exec_query(query,timeout): try: df = spark.createDataFrame(sf.bulk.MyTable.query(query)) except: if timeout > 0 : sleep(60) exec_que...
- 3 kudos
- 3768 Views
- 2 replies
- 2 kudos
Resolved! mail configuration azure data bricks pyspark notebook
Hi All,i am new to azure databricks , i am using pyspark .. we need to configure mail alerts when notebook failed or succeeded ..please can some one help me in mail configuration azure data bricks .Thanks
- 3768 Views
- 2 replies
- 2 kudos
- 2 kudos
the easiest way to schedule notebooks in Azure is to use Data Factory.In Data Factory you can schedule the notebooks and define the alerts you want to send.The other option is the one Hubert mentioned.
- 2 kudos
- 9255 Views
- 2 replies
- 9 kudos
databricks-connect' is not recognized as an internal or external command, operable program or batch file on windows
Hello,I've installed databricks-connect on Windows 10:C:\Users\danoshin>pip install -U "databricks-connect==9.1.*" Collecting databricks-connect==9.1.* Downloading databricks-connect-9.1.2.tar.gz (254.6 MB) |████████████████████████████████| 2...
- 9255 Views
- 2 replies
- 9 kudos
- 9 kudos
@Dmitry Anoshin​ , that seems messed up.the best you can do is to remove databricks connect and also to uninstall any pyspark installation.And then follow the installation guide.It should work after following the procedure.I use a Linux VM for this p...
- 9 kudos
- 14552 Views
- 4 replies
- 3 kudos
Resolved! Use of private endpoints for storage in workspace with EnableNoPublicIP=Yes and VnetInjection=No
We know that Databricks with VNET injection (our own VNET) allows is to connect to ADLS Gen2 over private endpoints. This is what we typically do.We have a customer who created Databricks with EnableNoPublicIP=Yes (secure cluster connectivity) and Vn...
- 14552 Views
- 4 replies
- 3 kudos
- 3 kudos
Managed VNET is locked and allows very limited config tuning like VNET peering that too facilitated and needs to be done from Databricks UI. If they want more control on VNET they need to migrate to VNET injected workspace.
- 3 kudos
- 9719 Views
- 9 replies
- 8 kudos
Resolved! Is there a way to persist the delta cache even after the cluster restart?
Hi Team, we are planning to connect Power BI directly to Data bricks, however data fetching using direct query isn't giving great performance, though we are using Zorder by and Partition etc.. We decided to use Delta Cache, but the cache tables area ...
- 9719 Views
- 9 replies
- 8 kudos
- 8 kudos
@Hubert Dudek​ , I got a good news, I agree with @Werner Stinckens​ , SQL End Point is super fast, I have tested for 143 million records with Direct Query from power bi, result returned in 10-12 seconds. Don't even try doing incremental in power bi, ...
- 8 kudos
- 23728 Views
- 7 replies
- 5 kudos
Resolved! [SOLVED] maxPartitionBytes ignored?
Hello all!I'm running a simple read noop query where I read a specific partition of a delta table that looks like this:With the default configuration, I read the data in 12 partitions, which makes sense as the files that are more than 128MB are split...
- 23728 Views
- 7 replies
- 5 kudos
- 5 kudos
AQE doesn't affect the read time partitioning but at the shuffle time. It would be better to run optimize on the delta lake which will compact the files to approx 1 GB each, it would provide better read time performance.
- 5 kudos
- 4515 Views
- 4 replies
- 1 kudos
Resolved! Reply on inline runtime commands
I feel like the answer to this question should be simple, but none the less I'm struggling.I run a python code that prompts me with the following warning:On my local machine, I can accept this through my terminal and my machine do not run out of memo...
- 4515 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Nickels Köhling​ ,In Databricks, you will only be able to see the output in the driver logs. If you go to your driver logs, you will be able to see 3 windows that are displaying the output of "stdout", "stderr" and "log4j".If in your code you do ...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
