- 2885 Views
- 1 replies
- 3 kudos
Retry api request if fails
I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place? Here is an api request:d...
- 2885 Views
- 1 replies
- 3 kudos
- 3 kudos
@Bartosz Wachocki​ ,Use timeout, retry interval ,recursion and exception handling pseudo code belowtimeout = 300def exec_query(query,timeout): try: df = spark.createDataFrame(sf.bulk.MyTable.query(query)) except: if timeout > 0 : sleep(60) exec_que...
- 3 kudos
- 3173 Views
- 2 replies
- 2 kudos
Resolved! mail configuration azure data bricks pyspark notebook
Hi All,i am new to azure databricks , i am using pyspark .. we need to configure mail alerts when notebook failed or succeeded ..please can some one help me in mail configuration azure data bricks .Thanks
- 3173 Views
- 2 replies
- 2 kudos
- 2 kudos
the easiest way to schedule notebooks in Azure is to use Data Factory.In Data Factory you can schedule the notebooks and define the alerts you want to send.The other option is the one Hubert mentioned.
- 2 kudos
- 8470 Views
- 2 replies
- 9 kudos
databricks-connect' is not recognized as an internal or external command, operable program or batch file on windows
Hello,I've installed databricks-connect on Windows 10:C:\Users\danoshin>pip install -U "databricks-connect==9.1.*" Collecting databricks-connect==9.1.* Downloading databricks-connect-9.1.2.tar.gz (254.6 MB) |████████████████████████████████| 2...
- 8470 Views
- 2 replies
- 9 kudos
- 9 kudos
@Dmitry Anoshin​ , that seems messed up.the best you can do is to remove databricks connect and also to uninstall any pyspark installation.And then follow the installation guide.It should work after following the procedure.I use a Linux VM for this p...
- 9 kudos
- 11568 Views
- 4 replies
- 3 kudos
Resolved! Use of private endpoints for storage in workspace with EnableNoPublicIP=Yes and VnetInjection=No
We know that Databricks with VNET injection (our own VNET) allows is to connect to ADLS Gen2 over private endpoints. This is what we typically do.We have a customer who created Databricks with EnableNoPublicIP=Yes (secure cluster connectivity) and Vn...
- 11568 Views
- 4 replies
- 3 kudos
- 3 kudos
Managed VNET is locked and allows very limited config tuning like VNET peering that too facilitated and needs to be done from Databricks UI. If they want more control on VNET they need to migrate to VNET injected workspace.
- 3 kudos
- 8159 Views
- 9 replies
- 8 kudos
Resolved! Is there a way to persist the delta cache even after the cluster restart?
Hi Team, we are planning to connect Power BI directly to Data bricks, however data fetching using direct query isn't giving great performance, though we are using Zorder by and Partition etc.. We decided to use Delta Cache, but the cache tables area ...
- 8159 Views
- 9 replies
- 8 kudos
- 8 kudos
@Hubert Dudek​ , I got a good news, I agree with @Werner Stinckens​ , SQL End Point is super fast, I have tested for 143 million records with Direct Query from power bi, result returned in 10-12 seconds. Don't even try doing incremental in power bi, ...
- 8 kudos
- 19677 Views
- 7 replies
- 5 kudos
Resolved! [SOLVED] maxPartitionBytes ignored?
Hello all!I'm running a simple read noop query where I read a specific partition of a delta table that looks like this:With the default configuration, I read the data in 12 partitions, which makes sense as the files that are more than 128MB are split...
- 19677 Views
- 7 replies
- 5 kudos
- 5 kudos
AQE doesn't affect the read time partitioning but at the shuffle time. It would be better to run optimize on the delta lake which will compact the files to approx 1 GB each, it would provide better read time performance.
- 5 kudos
- 3545 Views
- 4 replies
- 1 kudos
Resolved! Reply on inline runtime commands
I feel like the answer to this question should be simple, but none the less I'm struggling.I run a python code that prompts me with the following warning:On my local machine, I can accept this through my terminal and my machine do not run out of memo...
- 3545 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Nickels Köhling​ ,In Databricks, you will only be able to see the output in the driver logs. If you go to your driver logs, you will be able to see 3 windows that are displaying the output of "stdout", "stderr" and "log4j".If in your code you do ...
- 1 kudos
- 5643 Views
- 3 replies
- 0 kudos
How to make terminal available when starting a cluster with a docker image?
Hello everybody, I'm starting a cluster with a docker image because I my team use a lot of R libraries that would be time consuming to install them on a init script. The thing is that the cluster starts ok, but I can't acces to the cluster terminal a...
- 5643 Views
- 3 replies
- 0 kudos
- 0 kudos
@Francesco Camussoni​ are you using databricksruntime/rbase as your base image?
- 0 kudos
- 4201 Views
- 1 replies
- 2 kudos
- 4201 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Yatharth Kaushik​ ,You can get the data into a table using the clusters event API: https://docs.databricks.com/dev-tools/api/latest/clusters.html#events
- 2 kudos
- 3526 Views
- 3 replies
- 5 kudos
Serverless SQL endpoints on Azure?
Serverless SQL Endpoints sound exciting! Sounds like they've been in preview on AWS for a couple months. Any idea if/when they're coming to Azure?
- 3526 Views
- 3 replies
- 5 kudos
- 6325 Views
- 2 replies
- 5 kudos
Resolved! Reference py file from a notebook
Hi All,I'm trying to reference a py file from a notebook following this documentation: Files in repoI downloaded and added the files to my repo and when I try to run the notebook, the modules is not recognized: Any idea why is this happening? Thanks ...
- 6325 Views
- 2 replies
- 5 kudos
- 5 kudos
In this topic you can find some more info:https://community.databricks.com/s/question/0D53f00001Pp5EhCAJThe docs are not that clear.
- 5 kudos
- 8462 Views
- 6 replies
- 4 kudos
Resolved! Copy a script from the current subscription to new subscription
I would like to check if there is a process to copy a script/code or migrate the script from the current subscription of the Azure Databricks - Notebooks to new subscription of Databricks (new notebook).
- 8462 Views
- 6 replies
- 4 kudos
- 4 kudos
how quickly move the Databricks notebooks from one account to another?
- 4 kudos
- 8055 Views
- 5 replies
- 1 kudos
Resolved! ciso8601 library stopped installing out of the blue on DB clusters
We have multiple DB clusters (6.4 Extended Support) that have not changed in terms of libs installed or nodes etc. Sudden from one day to the other, after a cluster restart August 7th, they stopped installing ciso8601 lib as they would usually. Anyb...
- 8055 Views
- 5 replies
- 1 kudos
- 1 kudos
Just to close this old qustion:We solved this by switching to a PEP517 free pip install, using the a Global Init Script:/databricks/python/bin/pip install ciso8601 --disable-pip-version-check --no-use-pep517Now it works for us.
- 1 kudos
- 12010 Views
- 2 replies
- 3 kudos
Please let me know how i can install PyAudio using the Databricks notebook
Hi,i am trying to install the PyAudio package.but i am getting the following error. Collecting pyaudio Using cached PyAudio-0.2.11.tar.gz (37 kB)Building wheels for collected packages: pyaudio Building wheel for pyaudio (setup.py) ... error ERROR: Co...
- 12010 Views
- 2 replies
- 3 kudos
- 3 kudos
looks like missing dependencies on the server (linux): portaudioThis should be installed:https://stackoverflow.com/questions/48690984/portaudio-h-no-such-file-or-directory
- 3 kudos
- 3592 Views
- 1 replies
- 0 kudos
Set tags for an MLFlow Experiment using Python?
There is this rest API: https://www.mlflow.org/docs/latest/rest-api.html#set-experiment-tagCan I do the same from python's MLFlow API?
- 3592 Views
- 1 replies
- 0 kudos
- 0 kudos
Someone answered first in StackOverflow. Here it is:from mlflow.tracking import MlflowClient # Create an experiment with a name that is unique and case sensitive. client = MlflowClient() experiment_id = client.create_experiment("Social NLP Experime...
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 518 | |
| 349 | |
| 287 |
