- 5087 Views
- 2 replies
- 1 kudos
Resolved! Docker image with libraries + MLFlow Experiments
Hi everybody,I have a scenario where we have multiple teams working with Python and R, and this teams uses a lot of different libraries. Because of this dozen of libraries, the cluster start took much time. Then I created a Docker image, where I can ...
- 5087 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Fabio Simoes​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 8483 Views
- 2 replies
- 2 kudos
Resolved! "Photon ran out of memory" while when trying to get the unique Id from sql query
I am trying to get all unique id from sql query and I always run out of memoryselect concat_ws(';',view.MATNR,view.WERKS) from hive_metastore.dqaas.temp_view as view join hive_metastore.dqaas.t_dqaas_marc as marc on marc.MATNR = view.MATNR where view...
- 8483 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Anil Kumar Chauhan​ We haven't heard from you since the last response from @Werner Stinckens​ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 2 kudos
- 7770 Views
- 4 replies
- 8 kudos
Resolved! Issue in Converting Pyspark Dataframe to dictionary
I have 3 questions listed below.Q1. I need to install third party library in Unity Catalog enabled shared cluster. But I am not able to install. It is not accepting dbfs path dbfs:/FileStore/jars/Q2. I have a requirement to load the data to salesforc...
- 7770 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @SK ASIF ALI​ We haven't heard from you since the last response from @werners (Customer)​ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 8 kudos
- 3993 Views
- 2 replies
- 0 kudos
Running Keras model training with HorovodRunner works until the training function is exited ("The MPI_Query_thread() function was called after MPI_FINALIZE was invoked.")
I am running training of a Keras/Tensorflow deep learning model on a cluster of (for now) 2 workers and 1 driver (T4 GPU, 28GB, 4 core) using the Databricks provided HorovodRunner. It all seems to go well and the performance scales quite nicely over ...
- 3993 Views
- 2 replies
- 0 kudos
- 0 kudos
I personally suspect it's your callbacks. Can you remove all those state callbacks and see if that is it?
- 0 kudos
- 24338 Views
- 1 replies
- 4 kudos
Resolved! [Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP Response code: 403
I am trying to connect to databricks using java code. Can someone help me please? Here is the code so far I have got:: import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.Properties; ...
- 24338 Views
- 1 replies
- 4 kudos
- 4 kudos
@Dipak Bachhav​ do you have any restriction in terms if IP to access databricks, in case of that you need to enable particular ip from security groups
- 4 kudos
- 7110 Views
- 2 replies
- 6 kudos
Resolved! Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"
In my ETL case, I want to be able to adjust the table schema as needed, meaning the number of columns may increase or decrease depending on the ETL script. Additionally, I would like to use dynamic partition overwrite to avoid potential errors when u...
- 7110 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @Thanapat Sontayasara​,Does @Werner Stinckens​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? If not, would you be happy to give us more information?Thanks!
- 6 kudos
- 11608 Views
- 5 replies
- 6 kudos
Resolved! Merge 12 CSV files in Databricks.
Hello everybody,I am absolutely new in Databricks, so I need your help.Details:Task: merge 12 CSV files in Databricks with the best way.Location of files: I will describe it in details, because I can not good orientate yet. If i go to Data -> Browse ...
- 11608 Views
- 5 replies
- 6 kudos
- 6 kudos
It seems that all your csv files are present under one folder and since you are able to union them, all these files must have same schema as well.Given the above conditions, you can simply read all the data by referring the folder name instead of ref...
- 6 kudos
- 4761 Views
- 2 replies
- 4 kudos
Runtime error using MLFlow and Spark on databricks
Here is some model I created:class SomeModel(mlflow.pyfunc.PythonModel): def predict(self, context, input): # do fancy ML stuff # log results pandas_df = pd.DataFrame(...insert predictions here...) spark_df = spark...
- 4761 Views
- 2 replies
- 4 kudos
- 4 kudos
Any updates on this? I am running into the same issue@Patrick Tawil​ were you able to solve this problem? If so, do you mind sharing?
- 4 kudos
- 2588 Views
- 1 replies
- 1 kudos
Resolved! Comparative study of Azure Databricks MLOps capabilities in conjuction with Azuredevops, GIT, Jenkins
Looking for Comparative study of capabilities of below tools combination. In what situation I should use which of the below combination for MLOps project?a) Azure Databricks MLb) Azure Databricks ML + Azure Devops + GITc) Azure Databricks ML + Jenkin...
- 2588 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @saurabh707344, you can use Azure Databricks ML when you're in the initial stages and developing some POCs. The other tools you mentioned were used based on your usecase when you moved some of the models to production and actively developing and ...
- 1 kudos
- 2083 Views
- 1 replies
- 1 kudos
Databricks exam got suspended
today my databricks exam got suspended in the middle, the support team said they cannot reschdule my exam and asked me to contact my test sponsor i mailed tem many times but i haven't receied any reply can u please help with this.
- 2083 Views
- 1 replies
- 1 kudos
- 1 kudos
Adding @Suteja Kanuri​ , @Kaniz Fatma​ and @Vidula Khanna​ for visibility, to help you with your request
- 1 kudos
- 4315 Views
- 2 replies
- 3 kudos
Resolved! ImportError: cannot import name 'FMIN_CANCELLED_REASON_EARLY_STOPPING' from 'hyperopt.spark'
I am trying to run an automl experiment from notebook. The error occurs when I import automl from databricks. Finally the error points to hyperopt. I have tried using the older version of hyperopt as well but nothings working.
- 4315 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Salar Ahmad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 3 kudos
- 10105 Views
- 2 replies
- 2 kudos
Resolved! Invalid catalog and schema for table name error when creating a Feature Store
I've created a dataset that I want to create a feature store table for. I created a database in my dev unity catalog to store the feature tables, but each time I try to create the table I get this error:ValueError: Invalid catalog 'dev' or schema 'fe...
- 10105 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello,The error message suggests that there might be an issue with the catalog or schema name you are using when trying to create the feature store table. Here are a few tips to help you troubleshoot this problem:Verify the catalog and schema names: ...
- 2 kudos
- 14972 Views
- 5 replies
- 2 kudos
Trying out Dolly - how to load pytorch_model.bin so it's not downloading it every time the cluster is restarted
Hi, I am new to LLM and am curious to try it out. I did the following code to test from the databricks website:import torch from transformers import pipeline instruct_pipeline = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, tr...
- 14972 Views
- 5 replies
- 2 kudos
- 2 kudos
Just set the HF cache dir to a persistent path on /dbfs:import os os.environ['TRANSFORMERS_CACHE'] = "/dbfs/..."
- 2 kudos
- 1640 Views
- 0 replies
- 0 kudos
I found these videos to be beneficial CI/CD with Azure Dev OpsCI/CD with GitHub Actions
I found these videos to be beneficialCI/CD with Azure Dev OpsCI/CD with GitHub Actions
- 1640 Views
- 0 replies
- 0 kudos
- 4491 Views
- 4 replies
- 3 kudos
Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.
hi,We have Real time streaming usecase where we have to build pipeline using Azure Databricks.My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.Can you please provide c...
- 4491 Views
- 4 replies
- 3 kudos
- 3 kudos
I found these youtube videos to be beneficial. CI/CD with Azure Dev Ops Terraform Enablement - Part 1 of 2
- 3 kudos
