- 2944 Views
- 2 replies
- 0 kudos
TF SummaryWriter flush() don't send any buffered data to storage.
Hey guys, I'm training a TF model in databricks, and logging to tensorboard using SummaryWriter. At the end of each epoch SummaryWriter.flush() is called which should send any buffered data into storage. But i can't see the tensorboard files while th...
- 2944 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @orian hindi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
- 0 kudos
- 5696 Views
- 2 replies
- 1 kudos
Is there a way to change the default artifact store path on Databricks Mlflow?
I have a cloud storage mounted to Databricks and I would like to store all of the model artifacts there without specifying it when creating a new experiment.Is there a way to configure the Databricks workspace to save all of the model artifacts to a ...
- 5696 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Eero Hiltunen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 1 kudos
- 7667 Views
- 1 replies
- 1 kudos
Resolved! Using databricks in multi-cloud, and querying data from the same instance.
I'm looking for a good product to use across two clouds at once for Data Engineering, Data modeling and governance. I currently have a GCP platform, but most of my data and future data goes through Azure, and currently is then transfered to GCS/BQ.Cu...
- 7667 Views
- 1 replies
- 1 kudos
- 1 kudos
@Karl Andrén​ :Databricks is a great option for data engineering, data modeling, and governance across multiple clouds. It supports integrations with multiple cloud providers, including Azure, AWS, and GCP, and provides a unified interface to access ...
- 1 kudos
- 2525 Views
- 1 replies
- 7 kudos
Have you heard about databricks latest open-source language model called Dolly? It’s a ChatGPT like model that uses the tatsu-lab/alpaca dataset with ...
Have you heard about databricks latest open-source language model called Dolly? It’s a ChatGPT like model that uses the tatsu-lab/alpaca dataset with examples of questions and answers. To train Dolly, you can combine this dataset (simple solution on ...
- 2525 Views
- 1 replies
- 7 kudos
- 7 kudos
Thanks for posting this! I am so excited about the possibilities that this can do for us. It's an exciting development in the natural language processing field, and it has the potential to be a valuable tool for businesses looking to implement chatb...
- 7 kudos
- 5461 Views
- 2 replies
- 1 kudos
Model serving with GPU cluster
Hello Databricks community!We are facing a strong need of serving some of public and our private models on GPU clusters and we have several requirements:1) We'd like to be able to start/stop the endpoints (best with scheduling) to avoid excess consum...
- 5461 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Alisher Akh​ Does @Debayan Mukherjee​'s answer help? If yes, would you be happy to mark the answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you further. Cheers!
- 1 kudos
- 20287 Views
- 5 replies
- 7 kudos
Resolved! What does "Command exited with code 50 mean" and how do you solve it?
Hi!We have this dbt model that generates a table with user activity in the previous days, but we get this vague error message in the Databricks SQL Warehouse.Job aborted due to stage failure: Task 3 in stage 4267.0 failed 4 times, most recent failure...
- 20287 Views
- 5 replies
- 7 kudos
- 7 kudos
@Mattias P​ - For the executor lost failure, is it trying to bring in large data volume? can you please reduce the date range and try? or run the workload on a bigger DBSQL warehouse than the current one.
- 7 kudos
- 5195 Views
- 2 replies
- 5 kudos
Share information between tasks in a Databricks job  You can use task values to pass arbitrary parameters between tasks in a Databricks job. You pass ...
Share information between tasks in a Databricks jobYou can use task values to pass arbitrary parameters between tasks in a Databricks job. You pass task values using the taskValues subutility in Databricks Utilities. The taskValues subutility provide...
- 5195 Views
- 2 replies
- 5 kudos
- 5 kudos
We urgently hope for this feature, but to date, we have found that it is only available in Python. Do you have any plans to support Scala?
- 5 kudos
- 9825 Views
- 2 replies
- 0 kudos
Resolved! How to resolve this error "Error: cannot create global init script: default auth: cannot configure default credentials"
I'm trying to set the global init script via my Terraform deployment. I did a thorough google search and can't seem to find guidance here.I'm using a very generic call to set these scripts in my TF Deployment.terraform { required_providers { data...
- 9825 Views
- 2 replies
- 0 kudos
- 0 kudos
Ok in case this helps anyone else, I've managed to resolve.I confirmed in this documentation the databricks CLI is required locally, wherever this is being executed. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cluster-note...
- 0 kudos
- 32004 Views
- 4 replies
- 7 kudos
The Python process exited with exit code 137 (SIGKILL: Killed). This may have been caused by an OOM error. Check your command's memory usage.
I am running a hugging face model on a GPU cluster (g4dn.xlarge, 16GB Memory, 4 cores). I run the same model in four different notebooks with different data sources. I created a workflow to run one model after the other. These notebooks run fine indi...
- 32004 Views
- 4 replies
- 7 kudos
- 7 kudos
You might accumulate gradients when running your Huggingface model, which typically leads to out-of-memory errors after some iterations. If you use it for inference only, dowith torch.no_grad(): # The code where you apply the model
- 7 kudos
- 6338 Views
- 3 replies
- 3 kudos
Resolved! MLFlow: How to load results from model and continue training
I'd like to continue / finetune training of an existing keras/tensorflow model. We use MLFlow to store the model. How can I load the wieght from an existing model to the model and continue "fit" preferable with a different learning rate.Just loading ...
- 6338 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Tilo Wünsche​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 3 kudos
- 6973 Views
- 4 replies
- 5 kudos
Resolved! Error loading model from mlflow: java.io.StreamCorruptedException: invalid type code: 00
Hello,I'm using, in my IDE, Databricks Connect version 9.1LTS ML to connect to a databricks cluster with spark version 3.1 and download a spark model that's been trained and saved using mlflow.So it seems like it's able to find a copy the model, but ...
- 6973 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Kaniz Fatma​ and @Shanmugavel Chandrakasu​,It works after putting hadoop.dll into C:\Windows\System32 folder.I have hadoop version 3.3.1.I already had winutils.exe in the Hadoop bin folder.RegardsNath
- 5 kudos
- 6675 Views
- 7 replies
- 4 kudos
- 6675 Views
- 7 replies
- 4 kudos
- 9093 Views
- 2 replies
- 2 kudos
How to download a .csv or .pkl file from databricks?
When I save files on "dbfs:/FileStore/shared_uploads/brunofvn6@gmail.com/", it doesn't appear anywhere in my workspace. I've tried to copy the path of the workspace with the right mouse button, pasted on ("my pandas dataframe").to_csv('path'), but wh...
- 9093 Views
- 2 replies
- 2 kudos
- 2 kudos
I think I discover how to do this. Is in the label called data in the left menu of the databricks environment, in the top left of the menu there are two labels "Database Tables" and "DBFS" in which "Database Table" is the default label. So it is just...
- 2 kudos
- 3086 Views
- 1 replies
- 2 kudos
Resolved! File not found error. Does OPTIMIZE deletes initial versions of the delta table?
df = (spark.readStream.format("delta")\ .option("readChangeFeed", "true")\ .option("startingVersion", 1)\ .table("CatalogName.SchemaName.TableName") )display(df)A file referenced in the transaction l...
- 3086 Views
- 1 replies
- 2 kudos
- 2 kudos
Had you run vacuum on the table? Vacuum can clean up data files marked for removal and are older than retention period.Optimize compacts files and marks the small files for removal, but does not physically remove the data files
- 2 kudos
- 4113 Views
- 1 replies
- 2 kudos
Free Databricks Training on AWS, Azure, or Google Cloud Good news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud...
Free Databricks Training on AWS, Azure, or Google CloudGood news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud. Our on-demand training series walks through how to:Streamline data ingest and management to build ...
- 4113 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing this!!
- 2 kudos
