Databricks Community

datastones · 09-26-2024

Hey guys, after some successful data preprocessing without any errors, i have a final dataframe shape with the shape of ~ (200M, 150). the cluster i am using has sufficient ram + cpus + autoscaling, all metrics look fine after the job was done.The pr...

datastones · 07-08-2024

Hi everybody, I have a question re: the deployment as code pattern on databricks. I found and watched a great demo here: https://www.youtube.com/watch?v=JApPzAnbfPIMy question is, in the case where I can get read access to prod data in dev env, the d...

datastones · 07-03-2024

Hi everybody. I am relatively new to Databricks. I am working on an ML model promotion process between different Databricks workspaces. I am aware that best practice should be deployment as code (e.g. export the whole training pipeline and model regi...

datastones · 10-01-2024

i think the dab mlop stack template is pretty helpful re: how to bundle, schedule and trigger custom jobshttps://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.htmlyou can bundle init it locally and it should give you the skeleton of how to bu...

datastones · 09-30-2024

you should create an IAM assumed role, adding the principal as the aws account that hosts your databricks env.

datastones · 09-27-2024

@szymon_dybczak i could resolve it now! basically, i broke the process down into further subprocesses, for each sub process, i cached and wrote them all into delta table (without overwritting), the next subprocess needs to read data in the delta tabl...

datastones · 09-26-2024

1 interesting thing that I observed when playing around with different repartition size: 2,4,8,16,32,64. i have 32 cores per node. The total written result increased up to 65% of the final df at repartition = 8 (but this is very inconsistent and hard...

datastones · 09-26-2024

Hey @szymon_dybczak, i appreciate your prompt response! yup, I check the dataframe shape and show some output along the way prior to writing it out. the shape looks checked out right before i perform the write... what a mystery this is!

Databricks Community

User Stats

User Activity

Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Deployment as code pattern with double training effort?

ML model promotion from Databricks dev workspace to prod workspace

Re: Asset Bundles from Workspace for CI/CD

Re: Connecting to redshift from databricks notebooks without password using IAM

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog