Databricks Community

datastones

Hey guys, after some successful data preprocessing without any errors, i have a final dataframe shape with the shape of ~ (200M, 150). the cluster i am using has sufficient ram + cpus + autoscaling, all metrics look fine after the job was done.The pr...

datastones · 07-08-2024

Hi everybody, I have a question re: the deployment as code pattern on databricks. I found and watched a great demo here: https://www.youtube.com/watch?v=JApPzAnbfPIMy question is, in the case where I can get read access to prod data in dev env, the d...

datastones · 07-03-2024

Hi everybody. I am relatively new to Databricks. I am working on an ML model promotion process between different Databricks workspaces. I am aware that best practice should be deployment as code (e.g. export the whole training pipeline and model regi...

datastones

i think the dab mlop stack template is pretty helpful re: how to bundle, schedule and trigger custom jobshttps://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.htmlyou can bundle init it locally and it should give you the skeleton of how to bu...

datastones

you should create an IAM assumed role, adding the principal as the aws account that hosts your databricks env.

datastones

@szymon_dybczak i could resolve it now! basically, i broke the process down into further subprocesses, for each sub process, i cached and wrote them all into delta table (without overwritting), the next subprocess needs to read data in the delta tabl...

datastones

1 interesting thing that I observed when playing around with different repartition size: 2,4,8,16,32,64. i have 32 cores per node. The total written result increased up to 65% of the final df at repartition = 8 (but this is very inconsistent and hard...

datastones

Hey @szymon_dybczak, i appreciate your prompt response! yup, I check the dataframe shape and show some output along the way prior to writing it out. the shape looks checked out right before i perform the write... what a mystery this is!

Databricks Community

User Stats

User Activity

Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Deployment as code pattern with double training effort?

ML model promotion from Databricks dev workspace to prod workspace

Re: Asset Bundles from Workspace for CI/CD

Re: Connecting to redshift from databricks notebooks without password using IAM

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog

Re: Data loss after writing a transformed pyspark dataframe to delta table in unity catalog