Databricks Community

dbrx_user

Hi all,A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?What will the move to serverl...

dbrx_user

Hmmm, what's your ELT setup like? Are you using ADF/ Synapse to trigger databricks notebooks? If the databricks tables are external with underlying ADLS storage, might be worth using an ADF copy activity instead?

dbrx_user

Use the OpenAI API. They have a comprehensive guide with lots of example code here: Quickstart tutorial - OpenAI APIJust remember to save your API keys as databricks secrets and access them wit dbutils.secrets - Secret management | Databricks on AWS

dbrx_user

To reduce memory spill, I'd suggest increasing number of partitions. Try playing around with 32 or 48 to see if any difference.On the jdbc part, I wonder if there is something spark doesn't like re your syntax. Might help to re-order your options so ...

dbrx_user

Agree with @Witold to apply CDC as early as possible. Depending on where the initial files get deposited, I'd recommend having an initial raw layer to your medallion which is just your cloud storage account - so each day or week the files get deposit...

dbrx_user

Hey, a few suggestions to try below!Is there a reason you're using coalesce instead of repartition? Coalesce can result in uneven sized partitions which are slower to work with (even though the initial function might be faster).Also, why are you spec...

Databricks Community

User Stats

User Activity

Move to 100% Serverless

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: How can I call open AI model GPT3.5 within note book using python

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: Optimal process for loading data where the full dataset is provided every day?

Re: Writing a single huge dataframe into Azure SQL Database using JDBC