Databricks Community

dbrx_user · 07-01-2024

Hi all,A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?What will the move to serverl...

dbrx_user · 08-06-2024

Thanks @Kaniz_Fatma - by account admin, do you mean at an Azure level? I am already a Databricks account admin and cannot see the schema

dbrx_user · 07-05-2024

Hmmm, what's your ELT setup like? Are you using ADF/ Synapse to trigger databricks notebooks? If the databricks tables are external with underlying ADLS storage, might be worth using an ADF copy activity instead?

dbrx_user · 07-03-2024

Use the OpenAI API. They have a comprehensive guide with lots of example code here: Quickstart tutorial - OpenAI APIJust remember to save your API keys as databricks secrets and access them wit dbutils.secrets - Secret management | Databricks on AWS

dbrx_user · 07-03-2024

To reduce memory spill, I'd suggest increasing number of partitions. Try playing around with 32 or 48 to see if any difference.On the jdbc part, I wonder if there is something spark doesn't like re your syntax. Might help to re-order your options so ...

dbrx_user · 07-03-2024

Agree with @Witold to apply CDC as early as possible. Depending on where the initial files get deposited, I'd recommend having an initial raw layer to your medallion which is just your cloud storage account - so each day or week the files get deposit...

Databricks Community

User Stats

User Activity

Move to 100% Serverless

Re: System Tables

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: How can I call open AI model GPT3.5 within note book using python

Re: Writing a single huge dataframe into Azure SQL Database using JDBC

Re: Optimal process for loading data where the full dataset is provided every day?