Hi @MathewSa ,if you take the self paced course of databricks generative AI engineering course that i available at Databricks Academy website , you can run some of the LLM notebooks on databricks community edition with  Databricks runtime(s): 13.3.x-...

Hi @pavansharma36, Actually, you cannot install the same version a jar/package more than once. If for some reason, your existing package needs to be re-installed, remove it, restart your cluster and install it again.To have more context :Are you usin...

Hi, There is a way to retain the copy of data frame, even if the data in underling table is manipulated but that's a memory expensive operation, be careful while using it.df1 = spark.createDataFrame(df.rdd.map(lambda x: x), schema=df.schema)Here we a...

Hey, at my team we are faced with the same problem.Given that MLflow usage is part of the "Scalable Machine Learning with Apache Spark" course I got the impression that Spark ML models would work well with MLflow and Databricks model logging, but it'...

Thanks for the information

Hi @GeKo ,     You can get it from the UC's List tables (GET /api/2.1/unity-catalog/tables) API for now. That API response's payload has the following field: "storage_location": "string"   In future, this should be supported by Databricks as storage ...

When working with a pyspark.pandas.frame.DataFrame object and needing to apply transformations to grouped data based on a specific column, you can utilize the groupby method followed by the apply function. This way allows you to group the data based ...