Databricks Community

If you have different folders for each of your source tables, you can leverage python loops to naturally iterate over the folders.To do this, you need to create a create_pipeline function that has table_name, schema, path as your parameters. Inside t...

@felix_counter I think I have your answer.To create a databricks provider to manage your workspace using an SPN, you need to create the provider like this:provider "databricks" { alias = "workspace" host = <your workspace URL> azure_...

Hello @jura ,I'm Raphael and i'll be helping you out. The approach below should work: USE CATALOG dev; CREATE OR REPLACE TABLE IDENTIFIER("bronze.jura_test") as SELECT ... Please let me know the outcome and feel free to ask any further questions. ...

Hi, could you please refer to something or explain in more detail your point about querying Delta Lake files directly instead of through the AWS Glue catalog and why it was highlighted as a best practice?

@NDK_1 - Cron syntax won't allow the combination of day of month and day of week. you can try creating two different schedules  - one for the first day, second day of the month and then add custom logic to check if it is an working day and then trigg...

It seems that the issue you're experiencing with Autoloader not detecting changes in XML files might be related to how Autoloader handles schema inference and evolution. Autoloader can automatically detect the schema of loaded XML data, allowing you...

@SyedGhouri You need to setup self-hosted Azure DevOps Agent inside your VNET.

If you want to copy files in Azure, ADF is usually the fastest option (for example TB of csvs, parquets). If you want to copy tables, just use CLONE. If it is files with code just use Repos and branches.

Hi @MarinD, Delta Live Tables (DLT) pipelines can indeed use Unity Catalog as a destination. Here’s how you can specify the catalog and target schema: Create a DLT Pipeline with Unity Catalog: When creating a DLT pipeline, in the UI, select “Uni...

@Kaniz_Fatma Sorry if I have to ask again, but I am a bit confused by this.I thought, that pysparks `orderBy()` and `sort()` do a shuffle operation before the sorting for exact this reason. There is another command `sortWithinPartitions()` that does ...

Hi @ac0,  Let’s explore how you can determine the size of your Databricks Metastore and estimate the storage requirements for the Azure Storage Account hosting the metastore. Metastore Size: The metastore in Unity Catalog is the top-level contain...