Databricks Community

matthew_m

SQL is a first-class language in SDP SQL is a first-class language in Spark Declarative Pipelines (SDP). You can build, test, and ship an entire pipeline in SQL. This makes it a powerful tool for those with a SQL background, as SDP enables both batch...

matthew_m · 09-12-2023

Learn to build fast, stateful pipelines for operational workloads. Discover stateless vs. stateful streams, how to setup your cluster and more. Get hands-on building a pipeline with code snippets and access to the repo. Start your real-time journey n...

matthew_m · 10-24-2024

Hi @ChristopherQ1, Salesforce has released a zero-copy connection that relies on the SQL Warehouse to ingest data when needed. I suggest you consider that instead of OData. Matthew

matthew_m · 05-22-2024

Hi @Sambit_S , I misread inputRows as inputFiles which aren't the same thing. Considering the limitation on Azure queue, if you are already at the limit then you may need to consider to switching to an event source such as Kafka or Event Hub to get b...

matthew_m · 05-21-2024

Hi @Sambit_S , From that last log I can see that it ingested 998 files, which is close to the expected default max, which appears to be set in your code. maxBytesPerTrigger and maxFilesPerTrigger work together when defining the soft upper bound max. ...

matthew_m · 05-21-2024

Hi @Sambit_S, File notification would only impact any new arriving files, so unless you have a backfill interval to load any existing data it will not load existing data. You could also backfill initially by using directory mode and then switch to fi...

matthew_m · 10-12-2023

This is an old post but still relevant for future readers, so will answer how it is done. You need to add base_parameters flag in the notebook_task config, like the following. "notebook_task": { "notebook_path": "...", "base_parameters": { ...

Databricks Community

User Stats

User Activity

How to Build, Test, and Ship SQL Pipelines on Databricks

How to build operational low-latency, stateful Spark Structured Streaming pipelines

Re: Can we share Delta table data with Salesforce using OData?

Re: Databricks Autoloader File Notification Not Working As Expected

Re: Databricks Autoloader File Notification Not Working As Expected

Re: Databricks Autoloader File Notification Not Working As Expected

Re: How to create a databricks job with parameters via CLI?