Databricks Community

anardinelli

If you're trying to run the Ataccama profiler on tables with multiple joins or are incredibly large, please note that there are some processes on the Ataccama profiler that will lead to bad performance issues. If you are having jobs that are crashing...

anardinelli

Hi @tuckera how are you? When you mean "file in workflows" you mean files that are added to a source on a Structured Streaming? If you use Autoloader (AKA CloudFiles) on the process of ingesting, there are two ways it can identify files: The first on...

anardinelli

@Cert-Team Can you please provide guidance on this? Best, Alessandro

anardinelli

Hello @vishva-fivetran how are you? To set the spark.driver.maxResultSize property, you can do so in the cluster Spark config. The property spark.driver.maxResultSize can be set to a value higher than the value reported in the exception message. For ...

anardinelli

@Cert-Team Can you please review/consider? Best, Alessandro

anardinelli

Hello @galzamo how are you? You can check on the SparkUI for long running stages that might give you a clue where it's spending the most time on each task. Somethings can be the reason: 1. Increase of data and partitions on your source data 2. Cluste...

Databricks Community

User Stats

User Activity

Job running on Attacama Profiler takes long to complete or crashes

Re: File trigger for workflows

Re: Certifications questions

Re: Databricks query on a warehouse failing with ` spark.driver.maxResultSize` error

Re: Databricks Certification Test centre Examination Request

Re: Job running time too long