Hi, I am wondering if multithreading will help with the performance for z-ordering optimization on multiple delta tables.We are periodically doing optimization on thousands of tables and it easily takes a few days to finish the job. So we are looking...
I am using %run command to import shared resources for each of my processes. Because it was the most easy way to import my common libraries. However, in that way, pyflake can't resolve the dependencies quite well. And I end up working in code with ma...
You could use something like flake8 and customize the rules in the .flake8 file or ignore specific lines with #noqa. https://flake8.pycqa.org/en/latest/user/configuration.html
Hi All,I am wondering if Pandas 2.x will be available soon or is it an available option to install.I have a small job I built to manipulate some strings from a database table when technically did the job, but doesn't scale with older versions of pan...
I am attempting in ingesting data into databricks via CSV, with the following statement below, this brings in my data looking perfect:Although, the bad part is I have to group this data by summing highlighted amt field. Given it is string it spits ou...
Hello all, I'm new to Databricks and can't figure out why I'm getting an error in my SQL code.Error in SQL statement: ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near 'if'.(line 1, pos 0) == SQL == if OBJECT_ID('tempdb.#InitialData') IS N...
Hi folks!I would like to know if there is a way to pass parameters to a "run job" task.For example:Let's have a Job A with:a notebook task A.1 that takes as input a parameter year-month in the format yyyymma "run job" task A.2 that calls a Job BI wou...
This feature will be available soon as part of Job Parameters. Right now it is not possible to easily pass parameters to a child job.
I am trying to write a process that will programmatically update the “run_as_user_name” parameter for all jobs in an Azure Databricks workspace, using powershell to interact with the Jobs API. I have been trying to do this with a test job without suc...
Solution you've submitted is a solution for different topic (permission to run job, the job still runs as the user in run_as_user_name field). Here is an example of changing "run_as_user_name"Docs:https://docs.databricks.com/api/azure/workspace/job...
spark.databricks.aggressiveWindowDownS This parameter is designed to determine the frequency, in seconds, at which the cluster decides to downscale.By adjusting this setting, you can fine-tune how rapidly clusters release workers. A higher value will...
I wish there was a configuration to toggle upscaling behavior. I want the clusters to scale up only if the bottleneck is approaching 70% memory usage. Currently the autoscaling is only based on CPU not Memory (RAM).
Is there any way to clear the memory driver during the execution of my notebook? I have several functions that are executed in the driver and that generate in it different dataframes that are not necessary (these dataframes are created just to do som...
since spark uses lazy execution, those dataframes you do not need cannot be cleared unless you do use them (why define them otherwise?).So when doing an action, spark will execute all code that is necessary. If you run into memory issues, you can do...
Is it possible to make a table in Azure Table Storage based on a Spark Dataframe using Python, any ideas?
Hello, I have to export all my notebooks from DEV to PROD.My problem is that I can't find a way to export my jobs (not the outputs, the actual notebook schedulation), is it even possible? I have hundreds of jobs to export and have to keep the same pa...
You can export the complete job configuration in different ways: You can use the REST API: https://docs.databricks.com/api/azure/workspace/jobs/getYou can use the python SDK: https://github.com/databricks/databricks-sdk-pyYou can use Terraform: http...
I have process in Data factory, that loads CDC changes from sql server and then trigger notebook with merge to bronze and silver zone. Single notebook takes about 1 minute to run but when all 50 notebooks are fired at once the whole process takes 25 ...
I have 1 delta table that I continuously append events into, and a 2nd delta table that I continuously merge into (streamed from the 1st table) that has unique ID's where properties are updated from the events (An ID represents a unique thing that ge...
I have a DLT pipeline that has been running for weeks. Now, trying to rerun the pipeline with the same code and same data fails. I've even tried updating the compute on the cluster to about 3x of what was previously working and it still fails with ou...
I'd focus on understanding the codebase first. It'll help you decide what logic or data asset to keep or not keep when you try to optimize it. If you share the architecture of the application, the problem it solves, and some sample code here, it'll h...
I want to run an ETL job and when the job ends I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling spark.stop() the job returns with status failed even though it has f...
Please refer to this Job fails, but Apache Spark tasks finish - Databricks
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group
