- 1543 Views
- 1 replies
- 3 kudos
row-level concurrency
With the introduction of Databricks Runtime 14, you can now enable row-level concurrency using these simple techniques!
- 1543 Views
- 1 replies
- 3 kudos
- 3 kudos
Thank you for sharing this @Hubert-Dudek
- 3 kudos
- 1298 Views
- 0 replies
- 0 kudos
Window functions in Change Data Feed
Hello!I am currently exploring the possibility of implementing incremental changes in our company's ETL pipeline and looking into Change Data Feed option. There are a couple of challenges I'm uncertain about.For instance, we have a piece of logic lik...
- 1298 Views
- 0 replies
- 0 kudos
- 15244 Views
- 2 replies
- 1 kudos
Resolved! External location + Failure to initialize configuration for storage account
Hello,I created a storage credential and an external location. Test is ok, I'm able to browse it from the portal. I have a notebook to create a table :%sqlCREATE OR REPLACE TABLE myschema.mytable( data1 string, data2 string)USING DELTA LOCATION "abf...
- 15244 Views
- 2 replies
- 1 kudos
- 1335 Views
- 0 replies
- 0 kudos
Debugging options if you are using streaming, RDDs and SparkContext?
Hi all,I've been trying to make use of some of the more recent tools for debugging in Databricks: pdb in the Databricks web interface with the variable explorer described in this article.I've also been trying to debug locally using the VSCode extensi...
- 1335 Views
- 0 replies
- 0 kudos
- 13951 Views
- 1 replies
- 1 kudos
[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]
Hi all,I'm trying to join 2 views in SQL editor for some analysis. I get the following error:[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0: Fail to parse '22/12/...
- 13951 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Kaniz, I found the equivalent SQL code for this but it didn't seem to store the operation past the execution. I.e I would run the code to configure settings, then run the troublesome code afterwards and still get the same result. The problem has b...
- 1 kudos
- 14608 Views
- 2 replies
- 1 kudos
Z-ordering optimization with multithreading
Hi, I am wondering if multithreading will help with the performance for z-ordering optimization on multiple delta tables.We are periodically doing optimization on thousands of tables and it easily takes a few days to finish the job. So we are looking...
- 14608 Views
- 2 replies
- 1 kudos
- 23385 Views
- 4 replies
- 5 kudos
Pyflake errors when using %run
I am using %run command to import shared resources for each of my processes. Because it was the most easy way to import my common libraries. However, in that way, pyflake can't resolve the dependencies quite well. And I end up working in code with ma...
- 23385 Views
- 4 replies
- 5 kudos
- 5 kudos
You could use something like flake8 and customize the rules in the .flake8 file or ignore specific lines with #noqa. https://flake8.pycqa.org/en/latest/user/configuration.html
- 5 kudos
- 7516 Views
- 0 replies
- 0 kudos
Pandas 2.x availability
Hi All,I am wondering if Pandas 2.x will be available soon or is it an available option to install.I have a small job I built to manipulate some strings from a database table when technically did the job, but doesn't scale with older versions of pan...
- 7516 Views
- 0 replies
- 0 kudos
- 1493 Views
- 0 replies
- 0 kudos
Uploading Data into Databricks from CSV - Puts Amount column as a string
I am attempting in ingesting data into databricks via CSV, with the following statement below, this brings in my data looking perfect:Although, the bad part is I have to group this data by summing highlighted amt field. Given it is string it spits ou...
- 1493 Views
- 0 replies
- 0 kudos
- 8583 Views
- 0 replies
- 0 kudos
Parse_Syntax_Error Help
Hello all, I'm new to Databricks and can't figure out why I'm getting an error in my SQL code.Error in SQL statement: ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near 'if'.(line 1, pos 0) == SQL == if OBJECT_ID('tempdb.#InitialData') IS N...
- 8583 Views
- 0 replies
- 0 kudos
- 3646 Views
- 1 replies
- 1 kudos
Resolved! Workflows : pass parameters to a "run job" task
Hi folks!I would like to know if there is a way to pass parameters to a "run job" task.For example:Let's have a Job A with:a notebook task A.1 that takes as input a parameter year-month in the format yyyymma "run job" task A.2 that calls a Job BI wou...
- 3646 Views
- 1 replies
- 1 kudos
- 1 kudos
This feature will be available soon as part of Job Parameters. Right now it is not possible to easily pass parameters to a child job.
- 1 kudos
- 6913 Views
- 4 replies
- 0 kudos
Resolved! Programmatically updating the “run_as_user_name” parameter for jobs
I am trying to write a process that will programmatically update the “run_as_user_name” parameter for all jobs in an Azure Databricks workspace, using powershell to interact with the Jobs API. I have been trying to do this with a test job without suc...
- 6913 Views
- 4 replies
- 0 kudos
- 0 kudos
Solution you've submitted is a solution for different topic (permission to run job, the job still runs as the user in run_as_user_name field). Here is an example of changing "run_as_user_name"Docs:https://docs.databricks.com/api/azure/workspace/job...
- 0 kudos
- 3033 Views
- 1 replies
- 0 kudos
Spark Configuration Parameter for Cluster Downscaling
spark.databricks.aggressiveWindowDownS This parameter is designed to determine the frequency, in seconds, at which the cluster decides to downscale.By adjusting this setting, you can fine-tune how rapidly clusters release workers. A higher value will...
- 3033 Views
- 1 replies
- 0 kudos
- 0 kudos
I wish there was a configuration to toggle upscaling behavior. I want the clusters to scale up only if the bottleneck is approaching 70% memory usage. Currently the autoscaling is only based on CPU not Memory (RAM).
- 0 kudos
- 2099 Views
- 1 replies
- 0 kudos
Clean driver during notebook execution
Is there any way to clear the memory driver during the execution of my notebook? I have several functions that are executed in the driver and that generate in it different dataframes that are not necessary (these dataframes are created just to do som...
- 2099 Views
- 1 replies
- 0 kudos
- 0 kudos
since spark uses lazy execution, those dataframes you do not need cannot be cleared unless you do use them (why define them otherwise?).So when doing an action, spark will execute all code that is necessary. If you run into memory issues, you can do...
- 0 kudos
- 1481 Views
- 0 replies
- 0 kudos
How to upload a Spark Dataframe to Azure Table Storage?
Is it possible to make a table in Azure Table Storage based on a Spark Dataframe using Python, any ideas?
- 1481 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
