Can someone explain why this below code is throwing an error? My intuition is telling me it's my spark version (3.2.1) but would like confirmation:d = {'key':['a','a','c','d','e','f','g','h'],
'data':[1,2,3,4,5,6,7,8]}
x = ps.DataFrame(d)
x[x['...
@pjp94 - The error indicates the pandas pyspark implementation does not have the below method implemented.
pd.Series.duplicated()
Next steps is to use dataframe methods such as distinct, groupBy, dropDuplicates to resolve this.
TimeoutException: Stream Execution thread for stream [id = xxx runId = xxxx] failed to stop within 15000 milliseconds (specified by spark.sql.streaming.stopTimeout). See the cause on what was being executed in the streaming query thread.I have a data...
@User_1611 - could you please try the following ?
Reduce the number of streaming queries running on the same clusterMake sure your code does not try to re-trigger/start an active streaming queryMake sure to collect the thread dumps if this error hap...
I have 50k + parquet files in the in azure datalake and i have mount point as well. I need to read all the files and load into a dataframe. i have around 2 billion records in total and all the files are not having all the columns, column order may di...
@Shan1 - This could be due to the files have cols that differ by data type. Eg. Integer vs long , Boolean vs integer. can be resolved by schemaMerge=False. Please refer to this code. https://github.com/apache/spark/blob/418bba5ad6053449a141f3c9c31e...
Hi everyone,I am using DBR version 13 and Managed tables in a custom catalog location of table is AWS S3.running notebook on single user clusterI am facing MalformedInputException while saving data to Tables or reading it.When I am running my noteboo...
@Kaniz The issue is resolved as soon as I deployed it to mutlinode dev cluster.Issue is only occurring in single user clusters. Looks like limitation of running all updates in one node as distributed system.
There is no resource to create All Purpose Cluster, but I need it, so does it mean I should create it via Terraform or DBX and reference to it, which I dont prefer?
Is there a way to get a child Job Run status and show the result within the parent notebook execution?Here is the case: I have a master notebook and several child notebooks. As a result, I want to see which notebook failed: For example Notebook job s...
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
I have a notebook where I read multiple tables from delta lake (let say schema is db) and after that I did some sort of transformation (image enclosed) using all these tables lwith transformations like join,filter etc. After transformation and writin...
I have a DLT with a table that I want to contain the running aggregation (for the sake of simplicitly let's assume it's a count) for each value of some key column, using a session window. The input table goes back several years and to clean up aggreg...
Hi @vroste ,
• To configure the update output mode for running aggregation in Delta Live Tables (DLT), use the outputMode option when writing the DLT table.• By default, DLT writes data in complete mode, which outputs the complete result table after...
At this moment, I'm working on removing Legacy global and cluster-named init scripts due, it will be disabled for all workspaces on 01 Sept.At this moment, I'm facing a strange problem regarding moving init scripts from dbfs to the Workspace location...
Using the new CLI v0.214, uploading ".sh" file works fine.`databricks workspace import --overwrite --format AUTO --file init_setup /init/user/job/init_setup`
Hi, Im facing an issue while writing to Salesforce sandbox from Databricks. I have installed the "spark-salesforce_2.12-1.1.4" library and my code is as follows:-df_newLeads.write\ .format("com.springml.spark.salesforce")\ .option("username...
I made a function that used the code below and returned url, connectionProperties, sfwriteurl ="https://login.salesforce.com/"dom = url.split('//')[1].split('.')[0]session_id, instance = SalesforceLogin(username=connectionProperties['name'], password...
Hi everyone,We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach t...
For a streamlined migration of your Databricks workspace from one AWS account to another, start by exporting notebook, workflow, and saved query configurations using Databricks REST API or CLI. Employ Deep Clone or Delta Sharing for managed table dat...
Hi all,As part of an on-going exercise to refactor existing T-SQL code into Databricks, we've stumbled into an issue that we can't seem to overcome through Spark SQL.Currently we use dynamic SQL to loop through a number of tables, where we use parame...
Hi - We are migrating to Unity Catalog 14.3 LTS and have seen a change in behavior using withColumnRenamed.There is an error COLUMN_ALREADY_EXISTS on the join key, even though the column being renamed is a different column. The joined DataFrame do...
Hey @ksamborn I can think of 2 solutions:Rename the column in df_2 before joining: df_1_alias = df_1.alias("t1")
df_2_alias = df_2.alias("t2")
join_df = df_1_alias.join(df_2_alias, df_1_alias.key == df_2_alias.key)
rename_df = join_df.withColumnRenam...
Hello, I'm trying to setup a notebook for tests or data quality checks. The name is not important.I basically read a table (the ETL output process - actual data).Then I read another table and do the calculation in the notebook (expected data)I'm stuc...
Hello, I want to create an sql udf as follows:%sqlCREATE or replace FUNCTION get_type(s STRING) RETURNS STRING LANGUAGE PYTHON AS $$ def get_type(table_name): from pyspark.sql.functions import col from pyspark.sql import SparkSession ...
Hi @Avinash_Narala, The error message indicates that the execution of your user-defined function (UDF) get_type failed. This could be due to a variety of reasons.
Here are a few things you could check:
Data Type Mismatch: Ensure that the data ty...
