Databricks Community

andrew0117 · 05-19-2023

Normally, our ELT framework takes in batches one by one and loads the data into target tables. But if more than one batches come in at the same time, the framework will break due to the concurrency issue that multiple sources are trying to write the ...

andrew0117 · 04-16-2023

When I use SQL code like "create table myTable (column1 string, column2 string) using csv options('delimiter' = ',', 'header' = 'true') location 'pathToCsv'" to create a table from a single CSV file stored in a folder within an Azure Data Lake contai...

andrew0117 · 04-09-2023

The master notebook is calling a child notebook using dbutils.notebook.run("PathToChildnotebook"). The child notebook defines a user-defined function (UDF) and registers it using spark.udf.register. However, when the child notebook finishes running a...

andrew0117 · 03-26-2023

if I have two dataframes df_target and df_source, can I do df_target.as("t).merge(df_source.as("s"), "s.id=t.id").whenMatched().updateAll().whenNotMatched.insertAll.execute(). when I tried the code above, I got the error "merge is not a member of the...

andrew0117 · 03-21-2023

If I creat a table using the code below: CREATE TABLE IF NOT EXISTS jdbcTableusing org.apache.spark.sql.jdbcoptions( url "sql_server_url", dbtable "sqlserverTable", user "username", password "password")will jdbcTable always be automatically sync...

andrew0117 · 04-17-2023

so basically, you cannot create a partitioned table based on a single csv file by simply using sql create table partitioned by () location 'pathToCsv' file? because the single csv file does not have the paritioned file structure on that location. I u...

andrew0117 · 04-17-2023

I'm recreate the table from the same csv file in the same location, so I cannot delete the csv file. Thanks!

andrew0117 · 04-14-2023

@Debayan Mukherjee the error code is : undefined function: my_udf_name. This function is neither a bulit-in/temporary function nor a persistent function that is qualified as spark_catalog_.default.my_udf_name. To my understanding, the dbuils.noteboo...

andrew0117 · 04-09-2023

No really. Eventually we decided to fail the whole process and send the notification to the end user to have them to do the dedup.

andrew0117 · 03-22-2023

yes, I thought the internal table stored at hive warehouse will not get updated automatically. But to my surprise, the table was synchronized immediately after I manually updated the source table in azure Sql server database.

Databricks Community

User Stats

User Activity

what is best practice to handle the concurrency issue in batch processing?

partition on a csv file

master notebook cannot find the udf registered in the child notebook

Can merge() function be applied to dataframe?

Will a table backed by a SQL server database table automatically get updated if the base table in SQL server database is updated?

Re: partition on a csv file

Re: partition on a csv file

Re: master notebook cannot find the udf registered in the child notebook

Re: index a dataframe from a csv file based on the file's original order (not based on any specific column, based on the entire row) using spark

Re: Will a table backed by a SQL server database table automatically get updated if the base table in SQL server database is updated?