Databricks Community

ThomazRossito · 04-14-2024

Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancemen...

ThomazRossito · 04-16-2024

Hello,You will need to use some CSV configuration parametersfollow the documentation belowhttps://spark.apache.org/docs/latest/sql-data-sources-csv.html

ThomazRossito · 04-14-2024

Hello,I managed to parse using the "json_tuple" functionThere are other functions that can helpschema_of_jsonget_json_objectfrom_json dd = (df.select(json_tuple(col("AdditionalRequestParameters"), "Locale", "Setversion", "Flags", "ScalingID", "Versio...

ThomazRossito · 04-14-2024

Hi,I noticed that in the scheme you are creating, there are more columns than in your csv file, I was able to understand that the final result needs to include the 6 columnsI would use withColumn, for the 3 columns that do not exist in the fileBelow ...

ThomazRossito · 04-14-2024

Hello,It is possible to have the expected result with dbutils.fs.cp("Origin location", "Destination location", True) and then create the table with the LOCATION of the Destination locationHope this helps

ThomazRossito · 04-09-2024

Hello,in your workflow it is possible to end with a SQL task and in the SQL Task item you can select some options, see the image below:

Databricks Community

User Stats

User Activity

Post: Lakehouse Federation - Databricks

Re: DataFrame to CSV write has issues due to multiple commas inside an row value

Re: DataFrame to CSV write has issues due to multiple commas inside an row value

Re: When reading a csv file with Spark.read, the data is not loading in the appropriate column while

Re: Move a delta table from a non UC metastore to a UC metastore preserving history

Re: Trigger Dashboard Update At The End of a Workflow