Attended the Data and AI Summit 2023 and gained insights into the utility catalog and services that it has to offer, definitely going to try the data governance as it's a game changer.
Can I reference theUnity Catalog through my Glue Serverless jobs?
Hi,I want to convert column of XML strings to column of Json in PySpark., using withcolumn and xmltodict method as UDF, is giving Json with '=' instead of ':' in the dictionary. Please let me know if there is any alternative for this.
To convert a column of XML strings to a column of JSON in PySpark, you can use the `from_json` function along with the `xmltodict` library. However, instead of using a UDF with `withColumn`, you can use the `select` function to transform the column.
Having a great time at the community hub at the Summit. Highly recommend!
Today I walked into a session that talked about a fairly new language - Rust. The name can mislead you, I believe taking a look at the roots of how to best use CPU cycles is a game changer and Rust is traversing new areas that others might have ignor...
Yes, Rust is definitely part of the future. It brings performance and simplicity to us. I think it will add to the community, rather than replacing. Scala and R will never go away, Python will always be strong, but Rust gives us one other tool in ...
How are you offloading data from Snowflake & repointing raw S3 staging dumps to Parquet/Delta?
Attended training and few sessions.. great experience
How does one implement a databricks pipeline in a classified environment
Really enjoying my time at the summit!
Wow what was cool, so many of my workarounds have materialized into solutions!!
At Chevron we actively use Databricks to provide answers to business users. It was extremely interesting to see the use LakeHouseIQ initiatives as it can expedite how fast our users can receive their answers/reports. Is there any documentation that I...
I learned data lake house
Great conference. Exited about upcoming LakehouseIQ feature.
the new way to optimize my notebook is something that I learned and I will execute soon
I have mounted an Azure Blob Storage in the Azure Databricks workspace filestore. The mounted container has zipped files with csv files in them. What is the best way to read the zipped files and write into a delta table?@sasikumar sagabala​
Hello @Debayan I recently came across the similar scenario, is there a way to do this via autoloader. We have zip Folders added daily to our AWS S3 bucket and we want to be able to unzip and load the csv files continuously (Autoloading)
Forgive me for a nooby question, but what is the point of data governance if everyone is working at the same company? Is it for insider purposes?
Hey Zhudocode, Actually answered this is person, but Data Governance fundamentally is about the appropriate, efficient and effective use of data. Appropriate use has to do with ethical ai, use of personal information, and policy around confidential a...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group