Name: Vigneshraja Palaniraj 🌟
Community nickname: @Vigneshraja Palaniraj
Pronouns: He/Him
Company: Cognizant Technology Solutions, United Kingdom
Job Title: Manager – Senior Data Engineer
Databricks Certifications
Databricks Certified Data Engineer Associate
Databricks Certified Associate Developer for Apache Spark 3.0
Academy Accreditation – Databricks Lakehouse Fundamentals
Could you give a brief description of your professional journey to date?
My professional journey commenced in 2009 when I joined HCL Technologies as a Graduate Engineer Trainee. However, my foray into the data engineering field did not begin until my tenure with IBM India in 2014 when Hadoop gained traction and clients sought to integrate it into their data processing systems. My curiosity about this area was piqued, and I began learning about the technologies and gradually transitioned into the Data Engineering domain.
In 2015, upon joining Infosys, I became a full-time data engineer and worked on Apache Spark projects that entailed creating intricate data pipelines and integrating data from diverse sources. At present, I am employed with Cognizant and relishing the opportunities to handle an array of use cases, such as designing ingestion pipelines, implementing data governance and quality frameworks, and optimizing data loads for performance and scalability.
What is your favorite thing about your current job/role?
The aspect I cherish the most about my current position is the opportunity to tackle intricate and demanding data use cases. I'm fortunate to have a brilliant team and access to cutting-edge technologies such as Databricks, Snowflake, Airflow, AWS, QlikSense, among others.
Furthermore, I am privileged to have regular interactions with Consultants and Solution Architects from our organization and technology partners to discuss challenges, performance enhancements, new product features, and more. This enables me to approach problems from a fresh perspective and work towards finding solutions, making my job an enriching and satisfying experience every day.
What are three words your coworkers would use to describe you?
Curious, collaborative and reliable.
Do you have any mentors in your professional life?
In my thirteen-year professional career, I have had the wonderful pleasure of working with many colleagues from whom I have learned a lot. They have identified me, backed me up, given me excellent opportunities, and shared their support and guidance.
To name a few: Stalin Kamju, Rajesh Kumar Dayalan, Aravindhan Subramaniam, Jitendra Raisinghani, Jaydip Mitra, Raghava Madhira, and Sudheer HKV. I’m eternally grateful to all of them.
When and why did you start using Databricks?
I began using Databricks Community Edition initially around the year 2019 for my personal learning. The idea of using collaborative and interactive notebook interfaces to design and develop your data pipelines was simply amazing. In 2021, I got the opportunity to work with and explore the full power of Databricks Enterprise Edition. The platform is growing better and bigger every day, with new features getting released and making the lives of data engineers, ML engineers, and data analysts so convenient.
Do you have a favorite Databricks feature? Something that makes your life a lot easier? Or more interesting?
I’m going to take a luxury here and name a few favorites 🙂 First, absolutely the Databricks notebooks. So practical and convenient. Second, the delta lake. What a change it has brought to the data storage layer - ACID support, Time travel, Compaction, Schema Enforcement and so on. Third, the Unity Catalog. Apart from all the outstanding features it offers, I’m simply amazed by the real-time lineage. It captures the lineages of not just the table and column but also the notebooks. Simply WOW.
Is there a Databricks feature you want to see in the future?
I believe the Serverless option is only available for Databricks SQL as of now. I would like to see a similar Serverless feature for Job Computes as well. This might help developers focus only on data processing tasks.
I’m also interested to see the feature in Databricks Workflows where we can set dependencies across different jobs and also be able to execute the jobs using caller’s rights.
When and why did you start using Databricks Community?
In January 2023, I was introduced to the Databricks Community page by a Databricks Solution Architect. I figured it would be a wonderful platform for individuals to learn from and collaborate with others who share a passion for Data Engineering and of course, Databricks. I believed I could gain great knowledge by sharing information and helping each other through this Community.
What’s your favorite thing about being in the Databricks Community?
The beauty of this community lies in the collaboration that takes place here. When a member posts a problem, it is not only the Databricks team that provides answers, but also fellow members. As a result, every member who reads the post gains a better understanding of the problem and the various ways it can be tackled. In my view, this wealth of knowledge is invaluable.
I take this opportunity to thank and appreciate the Databricks Community team for their wonderful efforts in improving this platform, sharing frequent feature updates, and also conducting monthly Community meet-ups. I missed the meetups last time but am hopeful to attend them in the future.
Pivoting a little: What’s your favorite hobby?
I enjoy spending time with my family during the weekends. As the dad of a three year old, I spend most of my leisure time with him playing and taking him for activities. If I get some personal free time, I enjoy reading books.
Where do you see yourself in 3 years?
I wish to see myself as a Senior Solution Architect who has a holistic view of the data lifecycle and is able to translate business objectives into data architecture. I also love to see myself staying current with the latest technologies and trends and creating useful content in the form of blogs and videos to share my knowledge.
Social Media Platform:
Linkedin: https://www.linkedin.com/in/vigneshraja-palaniraj/
Thank you for sharing insights into your personal and professional journey, and for allowing us to learn from your greatness @Vigneshraja Palaniraj 💫
Feel free to leave your questions for Vigneshraja in the comments section below! 🚀
