Data Engineer: Data Platforms Job in Ibm India

In this role, you will join one of IBM Consulting's Client Innovation Centers (Delivery Centers), where we deliver deep technical expertise and industry insight to both public and private sector clients globally. Our centers provide locally based skills to drive the innovation and adoption of new technologies. A career in IBM Consulting offers long-term relationships and close collaboration with clients worldwide. You ll work alongside visionaries across multiple industries to improve hybrid cloud and AI journeys for some of the world s most innovative and valuable companies. Your ability to accelerate impact and foster meaningful change is enabled by IBM's robust technology platforms, including software and Red Hat. Curiosity and a constant quest for knowledge are key to success, allowing you to challenge the norm, think creatively, and produce groundbreaking solutions for our clients. This environment promotes long-term growth, with ample career development opportunities.

Your Role and Responsibilities:

As a Big Data Engineer, you will be responsible for the design, maintenance, evaluation, and testing of big data solutions. Your key responsibilities include:

• Designing, building, optimizing, and supporting data models and ETL processes based on client business requirements.
• Building, deploying, and managing data infrastructure that can support the needs of a rapidly growing, data-driven organization.
• Coordinating data access and security to ensure seamless data access for data scientists and analysts when needed.
• Developing data pipelines/workflows for Source to Target and implementing solutions to address client needs.
• Ensuring high performance, scalability, and reliability of big data solutions.

Required Technical and Professional Expertise:

• 3-5 years of experience working with Big Data technologies (Hadoop, Spark, HBase, Hive).
• Proficient in Scala and Python for data engineering tasks, including writing Pyspark programs for data analysis.
• Good working experience with Python to develop a custom framework for rule generation (similar to a rules engine).
• Developed Python code to gather data from HBase and implemented solutions using Pyspark.
• Strong knowledge of Apache Spark, including working with DataFrames and RDDs for business transformations.
• Experience using Hive Context objects for read/write operations in Hive.

Preferred Technical and Professional Expertise:

• Understanding of DevOps principles and practices.
• Experience in building scalable end-to-end data ingestion and processing solutions.
• Familiarity with AWS services (e.g., S3, Athena, DynamoDB, Lambda, Jenkins).
• Proficiency in object-oriented and/or functional programming languages such as Python, Java, and Scala.

Why Join IBM Consulting?

As a Big Data Engineer at IBM Consulting, you will have the opportunity to work with cutting-edge technologies, influence industry transformations, and drive meaningful change for a variety of clients. IBM fosters a culture of continuous learning and career development, giving you the resources and support to grow in your role and within the company. This role offers the chance to work on complex, high-impact projects and collaborate with a global team of industry leaders.