BriefCase
BriefCase

Lead Data Engineer Job in Affine Analytics Pvt Ltd

Lead Data Engineer

Affine Analytics Pvt Ltd 4+ weeks ago
Apply Now
Job Summary

Roles and Responsibilities

The responsibilities will include:

  • Hands-on experience with AWS technologies and at least one full life cycle project experience using various cloud technologies
  • Hands-on expertise with writing, modifying, tuning SQL queries
  • Optimize, scale and perform performance tuning on DW on AWS Redshift and the data lake on AWS S3
  • Analysis, design, development, testing, and support of Talend DI/ Big Data solution
  • Backtracking, troubleshooting failures and provide fix
  • On demand support data feeds from various ERP source systems into Data warehouse
  • Proactively identify gaps and propose better/best possible solutions to the existing system
  • Ability to independently work on full life cycle projects
  • Being able to support Spark and AWS EMR configurations along with Spark scripting
  • Be able to analyze large complex data sets to resolve data quality issues
  • Document requirements and create/update system specifications as well as documentation of code.
  • Design and Develop new Talend Big Jobs using best practices laid down by the GP Technical Lead
  • Be able to adopt to technical stack used by Procurement Data Lake and Data Warehouse
  • Position s day to day responsibility will be directed by the technical lead who is part of the KTC. All team members are required to be on a daily status call with GP Technical Lead

  • What You Will Need To Bring With You:

    (experience & education required)

    2+ years of Procure to Pay Domain knowledge

    Bachelor s degree in Engineering (preferably Analytics, MIS or Computer Science). Masters degrees preferred.

    Minimum 8+ of ETL with AWS, Talend Big Data, Data Analytics and Data Lake ETL Experience using Glue, Spark scripting, AWS EMR, AWS DMS, AWS Redshift, AWS S3, AWS EC2, AWS CloudWatch, AWS RDS, AWS IAM, Microsoft SQL, Microsoft SSIS, Java, GitLab, DevOps (CI/CD),

    Working / Hands on knowledge on Lamda

    Leverage Kinesis/ Firehose for real-time extraction

    Experience with PySpark, Glue

    Handle SSIS/TALEND/DMS/Glue (other AWS services) ETL Job failures - Code, Data fixes

    Application Configuration changes related to TALEND ETL packages as well as SSIS.

    Create new ETL Jobs on Talend DI

    Data engineering concepts (ETL, near-/real-time streaming, data structures, metadata) Day to day Job:
  • Provide on-demand support, and the activities will include all critical, high, medium and low priority issues.





    • Qualification :
    B.Tech/B.E. in Computers,M.Tech in Computers,MCA in Computers
    Experience Required :

    8 to 10 Years

    Vacancy :

    2 - 4 Hires

    Similar Jobs for you

    See more recommended jobs