Technical Lead - Data Engineering Job in To The New

Responsibilities:

• Define client needs and oversee project milestones to ensure expectations, timelines, and budgets are met

• Define data platform architecture and design, have hands on capability to review code and make required changes

• Leverage extensive knowledge of distributed systems, process flows and procedures to aid analyses and recommendations for solution offerings

• Responsible for the overall quality of project deliverables and the successful implementation of defined solution for the customer

• Build long-term, superior client relationships and proactively manage client expectations, and ensure that change control is used when scope boundaries are exceeded

• Communicate a compelling and inspired vision to all customer levels, from CTOs/CIOs, to engineering managers and programmers

• Maintain a strong network and promote the organization at various meetings, forums, panels, publications, and conferences to establish thought leadership in the industry

• Responsible for POCs on any tech stacks introduced, or possible version upgrades

Must Have:

• At least 8+ years of experience in design and development using various database technologies with recent 4+ years associated Hadoop technology stack and programming languages

• Hands-on and technical lead experience in 2 or more areas:

  • Hadoop, HDFS, MR

  • High Availability architecture and DR setup

  • Spark Streaming, Spark SQL, Spark ML

  • Kafka/Flume

  • Apache NiFi

  • Worked with Hortonworks Data Platform as Architect CDH (Cloudera Distribution for Hadoop) as developer/administrator

  • Hive / Pig / Sqoop

  • NoSQL Databases HBase/Cassandra/Neo4j/MongoDB

  • Visualisation & Reporting frameworks like D3.js, Zeppelin, Grafana, Kibana Tableau, Pentaho

  • Scrapy for crawling websites

  • Good to have knowledge of Elastic Search

  • Good to have understanding on Google Analytics data streaming.

  • Data security (Kerberos/OpenLDAP/Knox/Ranger)

• Should have a very good overview of the current landscape and ability to visualise technology and industry trends

• Ability to successfully lead a team of big data engineers and big data developers through all phases of the development life cycle, including requirements definition, architecture, design, development, implementation, and testing

• Working knowledge of Big Data Integration with Third party / in house built Metadata Management, Data Quality, Master Data Management Solutions, Structured/Unstructured data

• Have been active in the community in terms of articles / blogs / speaking engagements at conferences