Lead-data Engineer Job in Talent Et Au-dela

We will count on you to: Defines and implements solution for Data Analytics to meet business needs including data integration & pipeline needs and interfaces. Responsible for implementing robust data pipeline using Microsoft Stack, creating reusable and scalable data pipelines Responsible for development and deployment of new data platforms Create pipelines in ADF using linked services/Data sets to extract, Transform and load (ETL) data from different sources like Azure SQL, Blob storage, etc. Develop spark applications using the Py spark and Spark-SQL for data extractions, transformation & Aggregation from multiple file formats. Responsible for estimating the cluster size, monitoring and troubleshooting of the Data bricks cluster. Develop Json scripts for deploying the pipeline in Azure Data Factory (ADF) that process the data using SQL activity. Collaborating with Azure Cloud Solution Architects and Data Platform Engineers in developing complex end-to-end Enterprise solutions on Microsoft Azure platform. Maintains health and procedures on data pipelines for continued success Involve in technical design of solutions working collaboratively with development leads, Architecture teams and development/implementation of solutions on Data Engineering/Analysis. Develop initial guidelines and support structures to ensure the service and assets are properly managed and maintained Collaborating with Team members in developing applicable methodologies, tools, approaches, point of view, thought leadership and accelerators to differentiate EY in the Digital & Emerging Technology Advisory Market Develop reusable assets such as prototypes, solutions, documentation and other materials that can help grow the practice Design & build API & data integrations between the various applications in the Platform utilizing modern cloud based stack like Azure Service Bus, Azure Functions and Azure Data Factory.