BriefCase
BriefCase

Data Engineer (azure Cloud) Job in Datatobiz

Data Engineer (azure Cloud)

Datatobiz 4+ weeks ago
Apply Now
Job Summary

Skillset

  • Experience in Azure Databricks platform. Delta lake, Time travel design patterns, Lambda Data Architecture
  • Experience in Azure Cloud Platform. (ADLS, Blob , ADF ).
  • Understanding of big data file formats like JSON, Avro, Parquet and other prevalent big data file formats
  • Experience with Python, PySpark, Scala, Spark to write data pipelines and data processing layers
  • Demonstrate expertise in writing complex, highly-optimized SQL queries across large data sets
  • Experience with Data Governance (Data Quality, Metadata Management, Security, etc.)
  • Strong SQL skills and proven experience in working with large datasets and relational databases.
  • Understanding Data Warehouse design principles (Tabular, MD).
  • Experience in working with big data tools/environment to automate production pipelines
  • Familiarity with Azure Synapse and Snowflake
  • Experience with data visual analysis and BI tools (Matplotlib and Power BI)

    • Responsibilities

  • As a Data Engineer in our team, you should have a proven ability to deliver high quality work on time and with minimal supervision.
  • Developing new data pipelines and ETL jobs for processing millions of records and it should be scalable with growth.
  • Pipelines should be optimised to handle both real time data, batch update data and historical data.
  • Establish scalable, efficient, automated processes for complex, large scale data analysis.
  • Write high quality code to gather and manage large data sets (both real time and batch data) from multiple sources, perform ETL and store it in a data warehouse/data lake
  • Manipulate and analyse complex, high-volume, high-dimensional data from varying sources using a variety of tools and data analysis techniques.
  • Participate in data pipelines health monitoring and performance optimisations as well as quality documentation.
  • Adapt best practices and industry standards, ensuring top quality deliverables and playing an integral role in cross-functional system integration.
  • Data integration using enterprise development tool-sets (e.g. ETL, MDM, Quality, CDC, Data Masking, Quality).
  • Design and implement formal data warehouse testing strategies and plans including unit testing, functional testing, integration testing, performance testing, and validation testing.
  • Interact with end users/clients and translate business language into technical requirem
    •
    Experience Required :

    3 to 5 Years

    Vacancy :

    2 - 4 Hires

    Similar Jobs for you

    See more recommended jobs