Airflow Jobs in Hyderabad

Senior Data Engineer Hyderabad (On-site, Full-time) Location: Hyderabad, Telangana Work Type: On-site, Full-time Experience: 5+ Years Industry: AI & Data Science Company: Blend About Blend: Blend is a leading AI services company focused on solving real-world problems through the fusion of artificial intelligence, data science, and human ingenuity. We believe in co-creating meaningful impact by aligning top-tier talent with cutting-edge technology. Our goal is to deliver transformative solutions that unlock value for our clients and enrich the work and lives of our people. Job Overview: We are seeking a highly skilled Senior Data Engineer with strong experience in AWS, Databricks, Python, SQL, and Apache Spark/PySpark. You will be responsible for designing, developing, and maintaining scalable data architectures and pipelines to support analytics, machine learning, and business intelligence efforts across the organization. Key Responsibilities: Design and build scalable data pipelines using AWS services (Glue, Lambda, S3, EMR) and Databricks Notebooks & Workflows. Develop and manage robust ETL/ELT workflows using Python and SQL to handle structured and semi-structured data. Implement distributed data processing using Apache Spark or PySpark. Create and maintain data lakes in AWS Databricks. Collaborate with data scientists, analysts, and cross-functional teams to provide high-quality, accessible datasets. Ensure adherence to data governance, quality, and security best practices. Monitor, troubleshoot, and optimize the performance of data systems. Conduct code reviews and enforce engineering best practices. Mentor junior engineers and support their technical growth. Required Qualifications: Bachelor's or Master s degree in Computer Science, Engineering, or related field. 5+ years of experience in Data Engineering, with 2+ years working in AWS Databricks environments. Strong programming skills in Python for data processing and workflow automation. Advanced expertise in SQL for data transformation and querying large datasets. Proficient in distributed data processing using Apache Spark/PySpark. Solid understanding of data modeling, data warehousing, and performance optimization techniques. Hands-on experience with AWS services such as Glue, S3, Lambda, and EMR. Familiarity with version control tools like Git or CodeCommit. Experience with workflow orchestration tools like Airflow or AWS Step Functions (preferred but not mandatory). Be part of a forward-thinking, AI-driven organization. Work with cutting-edge cloud technologies and data platforms. Thrive in a culture of innovation, mentorship, and continuous learning. Enjoy opportunities to grow technically and professionally while driving real-world impact. Apply Now to join Blend as a Senior Data Engineer and help build data infrastructure that powers intelligent decision-making. Qualification : Bachelor's or Masters degree in Computer Science, Engineering, or related field.

Job Title: Data Engineering Manager Location: Hyderabad About Kanerika Kanerika Inc. is a leading global software products and services company dedicated to empowering data-driven enterprises. We help businesses accelerate digital transformation and maximize impact by delivering innovative, custom data solutions that optimize operations, enhance customer experiences, and fuel growth. Our Accolades Best Place to Work 2022 Great Place to Work Top 10 Most Recommended RPA Startups 2022 RPA Today Technology Innovation Award 2021 Frost & Sullivan India for Kompass composable solution Recognized for strong data privacy and security standards: ISO 27701, SOC2, GDPR compliant With a 4.5/5 Glassdoor rating, Kanerika fosters an inclusive, growth-oriented culture where employees thrive. We offer: Flexible working hours and generous parental leave Mandatory training on a safe and inclusive work environment Mentorship, professional development, and certification support Company-sponsored outings and work-life balance initiatives Join a vibrant, diverse community where your contributions truly matter. The Role We are looking for an experienced Data Engineering Manager to lead and inspire our data engineering team in Hyderabad. Partnering closely with Data Architects, you ll drive the design and development of scalable, cloud-based data pipelines and models, ensuring data quality, security, and performance. What You ll Do Lead and mentor a high-performing data engineering team, driving productivity and accountability. Collaborate with architects and development leads to build scalable, enterprise-grade data solutions. Oversee agile workflows, manage escalations, and coordinate hiring and resource planning. Use data-driven insights to guide team priorities and continuous improvement. Support pre-sales activities and work cross-functionally with sales and recruitment teams. Foster a culture of innovation, quality-first mindset, and customer focus. Who You Are A proactive leader with experience managing mid to large-sized teams and complex projects. An effective communicator, both verbally and in writing, with a knack for visually presenting ideas and core values. Results-oriented with a proven track record of delivering successful projects and publishing customer case studies. A process-driven, independent thinker who thrives under pressure and embraces continuous learning. Passionate about inclusivity and committed to contributing authentically to a diverse team environment.

Senior MLOps Engineer Location: Hyderabad, Telangana Work Type: On-site, Full-time Experience: 5+ Years (with 3 years specifically in ML Engineering) Company: Blend About Blend: Blend is a premier AI services provider committed to driving meaningful impact through data science, AI, and technology. We empower businesses by seamlessly aligning human expertise with artificial intelligence to unlock innovation and value. Job Summary: We are looking for a Senior MLOps Engineer with deep expertise in building, deploying, and managing ML pipelines on AWS. The ideal candidate will have hands-on experience with Amazon SageMaker and a strong foundation in Infrastructure as Code to support scalable and reliable ML operations. Key Responsibilities: Maintain and enhance ML pipelines on AWS, focusing on Infrastructure as Code (IaC) using CloudFormation. Implement essential pipeline extensions to support ongoing data science workflows. Document infrastructure usage, system architecture, and designs using tools such as Confluence, GitHub Wikis, and system diagrams. Serve as the internal expert on ML infrastructure, collaborating closely with data scientists to guide and support model deployment. Research and apply optimization strategies for ML workflows and infrastructure performance. Work independently and collaboratively with cross-functional teams to support ML product deployment and platform migration initiatives. Qualifications: 5+ years of hands-on DevOps experience with AWS Cloud. Proven track record of deploying at least two MLOps projects using Amazon SageMaker or similar AWS ML services. Strong proficiency in AWS services including SageMaker, ECR, S3, Lambda, and Step Functions. Expertise in Infrastructure as Code with CloudFormation across development, testing, and production environments. Solid understanding of MLOps best practices and fundamental data science principles. Proficient in Python for scripting and automation tasks. Experience with building and managing Docker images. Hands-on experience with Git-based version control systems such as AWS CodeCommit or GitHub, including GitHub Actions for CI/CD pipelines. Work with cutting-edge AI and ML technologies in a highly innovative environment. Collaborate with top-tier experts in data science and cloud engineering. Opportunities for professional growth, mentorship, and certification support. Competitive compensation package and dynamic work culture. Interested? Apply now to be part of Blend s mission to revolutionize AI-driven innovation!

Job Title: Senior Data Engineer Location: Hyderabad, Telangana, India Job Type: Full-Time | Permanent Experience Required: 6+ Years Open Positions: 2 About the Role: We are seeking a highly skilled and experienced Data Engineer to join our team in Hyderabad. As a key member of our data engineering unit, you will be responsible for designing and optimizing end-to-end data pipelines using Python, PySpark, and Databricks. This role requires strong technical expertise, a passion for solving complex data challenges, and the ability to lead best practices within the Supply Chain data domain. Key Responsibilities: Develop and maintain scalable data pipelines using Python, PySpark, and Databricks for complex data transformation within Supply Chain operations. Implement advanced data engineering solutions that support real-time and batch data processing for analytics and business intelligence. Analyze, optimize, and refactor existing data workflows to improve performance and reliability. Collaborate with cross-functional teams to understand data requirements and translate them into robust technical solutions. Take ownership of projects and deliver high-quality outcomes with minimal supervision. Provide leadership and guidance on coding standards, design patterns, and data engineering best practices. Clearly communicate technical ideas and solutions to both technical and non-technical stakeholders. Required Skills & Qualifications: 6+ years of hands-on experience in data engineering. Proficient in Python, PySpark, and Databricks for building and maintaining data pipelines. Strong problem-solving skills with a proven ability to create innovative solutions for complex data challenges. Excellent communication and interpersonal skills; able to work effectively with cross-functional teams. Experience leading and mentoring team members on technical best practices and architecture decisions. Ability to work independently, manage priorities, and drive projects to completion with minimal oversight. Perks & Benefits: Competitive compensation package Exposure to cutting-edge technologies in cloud-based data engineering Opportunity to work on large-scale Supply Chain data solutions Collaborative, innovation-driven work environment Career growth and mentorship opportunities

Role Overview: We are looking for a Software Engineer to play a key role in migrating our workflow orchestration system from Azkaban to Apache Airflow. This is a high-impact opportunity to contribute to modernizing our data processing infrastructure, working on a critical migration project that will enhance the performance and scalability of our systems. Key Responsibilities: Migration Strategy: Design, develop, and execute the migration strategy from Azkaban to Apache Airflow, ensuring a smooth transition and minimal disruption to ongoing operations. Automation: Develop Python scripts and automation tools to support the migration process and automate various tasks such as calling APIs and performing database interactions. Testing and Data Validation: Implement best practices for testing and data validation to ensure the accuracy and reliability of the migrated workflows. Process Automation: Collaborate with the development team to automate several manual processes related to release, upgrades, and deployments, improving operational efficiency. Collaboration: Work closely with platform owners and data teams to ensure that workflows continue to operate with the same performance and reliability post-migration. Optimization: Optimize Airflow DAGs for efficiency, modularity, and maintainability, ensuring scalable and high-performance data pipelines. DevOps Practices: Leverage previous DevOps experience focusing on continuous deployment and monitoring, ensuring the smooth and continuous operation of the migration process. Qualifications: Education: Bachelor s degree in Computer Science, Software Engineering, or a related field. Programming Languages: Proficiency in Python for writing automation scripts, including calling APIs and interacting with databases. Kubernetes Ecosystem: Experience creating YAMLs and Helm charts for managing Kubernetes resources. DevOps Experience: Hands-on experience with DevOps practices, particularly focusing on continuous deployment and monitoring. Orchestration System: Proficiency with Apache Airflow or any orchestration system; experience in creating DAGs to run custom workflows is a plus. Jupyter Notebooks: Familiarity with Jupyter notebooks for data processing and analysis is a plus. Qualification : Bachelors degree in computer science, Software Engineering, or a related field.

We are seeking a skilled and driven Senior Data DevOps Engineer with strong MLOps expertise to join our team. The ideal candidate will have a deep understanding of data engineering, automation in data pipelines, and operationalizing machine learning models. The role requires a collaborative professional capable of designing, deploying, and managing scalable data and ML pipelines that align with business goals. #LI-DNI#EasyApply Responsibilities Develop, deploy, and manage CI/CD pipelines for data integration and machine learning model deployment Implement and maintain infrastructure for data processing and model training using cloud-based tools and services Automate data validation, transformation, and workflow orchestration processes Collaborate with data scientists, software engineers, and product teams to ensure seamless integration of ML models into production Optimize model serving and monitoring to enhance performance and reliability Maintain data versioning, lineage tracking, and reproducibility of ML experiments Proactively identify opportunities to improve deployment processes, scalability, and infrastructure resilience Ensure robust security measures are in place to protect data integrity and compliance with regulations Troubleshoot and resolve issues across the data and ML pipeline lifecycle Requirements Bachelor s or Master s degree in Computer Science, Data Engineering, or a related field 4+ years of experience in Data DevOps, MLOps, or related roles Strong proficiency in cloud platforms such as Azure, AWS, or GCP Experience with Infrastructure as Code (IaC) tools like Terraform, CloudFormation, or Ansible Expertise in containerization and orchestration technologies (e.g., Docker, Kubernetes) Hands-on experience with data processing frameworks (e.g., Apache Spark, Databricks) Proficiency in programming languages such as Python, with knowledge of data manipulation and ML libraries (e.g., Pandas, TensorFlow, PyTorch) Familiarity with CI/CD tools (e.g., Jenkins, GitLab CI/CD, GitHub Actions) Experience with version control tools (e.g., Git) and MLOps platforms (e.g., MLflow, Kubeflow) Strong understanding of monitoring, logging, and alerting systems (e.g., Prometheus, Grafana) Excellent problem-solving skills and the ability to work independently and as part of a team Strong communication and documentation skills Nice to have Experience with DataOps concepts and tools (e.g., Airflow, dbt) Knowledge of data governance and tools like Collibra Familiarity with Big Data technologies (e.g., Hadoop, Hive) Certifications in cloud platforms or data engineering We offer Opportunity to work on technical challenges that may impact across geographies Vast opportunities for self-development: online university, knowledge sharing opportunities globally, learning opportunities through external certifications Opportunity to share your ideas on international platforms Sponsored Tech Talks & Hackathons Unlimited access to LinkedIn learning solutions Possibility to relocate to any EPAM office for short and long-term projects Focused individual development Benefit package: Health benefits Retirement benefits Paid time off Flexible benefits Forums to explore beyond work passion (CSR, photography, painting, sports, etc.) Qualification : Bachelors or Masters degree in Computer Science, Data Engineering, or a related field