Data Pipelines Jobs in Hyderabad
339 Jobs Found
Ai Solution Architect
Zl Tech
AI Solution Architect Full-Time | Hyderabad, India Department: Engineering Experience: 10+ years in AI/ML Location: On-site | Hyderabad, Telangana Salary: Competitive, based on experience About the Role We are looking for an experienced and visionary AI Solutions Architect to design, build, and scale advanced AI/ML solutions as part of our cutting-edge ZL Platform. This role requires deep technical expertise in machine learning, large language models (LLMs), model deployment, and a strong understanding of customer-driven product development. You'll collaborate with cross-functional teams Engineering, Product, Sales to translate business needs into impactful AI systems. Key Responsibilities Architect and lead the development of scalable AI/ML solutions integrated with the ZL Platform. Partner with Engineering, Product, and Sales teams to deliver PoC (proof of concept) demos and ensure successful customer adoption. Take ownership of AI and machine learning components, ensuring high quality, maintainability, and performance. Collaborate directly with customers and stakeholders to understand business problems and deliver technically sound solutions. Stay current with AI research and industry trends including model quantization, transformers, and vector databases and integrate advancements into platform architecture. Fine-tune and deploy LLMs (Large Language Models) using frameworks like PyTorch and Hugging Face Transformers. Provide mentorship to engineering teams and contribute to a culture of technical excellence and innovation. Optimize models for deployment in real-world environments with a focus on latency, scalability, and resource efficiency. Translate business problems into AI opportunities and present solutions to internal and external stakeholders. Preferred Skills & Expertise Strong background in deep learning, natural language processing (NLP), and LLMs. Experience with PyTorch, transformers, and AI model quantization and deployment. Production-level experience in deploying ML models in enterprise SaaS environments. Working knowledge of vector databases and semantic search optimization. Proven experience in PoC development, technical solution design, and delivering enterprise AI solutions. Familiarity with cloud platforms (e.g., Azure AI, AWS, GCP) for model hosting and orchestration. Qualitative Expectations Strong ownership mindset you own problems end-to-end and proactively fill knowledge gaps to deliver solutions. Thrive in fast-paced, evolving environments with loosely defined problems and multiple priorities. Excellent communicator with the ability to present technical concepts to both technical and non-technical audiences. Passionate about learning and sharing findings with peers; an enthusiastic self-starter and collaborator. Strong track record of innovation and continuous improvement. Education & Experience Master s or PhD in Computer Science, Artificial Intelligence, Machine Learning, or a related discipline. 10+ years of professional experience in AI/ML, with hands-on experience designing, deploying, and scaling machine learning systems in production. Work at the forefront of enterprise AI and LLM innovation. Collaborate with world-class engineering, research, and product teams. Contribute to the development of AI products used by global enterprises. Competitive compensation and professional growth opportunities. Apply now to join us as an AI Solution Architect in Hyderabad. Qualification : Masters or PhD in Computer Science, Artificial Intelligence, Machine Learning, or a related discipline.
Data Governance Architect
Kanerika Software
Job Title: Data Governance Architect Job Location: Hyderabad Job Description As a Data Governance Architect at Kanerika, you will play a pivotal role in shaping and executing the enterprise data governance strategy. Your responsibilities include: What You Will Do: Strategy, Framework, and Governance Operating Model Develop and maintain enterprise-wide data governance strategies, standards, and policies. Align governance practices with business goals like regulatory compliance and analytics readiness. Define roles and responsibilities within the governance operating model. Drive governance maturity assessments and lead change management initiatives. Stakeholder Alignment & Organizational Enablement Collaborate across IT, legal, business, and compliance teams to align governance priorities. Define stewardship models and create enablement, training, and communication programs. Conduct onboarding sessions and workshops to promote governance awareness. Architecture Design for Data Governance Platforms Design scalable and modular data governance architecture. Evaluate tools like Microsoft Purview, Collibra, Alation, BigID, Informatica. Ensure integration with metadata, privacy, quality, and policy systems. Microsoft Purview Solution Architecture Lead end-to-end implementation and management of Microsoft Purview. Configure RBAC, collections, metadata scanning, business glossary, and classification rules. Implement sensitivity labels, insider risk controls, retention, data map, and audit dashboards. Metadata, Lineage & Glossary Management Architect metadata repositories and ingestion workflows. Ensure end-to-end lineage (ADF Synapse Power BI). Define governance over business glossary and approval workflows. Data Classification, Access & Policy Management Define and enforce rules for data classification, access, retention, and sharing. Align with GDPR, HIPAA, CCPA, SOX regulations. Use Microsoft Purview and MIP for policy enforcement automation. Data Quality Governance Define KPIs, validation rules, and remediation workflows for enterprise data quality. Design scalable quality frameworks integrated into data pipelines. Compliance, Risk, and Audit Oversight Identify risks and define standards for compliance reporting and audits. Configure usage analytics, alerts, and dashboards for policy enforcement. Automation & Integration Automate governance processes using PowerShell, Azure Functions, Logic Apps, REST APIs. Integrate governance tools with Azure Monitor, Synapse Link, Power BI, and third-party platforms. Requirements 15+ years in data governance and management. Expertise in Microsoft Purview, Informatica, and related platforms. Experience leading end-to-end governance initiatives. Strong understanding of metadata, lineage, policy management, and compliance regulations. Hands-on skills in Azure Data Factory, REST APIs, PowerShell, and governance architecture. Familiar with Agile methodologies and stakeholder communication. Benefits Employee Benefits Culture: Open Door Policy: Encourages open communication and accessibility to management. Open Office Floor Plan: Fosters a collaborative and interactive work environment. Flexible Working Hours: Allows employees to have flexibility in their work schedules. Employee Referral Bonus: Rewards employees for referring qualified candidates. Appraisal Process Twice a Year: Provides regular performance evaluations and feedback. Inclusivity and Diversity: Hiring practices that promote diversity: Ensures a diverse and inclusive workforce. Mandatory POSH training: Promotes a safe and respectful work environment. Health Insurance and Wellness Benefits: GMC and Term Insurance: Offers medical coverage and financial protection. Health Insurance: Provides coverage for medical expenses. Disability Insurance: Offers financial support in case of disability. Child Care & Parental Leave Benefits: Company-sponsored family events: Creates opportunities for employees and their families to bond. Generous Parental Leave: Allows parents to take time off after the birth or adoption of a child. Family Medical Leave: Offers leave for employees to take care of family members' medical needs. Perks and Time-Off Benefits: Company-sponsored outings: Organizes recreational activities for employees. Gratuity: Provides a monetary benefit as a token of appreciation. Provident Fund: Helps employees save for retirement. Generous PTO: Offers more than the industry standard for paid time off. Paid sick days: Allows employees to take paid time off when they are unwell. Paid holidays: Gives employees paid time off for designated holidays. Bereavement Leave: Provides time off for employees to grieve the loss of a loved one. Professional Development Benefits: L&D with FLEX- Enterprise Learning Repository: Provides access to a learning repository for professional development. Mentorship Program: Offers guidance and support from experienced professionals. Job Training: Provides training to enhance job-related skills. Professional Certification Reimbursements: Assists employees in obtaining professional certifications.
Data Engineering Manager
Kanerika Software
Job Title: Data Engineering Manager Location: Hyderabad About Kanerika Kanerika Inc. is a leading global software products and services company dedicated to empowering data-driven enterprises. We help businesses accelerate digital transformation and maximize impact by delivering innovative, custom data solutions that optimize operations, enhance customer experiences, and fuel growth. Our Accolades Best Place to Work 2022 Great Place to Work Top 10 Most Recommended RPA Startups 2022 RPA Today Technology Innovation Award 2021 Frost & Sullivan India for Kompass composable solution Recognized for strong data privacy and security standards: ISO 27701, SOC2, GDPR compliant With a 4.5/5 Glassdoor rating, Kanerika fosters an inclusive, growth-oriented culture where employees thrive. We offer: Flexible working hours and generous parental leave Mandatory training on a safe and inclusive work environment Mentorship, professional development, and certification support Company-sponsored outings and work-life balance initiatives Join a vibrant, diverse community where your contributions truly matter. The Role We are looking for an experienced Data Engineering Manager to lead and inspire our data engineering team in Hyderabad. Partnering closely with Data Architects, you ll drive the design and development of scalable, cloud-based data pipelines and models, ensuring data quality, security, and performance. What You ll Do Lead and mentor a high-performing data engineering team, driving productivity and accountability. Collaborate with architects and development leads to build scalable, enterprise-grade data solutions. Oversee agile workflows, manage escalations, and coordinate hiring and resource planning. Use data-driven insights to guide team priorities and continuous improvement. Support pre-sales activities and work cross-functionally with sales and recruitment teams. Foster a culture of innovation, quality-first mindset, and customer focus. Who You Are A proactive leader with experience managing mid to large-sized teams and complex projects. An effective communicator, both verbally and in writing, with a knack for visually presenting ideas and core values. Results-oriented with a proven track record of delivering successful projects and publishing customer case studies. A process-driven, independent thinker who thrives under pressure and embraces continuous learning. Passionate about inclusivity and committed to contributing authentically to a diverse team environment.
Senior Python Developer
Effiasoft
Job Title: Senior Python Developer Location: Hyderabad (Onsite) Experience: 5+ Years Job Type: Full-Time Openings: 2 Category: IT Job Summary We are seeking an experienced Senior Python Developer with deep technical expertise in designing scalable, secure, and high-performance backend systems particularly within the travel and tourism industry. The ideal candidate will be proficient in Python frameworks (Django, Flask, FastAPI), capable of handling complex third-party API integrations, and demonstrate leadership in ensuring robust, compliant, and efficient backend infrastructure. Roles & Responsibilities Core Development Architect, build, and maintain scalable applications using Python and frameworks such as Django, Flask, and FastAPI. Integrate and optimize third-party travel APIs (e.g., GDS, CRS, OTA, airline APIs) for real-time booking, availability, and pricing data. Develop RESTful APIs, GraphQL, and asynchronous services for seamless frontend-backend communication. Data & Architecture Design and manage high-throughput data pipelines and ETL processes for structured and unstructured data. Work with SQL/NoSQL databases like PostgreSQL and MongoDB, and streaming tools like Apache Kafka or Apache Spark. Build modular microservices to enable independent scaling and deployment. Performance, Security & Compliance Optimize for performance, scalability, and reliability with efficient load balancing and low-latency design. Implement secure coding practices, ensure data encryption, and maintain compliance with PCI DSS, GDPR, and other relevant standards. DevOps & Automation Set up and maintain CI/CD pipelines (Jenkins, GitLab CI), automate testing and deployment. Leverage Docker and Kubernetes for containerization and orchestration. Use monitoring tools to ensure system health and proactive issue resolution. Collaboration Work closely with frontend teams, designers, and stakeholders to align backend systems with business requirements. Participate in code reviews, documentation, sprint planning, and continuous improvement. Required Qualifications Bachelor s or Master s degree in Computer Science, Engineering, or related field. Minimum 5 years of experience in backend development with at least 3 years in Django, Flask, or FastAPI. Hands-on experience in RESTful API, GraphQL, and asynchronous programming. Deep knowledge of PostgreSQL, MongoDB, and streaming/big data tools like Kafka, Spark. Experience in cloud platforms (AWS, Azure, GCP). Experience with Docker, Kubernetes, and CI/CD tools. Familiarity with testing frameworks such as PyTest, Selenium, and SonarQube. Travel domain knowledge experience integrating GDS/CRS, booking systems, and payment gateways. Key Competencies Strong problem-solving and analytical thinking. Ability to work in a fast-paced, high-pressure environment. Excellent communication and collaboration skills. Self-motivated with a proactive attitude and leadership qualities.
Senior Data Engineer
Blend360 India
Senior Data Engineer Hyderabad (On-site, Full-time) Location: Hyderabad, Telangana Work Type: On-site, Full-time Experience: 5+ Years Industry: AI & Data Science Company: Blend About Blend: Blend is a leading AI services company focused on solving real-world problems through the fusion of artificial intelligence, data science, and human ingenuity. We believe in co-creating meaningful impact by aligning top-tier talent with cutting-edge technology. Our goal is to deliver transformative solutions that unlock value for our clients and enrich the work and lives of our people. Job Overview: We are seeking a highly skilled Senior Data Engineer with strong experience in AWS, Databricks, Python, SQL, and Apache Spark/PySpark. You will be responsible for designing, developing, and maintaining scalable data architectures and pipelines to support analytics, machine learning, and business intelligence efforts across the organization. Key Responsibilities: Design and build scalable data pipelines using AWS services (Glue, Lambda, S3, EMR) and Databricks Notebooks & Workflows. Develop and manage robust ETL/ELT workflows using Python and SQL to handle structured and semi-structured data. Implement distributed data processing using Apache Spark or PySpark. Create and maintain data lakes in AWS Databricks. Collaborate with data scientists, analysts, and cross-functional teams to provide high-quality, accessible datasets. Ensure adherence to data governance, quality, and security best practices. Monitor, troubleshoot, and optimize the performance of data systems. Conduct code reviews and enforce engineering best practices. Mentor junior engineers and support their technical growth. Required Qualifications: Bachelor's or Master s degree in Computer Science, Engineering, or related field. 5+ years of experience in Data Engineering, with 2+ years working in AWS Databricks environments. Strong programming skills in Python for data processing and workflow automation. Advanced expertise in SQL for data transformation and querying large datasets. Proficient in distributed data processing using Apache Spark/PySpark. Solid understanding of data modeling, data warehousing, and performance optimization techniques. Hands-on experience with AWS services such as Glue, S3, Lambda, and EMR. Familiarity with version control tools like Git or CodeCommit. Experience with workflow orchestration tools like Airflow or AWS Step Functions (preferred but not mandatory). Be part of a forward-thinking, AI-driven organization. Work with cutting-edge cloud technologies and data platforms. Thrive in a culture of innovation, mentorship, and continuous learning. Enjoy opportunities to grow technically and professionally while driving real-world impact. Apply Now to join Blend as a Senior Data Engineer and help build data infrastructure that powers intelligent decision-making. Qualification : Bachelor's or Masters degree in Computer Science, Engineering, or related field.
Snowflake Architect
Blend360 India
Snowflake Architect Location: Hyderabad, Telangana Work Type: On-site, Full-time Company: Blend About Blend: Blend is a premier AI services provider dedicated to driving innovation and value through data science, AI, and technology. Our mission is to co-create meaningful impact by combining human expertise with advanced AI-driven strategies. Job Summary: We are seeking an experienced Snowflake Architect to lead our data engineering efforts by designing scalable Snowflake architectures, optimizing data pipelines, and managing a high-performing team. This role requires a strategic thinker and hands-on leader to deliver secure, reliable, and efficient data solutions. Key Responsibilities: Design and implement robust Snowflake data warehouse architectures and ETL pipelines supporting business intelligence and advanced analytics. Lead and mentor a team of data engineers, ensuring timely delivery of high-quality projects. Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and design effective data models. Develop, document, and enforce best practices for Snowflake architecture, data modeling, performance tuning, and ETL workflows. Optimize Snowflake environments for low latency, high availability, and cost efficiency. Drive continuous process improvements and evaluate emerging tools and technologies. Ensure data accuracy, completeness, and security, complying with data privacy regulations like GDPR and CCPA. Partner with cloud engineering and DevOps teams to integrate Snowflake solutions within AWS infrastructure. Participate in capacity planning, budgeting, and resource allocation for data engineering functions. Qualifications: Bachelor s degree in Computer Science, Information Technology, or related field (Master s preferred). 9+ years of cloud-based data engineering experience, with at least 4 years of hands-on Snowflake expertise. Proven track record in designing, deploying, and managing large-scale Snowflake data platforms. Expertise with AWS services including S3, Redshift, Lambda, Glue, etc. Strong proficiency in SQL, Python, and other data manipulation languages. Experience managing and mentoring engineering teams and leading cross-functional projects. Solid understanding of data governance, security, and compliance frameworks (GDPR, CCPA). Excellent problem-solving, communication, and leadership skills. Preferred Qualifications: Snowflake certification(s). Experience with other cloud data warehouses such as AWS Redshift or Google BigQuery. Familiarity with Agile methodologies, CI/CD pipelines, Git, Jira, and modern development practices. Apply now to join Blend and help shape the future of data-driven innovation! Qualification : Bachelors degree in Computer Science, Information Technology, or related field (Masters preferred).
Azure Data Engineer (mid)
Syren Technologies
Azure Data Engineer (Mid-Level) Hyderabad Location: Hyderabad, India Job Type: Full-Time | On-site/Hybrid Experience Level: 5+ Years Industry: Information Technology / Cloud Data Engineering Job Overview We are hiring an experienced Azure Data Engineer (Mid-Level) to join our team in Hyderabad. In this role, you will work closely with business stakeholders, analysts, and IT teams to design, implement, and manage scalable, high-performance data solutions on the Azure Cloud Platform. Your expertise in data modeling, ETL development, and cloud-based data warehousing will be critical in supporting data-driven decision-making across the organization. Key Responsibilities Analyze large datasets and deliver clear, actionable reports using statistical techniques. Design and implement data pipelines, data collection systems, and analytics strategies that enhance operational efficiency. Develop and maintain data systems, ensuring high data quality, consistency, and integrity. Collaborate with business stakeholders, analysts, and developers to translate business requirements into technical solutions. Design, build, and manage distributed data systems and applications on the Azure Cloud. Lead and execute ETL development, including data extraction, transformation, and loading using tools like Informatica or SSIS. Conduct technical design reviews, performance tuning, and testing of data workflows. Adapt to changing priorities and contribute as an individual contributor or team leader depending on project scope. Required Qualifications & Skills Bachelor s degree in Computer Science, Information Systems, or a related field. Minimum 5+ years of experience in data engineering, data analysis, or database design. Proficient in SQL and T-SQL for querying and managing relational databases. Hands-on experience with ETL frameworks such as Informatica or SSIS. Strong understanding of data modeling, data mining, and segmentation techniques. Experience working with structured and unstructured data sources. Proficient in statistical tools and packages (Excel, SSIS, SAS, etc.). Strong analytical mindset with excellent attention to detail and data accuracy. Preferred Skills (Good to Have) Experience with Python and Apache Spark for data manipulation and automation. Experience with modern data platforms: Azure Synapse Analytics Azure Databricks Amazon Redshift Prior experience in developing automation solutions using Python or similar scripting languages. Knowledge of Enterprise Data Warehousing (EDW) concepts and solutions. Competitive compensation package with benefits. Work with cutting-edge cloud technologies in real-world environments. Collaborative, fast-paced, and innovation-driven workplace. Career development support with continuous learning opportunities. A chance to make a tangible impact on enterprise-level data systems. Apply Now If you are a data-driven professional with a strong command of Azure and ETL tools, and you re looking to advance your career in data engineering, apply now and become part of a forward-thinking tech team based in Hyderabad! Qualification : Bachelors degree in Computer Science, Information Systems, or a related field.
Data Architect
Syren Technologies
Data Architect Hyderabad Experience: 10-17 Years Job Type: Full-Time | On-site/Hybrid Location: Hyderabad, India Industry: Data Engineering / Cloud Architecture Job Overview We are looking for an experienced Data Architect with 10+ years of expertise in designing, optimizing, and managing enterprise-scale data architectures on Azure Cloud. The ideal candidate will lead efforts to enhance data pipeline performance, ensure robust data governance, and drive cost-effective cloud resource utilization. If you excel in architecting scalable, secure, and high-performing data solutions, this is your chance to join a dynamic team in Hyderabad. Key Roles & Responsibilities Design, optimize, and maintain data pipelines, ETL processes, and complex queries to maximize performance and scalability. Improve query execution plans, indexing strategies, and optimize storage in Azure SQL, Azure Synapse Analytics, and Azure Data Lake. Implement advanced data management techniques such as data partitioning, caching, and compression for efficient data processing. Monitor cloud resource consumption and implement cost-saving strategies without impacting system performance. Optimize use of Azure services including Databricks, Data Factory, Synapse Analytics, and storage accounts to balance performance and cost. Establish and enforce coding standards, design patterns, and best practices for data pipeline development and cloud infrastructure management. Define and implement data governance, security, and compliance policies aligned with organizational and regulatory requirements. Develop and oversee CI/CD pipelines using DevOps methodologies tailored to data engineering workflows. Implement monitoring, alerting, and logging solutions using Azure Monitor, Log Analytics, and Application Insights to ensure system reliability. Architect solutions for high availability and disaster recovery to maintain business continuity. Optimize Power BI data models, DAX queries, and report performance; ensure governance and data refresh strategies meet business needs. Collaborate with cross-functional teams and business stakeholders to translate requirements into scalable and actionable data insights. Required Skills & Qualifications 10+ years of experience in data architecture, cloud data engineering, or related roles. Deep expertise in Azure Cloud data services: Azure SQL, Synapse Analytics, Data Lake, Databricks, Data Factory. Strong background in ETL pipeline optimization, query tuning, and storage management. Proven experience with data governance, security frameworks, and compliance standards. Familiarity with DevOps principles, especially CI/CD for data workflows. Expertise in monitoring tools like Azure Monitor, Log Analytics, and Application Insights. Hands-on experience optimizing Power BI solutions, including data modeling and report design. Excellent communication and stakeholder management skills. Work on large-scale, cutting-edge data projects impacting enterprise business decisions. Collaborate with a passionate, skilled team in a fast-paced, innovative environment. Competitive salary and benefits package. Opportunity to shape the future of data architecture within a leading organization. Apply Today If you are an expert Data Architect ready to take your career to the next level with advanced Azure data technologies, apply now and join our Hyderabad team!
Data Engineering Support Engineer - Sima
Syren Technologies
Data Engineering Support Engineer SIMA (5+ Years) Hyderabad Location: Hyderabad, India Job Type: Full-Time | On-site/Hybrid Experience: 5+ Years Industry: Data Engineering / Cloud Support Job Overview We are seeking an experienced Data Engineering Support Engineer with a strong background in maintaining and optimizing data pipelines, ensuring platform reliability, and supporting cloud data infrastructure. This role is critical to sustaining smooth data operations for the SIMA platform in Hyderabad. Key Responsibilities Maintain and support data pipelines and transformations on Azure cloud platforms. Ensure platform stability and reliability through proactive monitoring and issue resolution. Manage scheduling and monitoring using Azure Data Factory (ADF) triggers, Azure Monitor, and Log Analytics (KQL). Handle data ingestion and validation workflows via ADF, Event Hub, APIs, and PySpark, including managing schema drift. Manage CI/CD pipelines using Azure DevOps YAML, deploying solutions for ADF, Synapse, and Databricks. Optimize data transformations and performance using PySpark and Delta Lake techniques like partitioning and caching. Implement and maintain security best practices using Azure RBAC, Key Vault, NSGs, and Integration Runtime configuration. Conduct Root Cause Analysis (RCA), document findings, maintain disaster recovery playbooks, and implement proactive alerting. Required Skills 5+ years of hands-on experience in data engineering support roles. Expertise in Azure Data Factory (ADF) scheduling, triggers, and monitoring. Proficient with PySpark for data transformations and performance optimization. Experience with Delta Lake optimizations including partitioning and caching. Skilled in CI/CD practices using Azure DevOps for deploying data workflows. Strong knowledge of Azure security features: RBAC, Key Vault, Network Security Groups (NSGs). Ability to perform detailed Root Cause Analysis (RCA) and develop disaster recovery strategies. Familiarity with data ingestion tools like Event Hub and API integrations. Experience with Kusto Query Language (KQL) for Log Analytics queries. Join a dynamic team maintaining critical data infrastructure supporting business operations. Work with advanced Azure data services and cloud-based data platforms. Competitive salary and comprehensive benefits. Opportunity for growth and skill development in cloud data engineering. Apply Now If you are a detail-oriented data engineering professional with expertise in Azure platform support, apply today to become a vital part of our Hyderabad-based SIMA team!
Senior Data Engineer
Wallero Technologies
Job Title: Senior Data Engineer Location: Hyderabad, Telangana, India Job Type: Full-Time | Permanent Experience Required: 6+ Years Open Positions: 2 About the Role: We are seeking a highly skilled and experienced Data Engineer to join our team in Hyderabad. As a key member of our data engineering unit, you will be responsible for designing and optimizing end-to-end data pipelines using Python, PySpark, and Databricks. This role requires strong technical expertise, a passion for solving complex data challenges, and the ability to lead best practices within the Supply Chain data domain. Key Responsibilities: Develop and maintain scalable data pipelines using Python, PySpark, and Databricks for complex data transformation within Supply Chain operations. Implement advanced data engineering solutions that support real-time and batch data processing for analytics and business intelligence. Analyze, optimize, and refactor existing data workflows to improve performance and reliability. Collaborate with cross-functional teams to understand data requirements and translate them into robust technical solutions. Take ownership of projects and deliver high-quality outcomes with minimal supervision. Provide leadership and guidance on coding standards, design patterns, and data engineering best practices. Clearly communicate technical ideas and solutions to both technical and non-technical stakeholders. Required Skills & Qualifications: 6+ years of hands-on experience in data engineering. Proficient in Python, PySpark, and Databricks for building and maintaining data pipelines. Strong problem-solving skills with a proven ability to create innovative solutions for complex data challenges. Excellent communication and interpersonal skills; able to work effectively with cross-functional teams. Experience leading and mentoring team members on technical best practices and architecture decisions. Ability to work independently, manage priorities, and drive projects to completion with minimal oversight. Perks & Benefits: Competitive compensation package Exposure to cutting-edge technologies in cloud-based data engineering Opportunity to work on large-scale Supply Chain data solutions Collaborative, innovation-driven work environment Career growth and mentorship opportunities
Senior Data Scientist
Microsoft
Senior Data Scientist Location: Hyderabad, Telangana, India Employment Type: Full-Time Overview Microsoft Search powers delightful and relevant experiences for enterprise users, and AI-driven Copilots have turned futuristic visions into current-day reality. We are seeking candidates who can help build the next-generation machine learning systems to drive innovation in search ranking, clustering, recommendation systems, and much more. As a Senior Data Scientist in the team, you will work independently on complex problems, define success metrics, and contribute to design and review processes for machine learning/data pipelines across the team. If you're ready to tackle challenging data science problems and help shape the future of search, we want you on our team! Key Responsibilities End-to-End Ranking: Oversee the ranking of search results and contribute to enhancing search relevance. Large Language Models (LLMs): Work on developing and deploying large language models to improve search experience and AI-powered copilots. AI-powered Copilot Scenarios: Drive innovations in AI systems that enhance user productivity and engagement through Copilot capabilities. Compliance & Privacy: Develop machine learning training pipelines at scale that are privacy and compliance-aware. Personalization: Customize search results and recommend relevant content based on user preferences. Qualifications Required Qualifications: Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or a related field AND 2+ years of data science experience OR Master s Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or a related field AND 5+ years of data science experience OR Bachelor s Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years of data science experience. Experience working with both structured and unstructured data, applying statistical techniques, and reporting results. Hands-on experience with implementing and shipping production-quality machine learning models or data pipelines. 2+ years of customer-facing, project-delivery experience, professional services, and/or consulting experience. Strong programming skills in Python, R, or similar languages, and familiarity with machine learning libraries. Preferred Qualifications: Prior work or research experience in the Search domain is highly preferred. Familiarity with large-scale distributed systems, cloud platforms (such as Azure), and real-time machine learning applications. Experience with deep learning and large language models (LLMs). Familiarity with compliance and privacy in machine learning pipelines. Microsoft s mission is to empower every person and organization on the planet to achieve more. At Microsoft Search, you will be part of a team solving impactful problems for millions globally. Join us to drive cutting-edge advancements in AI and contribute to shaping the future of search and user experiences. Employee Benefits Industry-leading healthcare coverage Generous time off and family leave policies Access to educational resources and professional development Employee discounts on Microsoft products and services Opportunities for networking and global collaborations Inclusive and supportive work environment with a focus on work-life balance Microsoft is an equal opportunity employer. We are committed to creating a diverse and inclusive environment where every individual is treated with respect and can thrive. Join us to make a global impact through innovation, collaboration, and diversity.
Azure Data Engineer
Quadrant Technologies
Job Title: Azure Data Engineer Location: Hyderabad / Bangalore (Hybrid) Job Type: Full-time Salary: Negotiable Position Overview: We are looking for an Azure Data Engineer with 5-8 years of experience to join our team. As an Azure Data Engineer, you will design, implement, and manage data solutions on the Microsoft Azure cloud platform. The ideal candidate will have a strong background in writing complex SQL stored procedures, implementing OLTP database solutions using Microsoft SQL Server, and working with Azure Synapse, PySpark, and Azure Databricks for big data processing. Key Responsibilities: Data Integration and Collaboration: Collaborate with cross-functional teams to gather, analyze, and document business requirements for data integration projects. Complex SQL Expertise: Write and optimize complex T-SQL, Dynamic SQL, and Spark SQL queries, including the creation of stored procedures for data transformation and business validation logic. Big Data Processing: Utilize Azure Synapse/Databricks to write notebooks using Spark SQL or PySpark for processing and analyzing large volumes of data. Data Pipeline Development: Develop and maintain robust data pipelines using Azure Data Factory to ensure seamless data flow across systems. OLTP Systems: Contribute to the enhancement and optimization of OLTP systems using SQL Server. Data Quality and Integrity: Ensure data quality, integrity, and accuracy in all systems. Programming and Algorithm Development: Utilize strong algorithmic programming skills to handle data processing and transformation. Qualifications: Educational Background: Bachelor s degree in Computer Science, Information Technology, or a related field. Experience: 5-8 years of experience in database development, data integration, and writing complex T-SQL queries. Technical Skills: Expertise in T-SQL and Spark SQL/PySpark (experience with Azure Synapse and Databricks is essential). Hands-on experience with Azure Data Factory for building data pipelines. Strong experience with OLTP systems using SQL Server. Problem-Solving Skills: Excellent problem-solving abilities with a strong focus on attention to detail. Adaptability: Ability to work in a collaborative and fast-paced environment, focusing on delivering high-quality solutions. Other Details: Notice Period: 0-15 days MAX Work Location: Hyderabad / Bangalore (Hybrid) If you're an experienced Azure Data Engineer with a passion for big data and cloud technologies, apply now to join our team and make an impact with innovative data solutions! Qualification : Bachelors degree in Computer Science, Information Technology, or a related field.
Senior Data Engineer
Zessta Software Services Pvt.ltd.
Job Title: Senior Data Engineer Company Name: Zessta Job Type: Full-time Responsibilities: Analyze customer data to enhance customer experience and provide actionable insights. Build and maintain integrations to support advanced data science workflows, ensuring seamless interaction with the Product API. Actively participate in onboarding and training clients on how to use the product effectively. Provide insights into client-to-business interactions, ensuring that the data can be leveraged for strategic decisions. Act as a liaison between the client and the organization, ensuring both parties' needs and expectations are met. Identify opportunities for clients to expand their usage of the product, driving engagement and long-term value. Address customer complaints and requests, offering product support to improve overall satisfaction and experience. Skills and Experience: Excellent communication and interpersonal skills to work effectively with clients and internal teams. Strong understanding of data modeling and architecture. Hands-on experience with data integrations and data pipelining tools to connect systems and enable smooth data flow. Proficiency in programming languages, preferably Python or R, to process and analyze data. Experience with container orchestration platforms like Docker and Kubernetes. Familiarity with distributed computing frameworks such as Spark, Hadoop, and Dask to handle large-scale data processing. Experience with version control software such as Git to manage code and collaborate with teams. Knowledge of project management tools such as JIRA to track and manage project progress and tasks. Familiarity with tools like Figma and Mixpanel is a plus. Education and Experience: Bachelor s or Master s degree from premier institutes (preferred). 4-12 years of relevant experience in data engineering, data science, or related fields. What We Offer: Competitive salary and benefits package. Opportunities for career growth and skill development. The chance to work on cutting-edge technologies and impactful projects. If you have a passion for data engineering and improving customer experience, and you're ready to take on new challenges, we'd love to hear from you. Apply now to join our team at Zessta and contribute to driving innovation and business success! Qualification : Bachelors or Masters degree from premier institutes (preferred).
Data Engineer-data Platforms
International Business Machines Corporation
Introduction In this role, you'll be part of one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients globally. Our centers provide locally-based skills and technical expertise to drive innovation and the adoption of new technologies. A career at IBM Consulting involves building long-term relationships and collaborating closely with clients across industries. You'll work alongside visionaries to help them on their hybrid cloud and AI journeys, enhancing the value and impact of some of the most innovative companies in the world. With access to IBM s strategic partner ecosystem and cutting-edge technology platforms, including IBM Software and Red Hat, you will be empowered to accelerate client transformations and create meaningful change. Curiosity and a constant quest for knowledge serve as the foundation for success at IBM. In your role, you'll be encouraged to challenge the status quo, explore ideas beyond your current role, and come up with creative solutions that will make a significant impact across a wide network of clients. Our culture of evolution and empathy is centered on long-term career growth and development opportunities in an environment that celebrates your unique skills and experience. Your Role and Responsibilities As a Data Engineer at IBM, you will be a critical player in designing and developing applications, while also providing guidance to project teams on complex coding, troubleshooting, and execution. Your key responsibilities include: Solution Design and Construction: Lead the design and construction of new data solutions using the latest technologies, ensuring business value and meeting user requirements. Continuous Improvement: Advocate for continuous improvement by testing solutions and working within an Agile framework to enhance and iterate the developed solutions. Innovation: Stay current with the latest technology trends, discovering and implementing creative solutions to solve business problems. Collaboration: Work with diverse teams to deliver impactful data solutions while ensuring scalability, quality, and performance. Required Education Bachelor s Degree in a relevant field Required Technical and Professional Expertise Apache Spark (PySpark): In-depth knowledge of Spark s architecture, core APIs, and using PySpark for distributed data processing. Big Data Technologies: Familiarity with Hadoop, HDFS, Kafka, and other big data tools. Data Engineering Skills: Strong understanding of ETL pipelines, data modeling, and data warehousing concepts. Proficiency in Python: Expertise in Python programming, particularly focused on data processing and manipulation. Data Processing Frameworks: Experience with Pandas, NumPy, and other data processing libraries. SQL Proficiency: Experience writing optimized SQL queries for large-scale data analysis and transformation. Cloud Platforms: Experience working with cloud platforms such as AWS, Azure, or GCP, including working with cloud storage systems. Preferred Technical and Professional Experience Architecture Strategy: Experience in defining, driving, and implementing architecture strategies and standards for end-to-end monitoring of data solutions. Cross-Team Collaboration: Partnering with technology teams, including application development, enterprise architecture, testing services, and network engineering. Security: Familiarity with detection and prevention tools for company products and customer-facing platforms. How You ll Grow At IBM, we offer opportunities for career development and provide an environment where you are encouraged to continuously grow, learn, and expand your expertise. Working with cutting-edge technologies and a diverse set of clients, you will be able to stay ahead of industry trends and make meaningful contributions that have a real impact. About IBM Consulting IBM Consulting helps organizations tackle their most pressing challenges through innovative and agile solutions. By partnering with clients, we drive technology transformations, helping them optimize their operations and leverage new technologies like AI, cloud, and data. Our team s expertise spans across many industries, delivering solutions that enable clients to navigate digital transformations. This role offers an exciting opportunity to work with leading technologies while contributing to client success in a highly dynamic environment. If you re passionate about solving complex data problems and delivering impactful solutions, IBM Consulting is the place to grow your career. Qualification : Bachelors Degree in a relevant field
Azure DevSecOps Engineer
Valuelabs
Job Title: Azure DevSecOps Engineer Experience: 5-8 Years Location: Hyderabad About Us: We are a leading technology solutions provider focused on delivering secure, scalable, and innovative cloud solutions. As part of our expansion, we are looking for an experienced Azure DevSecOps Engineer to join our dynamic team. If you have expertise in Azure cloud services, security practices, and DevSecOps automation, we would love to have you onboard! Responsibilities: Cloud Services: Work extensively with Azure cloud services such as IAAS and PAAS, as well as Azure Data Bricks and Azure Data Factory (ADF) for managing data pipelines and analytics workflows. Ensure the security and scalability of cloud services and platforms. Azure Security & Policies: Manage Azure Active Directory (Azure AD) configurations and user access. Implement and maintain Azure Policies to enforce governance and compliance across Azure resources. Ensure secure management of cloud resources using industry-standard security practices. Networking & Monitoring: Work on Azure networking components and security, including network security groups, virtual networks, and firewalls. Configure and monitor Azure Monitoring and third-party monitoring tools to track and report performance, security, and availability metrics of cloud infrastructure. DevSecOps Automation: Implement Azure DevOps pipelines (YAML pipelines) for continuous integration and continuous deployment (CI/CD). Automate Infrastructure as Code (IAC) using Terraform or Bicep to provision and manage Azure resources in a scalable and efficient manner. Collaboration & Documentation: Work closely with cross-functional teams to ensure secure, automated, and reliable cloud infrastructure deployments. Provide documentation for cloud infrastructure, automation processes, and security procedures to maintain standard operating practices. Requirements: Experience: Total 5-7 years of experience in IT infrastructure and cloud technologies, with at least 2-3 years of hands-on experience in Azure cloud services, specifically IAAS, PAAS, Azure Data Bricks, and Azure Data Factory (ADF). Technical Skills: Expertise in Azure AD, Azure Policies, and Azure Networking and Security. Strong experience with Azure Monitoring and third-party monitoring tools for cloud infrastructure. Proficient in Azure DevOps and YAML pipelines for CI/CD automation. Hands-on experience with Infrastructure as Code (IAC) tools like Terraform or Bicep to automate cloud resource provisioning. Soft Skills: Excellent communication and collaboration skills to work with cross-functional teams and clients. Ability to troubleshoot and resolve complex cloud infrastructure issues. Strong problem-solving and analytical skills, with a focus on security in cloud infrastructure. Innovative Environment: Work with cutting-edge cloud technologies in a forward-thinking, fast-paced environment. Growth Opportunities: Enjoy continuous learning and career development through training and certifications. Collaborative Culture: Join a team of talented professionals committed to innovation and quality. Competitive Compensation & Benefits: Receive a competitive salary and comprehensive benefits package. Apply Today! If you're an experienced Azure DevSecOps Engineer with a passion for cloud security, automation, and DevOps, we invite you to apply and be part of our growing team!
Backend Developer (Python + AWS)
Valuelabs
Job Title: Backend Developer (Python + AWS) Experience: 5-8 Years Location: Hyderabad About Us: We are a cutting-edge technology company looking for an experienced Backend Developer (Python + AWS) to join our dynamic team. If you are passionate about backend development, have a strong expertise in Python, Django, and AWS, and want to work on innovative projects, we want to hear from you! Responsibilities: Backend Development: Design, develop, test, and deploy high-quality backend services using Python and Django. Develop robust RESTful APIs to facilitate communication between the frontend and microservices. Ensure seamless integration and data exchange between backend services and frontend applications. Collaboration: Work closely with the frontend development team to integrate backend services with the UI effectively. Collaborate with the Data Engineering team to design and implement data pipelines and storage solutions. Database Management: Develop and maintain database schemas, ensuring data consistency, integrity, and optimized performance. Troubleshooting & Debugging: Troubleshoot and debug complex issues in backend services and APIs, ensuring smooth and efficient operations. Code Quality & Best Practices: Participate in code reviews and ensure that the team adheres to high coding standards and best practices. Stay up-to-date with the latest trends and technologies in backend development and apply them to improve system efficiency. DevOps Collaboration: Collaborate with the DevOps team for smooth deployment and monitoring of backend services. Feature Implementation & Documentation: Contribute to the design and implementation of new features and functionalities. Develop and maintain clear and concise technical documentation and guides for backend services. QA Collaboration: Work closely with the QA team to ensure thorough testing, validation, and quality assurance of backend services. Task Planning: Participate in the planning and prioritization of project tasks and deliverables to ensure smooth progress. Requirements: Primary Skills: Backend Services Programming in Python Extensive experience with backend programming using Python. Django Framework: Experience working with Django for developing backend services. RESTful APIs: Solid experience in building and maintaining RESTful APIs. AWS Services: Proficiency with AWS Services such as S3, Lambda, EC2/ECS, IAM, and DynamoDB for cloud-based applications. Database Management: Strong experience in managing databases, schema design, and ensuring data consistency and integrity. Experience: 5+ years of experience in backend development with a focus on Python and AWS services. Additional Skills: Strong problem-solving skills and the ability to debug complex issues. Effective communication skills and ability to work well in a collaborative, team-oriented environment. Can-do attitude with a passion for developing scalable software solutions. A B.Sc. in Computer Science or related fields (or equivalent experience). Innovative Environment: Work on groundbreaking projects using the latest technologies in the backend development space. Collaborative Team Culture: Be part of a team that values open communication and collaboration. Career Growth: Opportunities to learn, grow, and advance your career with continuous skill development and training. Competitive Benefits: We offer competitive compensation and a comprehensive benefits package to support your well-being. Apply Today! If you re a skilled Backend Developer with a passion for Python, AWS, and backend technologies, we encourage you to apply and join our growing team! Qualification : A B.Sc. in Computer Science or related fields (or equivalent experience).
Data Engineer
Wallero Technologies
Job Description Location: Hyderabad, Telangana, India Job Type: Permanent (full-time) Experience: 6+ years No. of Openings: 2 Key Responsibilities: Tech Stack Expertise: Proficiency in Python, Pyspark, and Databricks, utilizing these tools to develop robust data pipelines, perform data transformations, and optimize data workflows within the Supply Chain realm. Innovative Problem-Solving: Demonstrate technical expertise by thinking creatively and providing innovative solutions to complex data-related challenges, pushing boundaries in Supply Chain data analytics. Communication Skills: Exceptional communication abilities to effectively convey technical concepts, collaborate with cross-functional teams, and articulate solutions to non-technical stakeholders. Autonomous Work: Ability to work independently and proactively, taking ownership of projects, and delivering high-quality solutions with minimal supervision. Guidance and Best Practices: Lead the team by setting coding standards, advocating for reusable and modularized code, implementing design patterns, and guiding on best practices in data engineering. Perks and Benefits: Strong expertise in Python, Pyspark, and Databricks, with hands-on experience in developing and optimizing data pipelines for complex data transformations. Exceptional problem-solving skills, with a demonstrated ability to think creatively and develop innovative solutions for data-related issues. Excellent communication and interpersonal skills, with the ability to collaborate effectively with diverse teams and stakeholders. Self-driven and autonomous work style, capable of taking ownership of projects and delivering high-quality results independently. Experience in guiding and mentoring teams on coding standards, design patterns, and best practices in data engineering.
Kafka Developer
Wilco Source Technologies Pvt. Ltd
Job Requirements What You Will Do Design and develop high-volume, low-latency applications for mission-critical systems, ensuring high availability and performance. Drive the design, development, and delivery of cutting-edge features for Kafka on Confluent / Apache Kafka. Develop programs using Kafka APIs and Kafka Streams API. Define partitioning strategies and configure brokers, topics, producers, consumers, schema registry, and KSQL. Optimize and tune Kafka performance to ensure efficiency and scalability. Implement kStream applications in Java, including handling grouping, aggregation, and more. Demonstrate scaling expertise in managing kStream applications. Ensure designs comply with specifications and best practices. Contribute to all phases of the development lifecycle, from requirement analysis to deployment and maintenance. Apply knowledge and experience with cloud technologies to support scalable and resilient Kafka deployments.
AI/ML Developer
Infor
Position: AI/ML Developer Company: Infor Location: Hyderabad, India Experience: 2-5 years Skills: AI/ML, Python, AWS, SQL Role Summary: As an AI/ML Developer at Infor, you will be at the forefront of creating and delivering state-of-the-art enterprise applications and automation solutions powered by Infor s AI, RPA, and OS platform technologies. Your primary mission will be to identify impactful AI/ML use cases, develop proof of concepts (PoCs), and deliver automation solutions that drive workforce productivity and enhance business outcomes for our clients. Key Responsibilities: Use Case Identification: Work closely with customers to understand their business challenges and identify opportunities for AI/ML solutions to address these needs. Data Insights: Perform exploratory data analysis on large, complex datasets to uncover key insights. Assess data quality and share findings that can inform decision-making. Data Preparation: Collect, clean, preprocess, and augment datasets for training and testing AI models, ensuring they are ready for AI tasks. Model Development: Design, train, and fine-tune AI/ML models, evaluating them against appropriate performance metrics and optimizing for maximum efficiency. Integration & Deployment: Partner with software engineers to integrate AI/ML models seamlessly into enterprise applications and systems. Address deployment challenges to ensure smooth production workflows. Continuous Improvement: Monitor the performance of deployed AI models, gather user feedback, and iterate to improve models, algorithms, and user experience. Proof of Concepts (PoCs): Develop PoCs to demonstrate the feasibility and effectiveness of AI/ML solutions, showcasing the potential of new technologies to clients. Collaboration: Work closely with development teams to explore and implement new use cases for AI and automation. Best Practices: Collaborate with your team to establish best practices and ensure high-quality development standards. Innovation: Contribute to the innovation of cloud-based enterprise automation solutions and the development of cutting-edge AI technologies. Key Requirements: Experience: Minimum 3 years of hands-on experience in implementing AI/ML models in enterprise environments. AI/ML Knowledge: Strong understanding of supervised and unsupervised learning, reinforcement learning, deep learning, and probabilistic models. Programming Skills: Proficiency in Python or R, with experience in SQL for querying databases. Data Handling: Expertise in working with large datasets, performing data preprocessing, and effectively managing data pipelines. Cloud Infrastructure: Experience with AWS Sagemaker or Azure ML is preferred for deploying machine learning solutions. Frameworks & Libraries: Familiarity with machine learning libraries like scikit-learn, Keras, TensorFlow, PyTorch, or NLTK is a plus. Analytical Skills: Strong critical thinking and problem-solving abilities, with the capacity to design and run experiments to validate hypotheses. Business Process Understanding: Knowledge of business processes and how they can be automated through AI and ML. Domain Expertise: Experience or interest in areas like Demand Forecasting, Anomaly Detection, Pricing, Recommendations, or Analytics. Global Experience: Proven experience in working with global customers on diverse projects, understanding their needs, and delivering tailored solutions. Communication Skills: Excellent verbal and written communication skills, with the ability to clearly explain complex AI concepts to clients and team members. Collaboration & Independence: Ability to work both independently and as part of a team, collaborating with cross-functional teams to drive success. Educational Background: A Bachelor's or Master's degree in Computer Science, Mathematics, Statistics, or a related field. Specialization: Coursework or expertise in AI, ML, Deep Learning, NLP/NLU, Computer Vision, or similar domains is highly desirable.
Senior Python Developer
Gramener
What Gramener Offers You Gramener will offer you an inviting workplace, talented colleagues from diverse backgrounds, career paths, and steady growth prospects with ample opportunities to innovate. Our goal is to create an ecosystem of easily configurable data applications focused on storytelling for public and private use. Role Description We are seeking a Backend Developer with 5+ years of experience in application development using Python and related backend technologies. The ideal candidate will be responsible for interpreting business requirements and translating them into clear, robust, and scalable technology solutions. Key Responsibilities Develop and maintain backend solutions using Python Tornado and FastAPI frameworks. Design, implement, and maintain RESTful APIs, ensuring secure token authentication and data compression. Work with various RDBMS models in the cloud and integrate different databases such as Clickhouse, MS SQL, Postgres, and Snowflake. Collaborate with cross-functional teams in an Agile environment to deliver high-quality solutions. Debug and troubleshoot issues by analyzing log files and error messages to identify and resolve root causes. Utilize version control and issue tracking systems such as Bitbucket, Git CLI, and Jira to manage projects efficiently. Follow best practices for code development, ensuring scalability, maintainability, and performance. Preferred Skills and Experience Strong understanding of RDBMS database models in the cloud environment. Basic knowledge of cloud components like Databricks, ADLS, and OAuth/Secrets is a plus. Proven experience with Python Tornado and FastAPI frameworks. Familiarity with token authentication mechanisms and secure API development. Good understanding of Agile methodologies and working in an Agile environment at scale. Excellent problem-solving skills, with the ability to analyze complex systems and find innovative solutions. Strong communication and teamwork skills, with a collaborative spirit. About Us At Gramener, we consult and deliver solutions that empower organizations to make data-driven decisions. We help our clients convert data into a strategic differentiator through strategic data consulting, advanced analytics, and visualization solutions. Our mission is to analyze and visualize large amounts of data, enabling organizations to gain valuable insights and achieve their business goals.
1 - 20 of 339 Data Pipelines in Hyderabad jobs
* No exact matches found. Showing closest results insteadNo results found
Modify search criteria or create an alert to get relevant jobs as soon as they’re posted
1 - 20 of 339