Spark Jobs in Nagpur

Senior Data Engineer AWS/Azure (5 8 Years) Nagpur, Pune, Bangalore Full-Time Job Type: Full-Time Location: Nagpur / Pune / Bangalore Experience Required: 5 to 8 years Work Shift: Rotational Open Positions: 4 Job Overview: We are hiring experienced Data Engineers with expertise in AWS and Azure cloud platforms for our offices in Nagpur, Pune, and Bangalore. Ideal candidates will be responsible for building and optimizing data pipelines, managing large-scale data solutions, and collaborating with cross-functional teams. If you are passionate about cloud data architecture, real-time processing, and analytics, this role is for you. Key Responsibilities: For AWS Data Engineer: Lead data engineering initiatives and mentor junior developers. Collaborate with product, application, and support teams for seamless data integration. Develop scalable data infrastructure using AWS services: S3, Glue, Lambda, SNS, SQS, Redshift, and RDS. Write optimized SQL queries for large datasets and handle complex data transformations. Build and manage ETL pipelines using Spark, PySpark, and Hadoop ecosystem. For Azure Data Engineer: Build, test, and maintain optimal data pipelines and data lake architectures on Microsoft Azure. Assemble large, complex data sets to meet functional and non-functional business requirements. Improve data reliability and pipeline performance by automating manual processes. Collaborate with analytics and data science teams to deliver meaningful insights and BI solutions. Work with Azure Data Factory, Spark, Kafka, Hive, Sqoop, MapReduce, and Spark Streaming. Required Skills: AWS Data Engineer: Strong hands-on experience with AWS cloud services: S3, Glue, SNS, SQS, Lambda, Redshift, RDS. Proficient in Python programming and SQL query optimization. Familiar with Spark, PySpark, Hadoop, Hive, and Spark-SQL. Azure Data Engineer: Hands-on experience with Azure data services and data architecture. Skilled in Big Data tools: Spark, Hive, Sqoop, Kafka, MapReduce. Experience with stream processing frameworks: Spark Streaming, Storm, etc. Join our cloud data engineering team and help shape intelligent, scalable data solutions! Apply now for the Data Engineer (AWS/Azure) position in Nagpur, Pune, or Bangalore.

Data Engineer Big Data Job Location: Nagpur Experience Required: 5 to 7 years Positions Open: 3 Shift: Rotational Shift Job Overview: We are looking for a talented Data Engineer with expertise in Big Data technologies to join our growing team. As a Data Engineer, you will be responsible for expanding and optimizing our data pipeline architecture. You will work closely with software developers, database architects, data analysts, and data scientists to ensure the seamless flow of data across various systems. This role requires proficiency in Hadoop, Hive, Spark, and other big data technologies, as well as solid experience in data transformation and ETL processes. If you are passionate about building and optimizing data systems, this is the role for you! Key Responsibilities: Data Pipeline Architecture: Expand and optimize the architecture for data flow and collection across various teams. Big Data Technologies: Work with Hadoop ecosystem tools such as Hive, Sqoop, and Spark to process large datasets. SQL Optimization: Craft and optimize complex SQL queries for data transformation, aggregation, and analysis. Python Programming: Use Python (including libraries like Pandas and NumPy) to process and manipulate data efficiently. ETL Processes: Ensure smooth data movement and transformation by applying industry best practices in ETL. Data Warehousing: Apply knowledge of data warehousing concepts to optimize data storage and retrieval. Data Quality Management: Troubleshoot data-related issues, ensuring the quality, consistency, and integrity of data. Collaboration: Work closely with cross-functional teams, translating complex technical concepts to non-technical stakeholders. Cloud Integration: Use cloud platforms (e.g., AWS, Azure, GCP) to integrate data systems and services. Automation: Write and maintain shell scripts or other automation tools for efficient data processing. Documentation: Create clear, concise technical documentation to facilitate knowledge sharing and communication across teams. Required Skills and Qualifications: 5 to 7 years of experience in Data Engineering, with a focus on Big Data technologies. Proficiency in SQL for data manipulation and analysis. Strong programming skills in Python (Pandas, NumPy) for data processing and manipulation. Hands-on experience with big data tools such as Hadoop, Hive, Sqoop, and Spark. In-depth understanding of ETL processes and data transformation methodologies. Experience in designing data models and optimizing data storage in data warehouses. Knowledge of cloud platforms (e.g., AWS, Azure, GCP) and their data engineering services is a plus. Ability to write shell scripts or other automation scripts for data processing. Ability to troubleshoot data-related issues and ensure data quality and consistency. Strong collaboration skills in a cross-functional team environment. Ability to communicate complex technical concepts to non-technical stakeholders. Preferred Skills (Good to Have): Experience with Data Lake architecture and integration. Familiarity with Stream Processing and related technologies. Knowledge of Cloud platforms (AWS, Azure, GCP) and their data services. Experience with Big Data frameworks and data modeling.