Pyspark Jobs in Pune

Job Title: Senior Azure Data Engineer Experience: 8 - 10 years Location: Kharadi, Pune Role & Responsibilities Design, develop, and maintain scalable, high-performance data pipelines using Azure Data Services such as Azure Data Factory, Azure Databricks, and Azure Synapse Analytics. Build and optimize data models, ETL/ELT processes, and data integration workflows to support analytics and BI needs. Develop and manage data lakes, data warehouses, and real-time streaming solutions on Azure. Collaborate with data analysts, scientists, and software engineers to ensure smooth data integration and processing. Ensure data security, governance, and compliance following industry best practices and regulations. Monitor and troubleshoot data pipelines, focusing on performance and reliability enhancements. Automate data workflows and infrastructure using DevOps and CI/CD methodologies. Work closely with stakeholders to translate business requirements into robust data solutions. Required Skills & Qualifications 8-10 years of experience in data engineering, with at least 3 years focused on Azure data services. Strong expertise in Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, Azure Data Lake, and Azure SQL Database. Proficient in SQL, Python, or Scala for data processing and transformation. Experience with Big Data technologies like Spark, Delta Lake, and Parquet. Hands-on experience with ETL/ELT development and orchestration of data pipelines. Knowledge of Azure DevOps, CI/CD pipelines, and Infrastructure as Code tools (Terraform, ARM templates, Bicep). Familiarity with data governance, security best practices, and compliance frameworks. Experience with Power BI, Snowflake, or other BI/analytics tools is a plus.

About Talentica Software: Talentica Software is a boutique software development company founded by industry veterans and alumni from IIT Bombay. We specialize in helping startups build cutting-edge products and thrive on leveraging the latest technologies to solve real-world problems. With over 21 years of experience, we have worked with 180+ startups, primarily in the US, leading to many successful exits. In 2022, Great Place to Work recognized Talentica Software as one of India s Great Mid-Size Workplaces. What We re Looking For: We re looking for a Big Data Developer who is passionate about building intelligent, scalable solutions and eager to work with the latest technologies. If you're ready to drive the future of tech and collaborate on exciting projects with top-tier developers, we d love to have you join us. What You ll Be Doing: As a Big Data Developer, you will: Develop intelligent, scalable engineering solutions from scratch. Collaborate with customers to understand product vision and goals. Contribute to high-level and low-level product designs and roadmaps alongside a team of talented developers. Build products using cutting-edge technologies such as Python (Open-source tech stack), Kafka, Storm, Spark/Flask, Hadoop, Cassandra, MongoDB, and more. To Be Successful in This Role, You Should Have: Qualification: BE/BTech in any branch from IIT, NIT, BITS, VJTI, COEP, or other top institutes in India. Experience: 4-6 years of experience working with Python and Spark/Flask. Technical Skills: Strong knowledge of NoSQL databases (Cassandra, Aerospike, MongoDB). Experience with REST API frameworks (a plus). What You ll Find at Talentica: A Culture of Innovation: We focus on creating cutting-edge solutions, not maintaining old projects. Our clients come to us for our technological expertise. Endless Learning Opportunities: Continuously expand your skills and stay at the forefront of the latest advancements, building better, faster, and simpler products. Talented Peers: Work with talented professionals from India s top engineering colleges (IITs, NITs, and others). Work-Life Balance: We understand the importance of well-being and offer flexible schedules, including remote work options, to help you thrive both professionally and personally. A Great Culture: Our employees love working here! 82% of our team recommends Talentica to their friends (according to Glassdoor). At Talentica, we invite you to be part of a dynamic team that pushes boundaries and doesn t just follow trends. If you re ready to shape the future of the tech industry with us, we d love to hear from you. Qualification : BE/BTech in any branch from IIT, NIT, BITS, VJTI, COEP, or other top institutes in India.

About Talentica Software: Talentica Software is a boutique software development company founded by industry veterans and IIT Bombay alumni. We specialize in helping startups build innovative products by leveraging cutting-edge tools and technologies to solve real-world challenges. With over 21 years of experience, we ve worked with over 180 startups, primarily in the US, leading to numerous successful exits. In 2022, Talentica Software was recognized by Great Place to Work as one of India s Great Mid-Size Workplaces. What We re Looking For: We are seeking a Senior Python Big Data Developer to join our team. If you're passionate about building scalable solutions from scratch, enjoy collaborating with clients, and love working with the latest technologies, this is the perfect opportunity for you. We re looking for proactive problem-solvers who are eager to contribute to innovative projects and deliver impactful solutions. What You ll Be Doing: As a Senior Python Big Data Developer, your responsibilities will include: Developing Scalable Solutions: Design and implement intelligent, scalable engineering solutions from the ground up. Client Collaboration: Work closely with clients to understand their product vision, goals, and technical requirements. Product Design: Contribute to both high-level and low-level product designs while collaborating with a team of skilled developers. Innovative Product Development: Utilize cutting-edge open-source Python technologies such as Kafka, Storm, Spark/Flask, Hadoop, Cassandra, MongoDB, and more to build innovative products. To Be Successful in This Role, You Should Have: Qualifications: BE/BTech from top engineering institutes (IIT, NIT, BITS, VJTI, COEP, or other top 100 engineering colleges in India). Experience: 3.6-6 years of experience in Python and Spark/Flask. Tech Skills: Strong expertise in NoSQL databases (e.g., Cassandra, Aerospike, MongoDB). Familiarity with REST API frameworks is a plus. What You ll Find Here: A Culture of Innovation: We focus exclusively on cutting-edge development, working on projects that push the boundaries of technology. Our clients come to us for innovative solutions, not maintenance work. Endless Learning Opportunities: At Talentica, you ll continuously expand your skill set by exploring and applying the latest advancements in technology. Talented Peers: Work alongside top-tier engineers from India's premier institutions (IITs, NITs, and others), offering numerous opportunities for collaboration and learning. Work-Life Balance: We prioritize your well-being with flexible schedules and remote work options, helping you maintain a healthy work-life balance. A Great Culture: Our employees love working here! 82% of our team members recommend Talentica to their friends, according to Glassdoor. If you re eager to be part of a dynamic team that doesn t just follow trends but leads and innovates, Talentica is the place for you. Join us and help shape the future of the industry by working on impactful, high-tech projects. Qualification : BE/BTech from IIT, NIT, BITS, VJTI, COEP, or other top 100 engineering institutes in India.

Data Engineering Lead BI Analytics & DWH Location: Pune About Calfus At Calfus, we build groundbreaking AI agents and enterprise software that are redefining what s possible for businesses. Whether it's automating workflows, integrating ERP systems, or deploying AI-powered solutions, we help companies scale smarter and faster. Our engineering and data teams are at the core of this innovation designing and delivering high-performance solutions that unlock massive business value. As we grow rapidly, we re looking for driven individuals ready to make a meaningful impact. About the Role We re hiring a Data Engineering Lead to own and drive our BI Analytics & Data Warehousing strategy. You'll lead the architecture, development, and optimization of data pipelines, models, and interactive dashboards that power strategic insights across the business. You ll work hands-on with ETL tools, SQL, Power BI, Tableau, Python, and cloud platforms like Azure and AWS while mentoring junior engineers and collaborating with cross-functional stakeholders. What You ll Do Data Architecture & Modeling Design and implement scalable data models that support self-service BI and analytical reporting. Manage data modeling across structured, semi-structured, and cloud-native data sources. ETL/ELT & Integration Oversee ETL processes using SSIS, Airflow, or equivalent tools for seamless data movement. Handle complex data transformation using SQL Server, Postgres, Snowflake, and Redshift. Visualization & Reporting Develop and lead dashboard/report design using Power BI, Tableau, QuickSight, Plotly, or Dash. Drive best practices in dashboard performance, UX, and storytelling with data. Advanced Data Engineering Use Python, PySpark, NumPy, and Pandas for data wrangling and exploratory analysis. Work with Azure Databricks, MongoDB, and cloud storage (S3, Azure Blob) to build robust pipelines. Automation & DevOps Orchestrate pipelines using Apache Airflow, and implement CI/CD for data engineering workflows. Manage version control and code quality across projects using Git-based workflows. Stakeholder Collaboration Partner with business leaders and analysts to translate requirements into scalable data solutions. Align technical delivery with strategic business goals and drive cross-functional data initiatives. Leadership & Mentorship Guide and mentor junior team members across data and BI functions. Foster a culture of innovation, ownership, and continuous learning. What You Bring Bachelor s in Computer Science, Information Systems, Data Engineering, or related field. 6 12 years of experience in BI architecture, data engineering, and analytics. Deep expertise in: ETL tools: SSIS, Airflow Databases: SQL Server, Snowflake, Postgres, Redshift, MongoDB BI Tools: Power BI, Tableau, QuickSight, Plotly/Dash Python for data analysis, automation, and pipeline development Cloud Platforms: Azure, AWS (S3, Lambda, Databricks) Strong SQL and data modeling skills (relational and dimensional). Familiarity with CRISP-DM, data governance practices, and performance tuning. Bonus Points If You Have Experience working with Azure SDK. Ability to work with REST APIs and perform web scraping. BI architecture design and deployment at scale. Growth-driven culture with clear career paths. Work on industry-defining AI and enterprise products. Exposure to diverse clients, industries, and technologies. Strong focus on wellness, flexibility, and learning. Benefits Medical, group, and parental insurance Provident fund & gratuity Birthday leave & employee wellness programs Highly collaborative and innovative work environment Diversity & Inclusion Calfus is an Equal Opportunity Employer. We believe that diversity fuels innovation. We re committed to creating a welcoming and inclusive workplace for everyone regardless of race, gender, age, background, or identity. Lead the future of data at Calfus. Apply now and help power decision-making through scalable, smart, and stunning data engineering solutions. Qualification : Bachelors in Computer Science Information Systems, Data Engineering, or related field

About Talentica Software: Talentica Software is a boutique software development company founded by industry veterans and alumni from IITB. For over 21 years, we have been helping startups build products using cutting-edge technologies. With a focus on solving real-world problems, we ve worked with over 180+ startups, primarily in the US, leading to numerous successful exits. In 2022, Great Place to Work recognized Talentica Software as one of India s Great Mid-Size Workplaces. What We re Looking For: We are seeking a Data Engineer to join the product engineering team for a SAAS-based Subscription Management Platform. If you have a passion for working with data and a drive to innovate, we encourage you to apply. What You ll Be Doing: Data Transformation: Use Python and PySpark to transform large datasets and build efficient data pipelines. Pipeline Creation: Work with Kafka and Debezium to create and deploy real-time data pipelines. SQL Expertise: Write and optimize complex SQL queries for data aggregation and transformation. Delta Lake & Databricks: Utilize Delta Lake and Databricks for efficient data storage and management. Collaboration: Work with cross-functional teams to ensure data is accessible, reliable, and well-organized for product teams. To Be Successful in This Role, You Should Have: Qualification: BE/BTech from a top-tier engineering institute. Experience: 5+ years of experience in the data engineering field. Must-Have Skills: Strong proficiency in Python and PySpark for data transformation. Experience with Kafka and Debezium for data pipeline creation and deployment. Excellent SQL skills, especially for complex data aggregation and transformation. Familiarity with Delta Lake and Databricks. A solid understanding of all stages of the SDLC and experience with Agile/SCRUM methodology. Good-to-Have Skills: Understanding of machine learning algorithms. Familiarity with ETL tools like Matillion or Google Data Studio. Experience with creating BI reports using tools like Looker or Tableau. Familiarity with writing APIs to extract data from databases. What You ll Find Here: Culture of Innovation: We focus on technology expertise and innovation, not maintenance projects. Our customers come to us for cutting-edge solutions. Endless Learning Opportunities: Stay ahead of the curve by exploring new advancements in your field and applying them to build better, faster, and simpler products. Talented Peers: Work alongside experienced graduates from India's top 20 engineering colleges, including IITs, NITs, and other top-tier institutes. Flexibility: We value work-life balance and offer flexible schedules, including remote work options. Great Culture: 82% of our employees recommend Talentica to their friends, according to Glassdoor. You'll love being part of our team! At Talentica, we don t just follow trends we lead them. If you re looking for a dynamic, fast-paced environment with opportunities to innovate, grow, and shape the future of technology, Talentica is the place for you. Join us and make a real difference! Qualification : BE/BTech from a top-tier engineering institute.

Tech Lead Data Overview: As a Tech Lead Data, you will provide technical leadership to a team of experts, helping implement information-driven strategies and systems that support agricultural scientists in developing solutions for growers. You will manage end-to-end implementation of data environments beyond architectural design, ensuring that real-life data systems are developed and executed efficiently. You are a highly motivated problem solver with the ability to prioritize shifting workloads in a dynamic environment. You are an effective communicator and a confident leader passionate about making things happen within a fast-paced organization. Key Responsibilities: Lead the development of data and analytics capabilities within the R&D node of the enterprise data mesh, implementing architectural blueprints for various data platforms. Develop and implement an organizational data strategy aligned with business processes, including data model design, database development standards, and management of data warehouses and analytics systems. Identify and manage both internal and external data sources, creating a data management plan aligned with the organizational strategy. Collaborate with cross-functional teams, stakeholders, and vendors to ensure the smooth functioning of the enterprise data ecosystem. Guide the technical direction of data projects and initiatives, ensuring best practices in data engineering and analytics. Integrate technical functionality to ensure data accessibility, accuracy, and security. Perform continuous audits of data management systems and address performance issues, reporting breaches or vulnerabilities to stakeholders. Experience & Skills Required: 9+ years of experience in Big Data, data warehousing, data analytics, and/or Information Management projects. Proven experience building large-scale enterprise data architectures using both commercial and open-source Data Analytics technologies. Strong data modeling and architecture skills, including expertise in data warehousing, data normalization, and dimensional modeling (e.g., OLAP, data vault). Understanding of predictive modeling, Natural Language Processing (NLP), text analysis, and Machine Learning. Knowledge of security integration, including Kerberos authentication, SAML, data security, privacy techniques like data masking, and tokenization. Experience with DevOps engineering and Continuous Integration/Delivery tools. In-depth understanding of AWS Cloud solutions and integrating into legacy systems and COTS solutions. Familiarity with the following tools: Data Ingestion: Snaplogic, Glue, Lambda Storage: S3 Processing: Databricks DWH: Redshift, Databricks SQL Warehouse Databases: RDS Languages: Python (pyspark), SQL (sparksql), Java, R BI Tools: PowerBI, Dash, Spotfire, Qlik Catalog: Glue catalog, Unity Catalog Streaming: Kafka, EventHub AI/ML: MLOps, SageMaker Storage/Dataframe: S3, Deltalake, Iceberg, Parquet Qualifications: University bachelor s degree in Science, Technology, Engineering, or Mathematics (STEM). Fluency in English. Strong customer focus with excellent communication, teamwork, and negotiation skills. In-depth technical experience, ideally with AWS certification. Experience implementing data-first solutions. Experiences to be Gained: Continuous upskilling in the latest technologies, tools, and design practices. Exposure to R&D operations. Development of leadership and relationship-building skills with internal customers. Company Description: Syngenta Group is one of the world s leading sustainable agriculture innovation companies, with over 53,000 people in more than 100 countries. We strive to transform agriculture through tailor-made solutions for farmers, society, and the planet. We are committed to maintaining high standards of ethics, integrity, and creating an inclusive, discrimination-free workplace. Additional Information: Syngenta is an Equal Opportunity Employer. We do not discriminate based on race, color, religion, gender, national origin, age, sexual orientation, gender identity, marital status, veteran status, disability, or any other legally protected status. Qualification : University bachelors degree in Science, Technology, Engineering, or Mathematics (STEM).

In this role, you will join one of IBM Consulting's Client Innovation Centers (Delivery Centers), where we deliver deep technical expertise and industry insight to both public and private sector clients globally. Our centers provide locally based skills to drive the innovation and adoption of new technologies. A career in IBM Consulting offers long-term relationships and close collaboration with clients worldwide. You ll work alongside visionaries across multiple industries to improve hybrid cloud and AI journeys for some of the world s most innovative and valuable companies. Your ability to accelerate impact and foster meaningful change is enabled by IBM's robust technology platforms, including software and Red Hat. Curiosity and a constant quest for knowledge are key to success, allowing you to challenge the norm, think creatively, and produce groundbreaking solutions for our clients. This environment promotes long-term growth, with ample career development opportunities. Your Role and Responsibilities: As a Big Data Engineer, you will be responsible for the design, maintenance, evaluation, and testing of big data solutions. Your key responsibilities include: Designing, building, optimizing, and supporting data models and ETL processes based on client business requirements. Building, deploying, and managing data infrastructure that can support the needs of a rapidly growing, data-driven organization. Coordinating data access and security to ensure seamless data access for data scientists and analysts when needed. Developing data pipelines/workflows for Source to Target and implementing solutions to address client needs. Ensuring high performance, scalability, and reliability of big data solutions. Required Technical and Professional Expertise: 3-5 years of experience working with Big Data technologies (Hadoop, Spark, HBase, Hive). Proficient in Scala and Python for data engineering tasks, including writing Pyspark programs for data analysis. Good working experience with Python to develop a custom framework for rule generation (similar to a rules engine). Developed Python code to gather data from HBase and implemented solutions using Pyspark. Strong knowledge of Apache Spark, including working with DataFrames and RDDs for business transformations. Experience using Hive Context objects for read/write operations in Hive. Preferred Technical and Professional Expertise: Understanding of DevOps principles and practices. Experience in building scalable end-to-end data ingestion and processing solutions. Familiarity with AWS services (e.g., S3, Athena, DynamoDB, Lambda, Jenkins). Proficiency in object-oriented and/or functional programming languages such as Python, Java, and Scala. As a Big Data Engineer at IBM Consulting, you will have the opportunity to work with cutting-edge technologies, influence industry transformations, and drive meaningful change for a variety of clients. IBM fosters a culture of continuous learning and career development, giving you the resources and support to grow in your role and within the company. This role offers the chance to work on complex, high-impact projects and collaborate with a global team of industry leaders.