Spark SQL Jobs in Bengaluru

Job Title: Sr. Data Science Engineer Location: Bangalore Experience: 5 8+ Years Job Description As a Senior Data Science Engineer, you will develop and maintain scalable data pipelines and manage API integrations to support growing data volume and complexity. You will play a key role in ensuring data quality and enabling accurate AI model development through effective data handling and automation. Responsibilities Monitor data quality and implement processes for data cleansing and validation. Analyze data to troubleshoot and resolve data-related issues promptly. Develop automation workflows for efficient data labeling, preparation, and augmentation to improve AI model accuracy and utility. Requirements Proven experience as a Data Scientist, Data Analyst, or related role, with experience in data mining. Strong proficiency in data manipulation using Python or R; familiarity with Scala, Java, or C++ for raw data processing is a plus. Experience with business intelligence tools (e.g., Tableau) and big data frameworks like Hadoop and Spark. Strong mathematical foundation, especially in statistics and algebra. Advanced SQL skills with experience in database development, data migration, and integration. Expertise in exploratory data analysis and familiarity with common data science toolkits. Ability to communicate complex data insights clearly and effectively to non-technical stakeholders. Familiarity with data management tools and experience applying machine learning and AI techniques.

Data Engineer Location: Bangalore Company: Capital One India About Capital One At Capital One, we're redefining how technology solves real-world financial challenges. As a technology-driven company, we bring together talented engineers, data scientists, and designers to innovate at scale and deliver meaningful impact to millions of customers. If you're passionate about building powerful data solutions, exploring cutting-edge technologies, and working in a collaborative, fast-paced environment this is the place for you. About the Role As a Data Engineer at Capital One, you ll join a team of innovators who design and build next-generation data platforms and pipelines that power real-time decision-making. You ll collaborate across disciplines engineering, product, machine learning, and cloud infrastructure to transform how we leverage data at scale. What You ll Do Collaborate across Agile teams to design, develop, test, and deploy data-driven solutions. Build and support scalable data pipelines using modern data engineering tools and cloud services. Work on real-time and batch data processing systems that integrate with distributed microservices and ML platforms. Use programming languages such as Python, Java, or Scala with SQL, NoSQL, and cloud data warehouses like Redshift or Snowflake. Contribute to code reviews, unit testing, and performance optimization to ensure high-quality data systems. Partner with product managers and platform teams to deliver robust, cloud-native data solutions that power business decisions. Stay ahead of tech trends, share knowledge, and mentor junior engineers. Basic Qualifications Bachelor s degree in Computer Science, Engineering, or a related field. 1.5+ years of hands-on experience in application or data engineering (excluding internships). At least 1 year of experience working with big data technologies. Preferred Qualifications 3+ years of application/data engineering experience using Python, Scala, Java, or SQL. 1+ year of experience with cloud platforms (AWS, Azure, or GCP). 2+ years of experience with distributed computing tools (Spark, Hadoop, Hive, EMR, Kafka, etc.). 1+ year working on real-time streaming applications. 1+ year of experience with NoSQL databases (MongoDB, Cassandra). 1+ year of experience with data warehousing (Redshift, Snowflake). 2+ years working with Linux/Unix systems and shell scripting. Familiarity with Agile methodologies and modern DevOps practices. Why Join Capital One Work on high-impact data solutions at one of the world s most innovative financial institutions. Be part of a collaborative tech culture that values experimentation and learning. Access to top-tier tools, mentorship, and career development opportunities. Competitive compensation and benefits in a mission-driven environment. Qualification : Bachelors degree in Computer Science, Engineering, or a related field

Position: Data Scientist (AI/ML Expert) Location: Pritech Park SEZ, Block 09, 4th Floor B Wing, Survey No. 51 to 64/4, Outer Ring Road, Bellandur V, Bangalore, Karnataka, India Department: Advanced Analytics Employment Type: Subexian Experience Required: 1 to 3 years Job Overview: We are looking for a talented Data Scientist with expertise in AI/ML to join our Advanced Analytics team. As a key contributor, you ll design, develop, and validate predictive models, recommendation systems, and forecasting solutions, while also collaborating with cross-functional teams to deliver cutting-edge solutions using the latest technologies. Key Responsibilities: Model Development: Design, develop, and validate predictive models, recommendation systems, and forecasting solutions using a mix of statistical, machine learning, and deep learning techniques. You will work both independently and as part of a collaborative team. Data Visualization & Reporting: Communicate actionable insights effectively through compelling dashboards, reports, and visualizations using tools such as Superset, Power BI, and Python libraries (Matplotlib, Seaborn, Plotly). AI & Tech Solutions: Collaborate with teams to design and deliver flexible, scalable solutions using advanced technologies such as AI and large language models (LLMs). API Development: Develop and integrate REST APIs and frameworks such as Flask or FastAPI for seamless deployment of machine learning models. Documentation: Maintain clear, comprehensive documentation for data workflows, model development, and analytical methodologies to ensure knowledge sharing and transparency across teams. Continuous Learning: Stay up-to-date with the latest trends and advancements in data science, algorithms, and technologies, ensuring your skills and knowledge remain cutting-edge. Required Technical Skills: Python Proficiency: Strong experience with Python and libraries like Scikit-learn, TensorFlow/PyTorch, and data visualization libraries (Matplotlib, Seaborn, Plotly). SQL: Solid hands-on experience in SQL for efficient data querying. ML Ops & Pipelines: Understanding of machine learning operations (ML Ops) and ML pipelines for streamlined model deployment. Cloud & Distributed Computing: Exposure to cloud platforms such as AWS, Azure, or GCP and distributed computing tools like Hadoop, Spark, or Pyspark is a plus. Soft Skills: Effective Communication: Strong ability to communicate complex analytical findings in a clear and engaging manner, tailoring insights for both technical and non-technical audiences. Problem-Solving: A proactive problem-solver with the ability to adapt and thrive in a fast-paced, dynamic environment. Continuous Growth: Self-motivated, curious, and always seeking opportunities for professional growth and learning. At Subex, we encourage a collaborative, innovative, and growth-driven work environment. If you're passionate about applying data science techniques to real-world challenges and want to work with cutting-edge AI/ML technologies, we d love to hear from you!

Senior Associate Data Engineering L2 Location: Bengaluru, India Department: Engineering Data Employment Type: Full-Time About the Role As a Senior Associate Data Engineering (L2) at Publicis Sapient, you will lead technical solutions that drive digital transformation by building scalable, high-performance data platforms. You ll be responsible for translating business and technical requirements into modern, data-centric solutions using Big Data technologies, cloud services (Azure), and advanced data engineering practices. Key Responsibilities Design and implement data ingestion, integration, and transformation processes from multiple heterogeneous sources in both batch and real-time. Build scalable data platforms using Hadoop stack components such as HDFS, Kafka, Spark, Hive, NiFi, Oozie, Airflow, Flink, and Storm. Develop real-time analytics, aggregation, and search features to support various data-driven applications. Collaborate closely with cross-functional teams on data infrastructure, computation frameworks, and data visualization. Apply cloud-native principles and Azure services to build and deploy data pipelines. Ensure performance optimization and data pipeline tuning. Work with NoSQL and MPP platforms like MongoDB, Cassandra, Redshift, Azure SQL DW, HBase, BigQuery. Contribute to infrastructure, automation, and DevOps for data pipelines using CI/CD practices. Ensure data governance, lineage, and cataloging using tools like Collibra or Alation. Required Qualifications 6 8 years of professional experience in software/data engineering. Minimum of 3 years hands-on experience with Big Data technologies. Strong programming expertise in Java (preferred), Scala, or Python. Expertise in the Hadoop ecosystem and real-time stream processing tools (Kafka, Pulsar, Spark Streaming, etc.). Hands-on experience with Azure data services (e.g., Data Factory, Synapse, ADLS, Databricks). Experience working with modern ETL tools (Informatica, Talend, etc.) and traditional RDBMS platforms (Oracle, PostgreSQL, SQL Server, MySQL). Bachelor's degree in Computer Science, Engineering, or related field. Nice to Have Certifications in Azure Data Engineer, GCP Big Data, or related cloud specializations. Experience with distributed messaging frameworks (ActiveMQ, RabbitMQ, Solace). Familiarity with microservices architecture and search technologies (Elasticsearch). Performance tuning of distributed data processing systems. Exposure to data governance, security, and metadata management. Benefits and Culture at Publicis Sapient Gender-neutral workplace policies 18 paid holidays annually Generous parental leave + new parent transition support Flexible work arrangements Access to Employee Assistance Programs (wellness & well-being) A dynamic culture focused on learning, creativity, and collaboration Qualification : Bachelor's degree in Computer Science, Engineering, or related field.

Job Title: Lead Data Scientist Location: Bangalore North, Karnataka, India Job Type: Full-Time Industry: Entertainment / Mobile Gaming About Us We are one of India s most exciting and fast-growing mobile gaming companies. Founded in 2014 and partnered with Modern Times Group (MTG), our vision is to create simple, impactful casual game experiences at massive scale. Our portfolio includes evergreen hits such as Daily Themed Crossword, WordTrip, WordJam, WordWars, WordTrek, TileMatch, and Jigsaw. We have built a global network of chart-topping games supported by powerful tech and analytics infrastructure that fuels rapid growth. Position Summary As a Lead Data Scientist in our Central Analytics team, you will play a critical role in shaping data-driven strategies that enhance player experience and business performance. This fast-paced role offers abundant opportunities to work alongside product leaders and game teams, transforming complex data into actionable insights that drive user acquisition, engagement, and monetization. Key Responsibilities Collaborate closely with product leaders to provide data-driven advisory on strategic decisions. Partner with game development teams to analyze gameplay data and generate actionable insights that improve user acquisition, engagement, and monetization. Perform advanced exploratory data analyses and ad-hoc reporting to identify trends, issues, and opportunities across our game portfolio. Design, execute, and lead data research projects, delivering practical recommendations based on rigorous statistical analyses. Drive continuous improvement in game performance through innovative machine learning models and analytics techniques. Requirements Bachelor s/Master s/PhD degree in Computer Science, Statistics, or a related field. Proven experience with machine learning, statistical modeling, and data science projects. Hands-on proficiency in Python and/or Spark for data manipulation, visualization, and building ML models. Strong SQL skills with experience querying large, complex datasets from data lakes or warehouses. Demonstrated ability to lead research projects and translate findings into actionable business recommendations. Excellent interpersonal skills and a collaborative approach to working with cross-functional teams. Knowledge of Deep Learning frameworks and techniques is highly desirable. Work with a top-tier gaming company known for its innovative and data-driven culture. Influence millions of users worldwide through impactful analytics. Collaborate with talented teams in a high-growth, dynamic environment. Access to cutting-edge tools and technologies for data science and machine learning. Competitive compensation and career growth opportunities. Qualification : Bachelors/Masters/PhD degree in Computer Science, Statistics, or a related field.

Job Title: Senior Manager Data Science, Data Modelling & Analytics Location: Bengaluru Department: Insights & Analysis About the Role: As a Senior Manager, you will lead a team of data scientists and analysts, driving the development and deployment of advanced analytics solutions that enable data-driven decision-making. This role blends strategic leadership with hands-on technical expertise, playing a critical part in delivering impactful insights and analytics across the organization. Key Responsibilities: Hands-On Technical Contribution: Design, develop, and deploy advanced machine learning models and statistical analyses to address complex business challenges. Utilize Python, R, SQL, and other tools to manipulate data and build predictive models. Manage end-to-end data pipelines including collection, cleaning, transformation, and visualization. Collaborate with IT and data engineering teams to integrate analytics solutions into production environments. Provide thought leadership on analytics solutions and metrics aligned with business needs. Team Leadership & Development: Lead, mentor, and manage a team of data scientists and analysts, fostering collaboration and innovation. Guide career development, conduct performance evaluations, and promote skill enhancement. Encourage continuous learning and adoption of best practices in data science methodologies. Strategic Planning & Execution: Collaborate with senior leadership to define and execute data science strategy aligned with business goals. Identify and prioritize high-impact analytics projects that deliver business value. Ensure timely and quality delivery of analytics solutions balancing scope and resources. Client Engagement & Stakeholder Management: Act as primary point of contact for clients, translating business challenges into data science solutions. Lead client presentations, workshops, and discussions, effectively communicating complex analytical concepts. Build and maintain strong client relationships, managing expectations and deliverables. Deliver regular reports and dashboards to senior management and stakeholders. Bridge communication between technical teams and business units to align analytics initiatives with organizational objectives. Cross-Functional Collaboration: Work closely with Business Intelligence, Market Analytics, and Data Engineering teams to integrate analytics into business processes. Translate complex insights into actionable recommendations for non-technical stakeholders. Facilitate data-driven workshops and presentations across the organization. Collaborate with support functions to provide timely leadership updates on operational metrics. Governance & Compliance: Ensure compliance with data governance policies and data privacy regulations (e.g., GDPR, PDPA). Implement best practices for data quality, security, and ethical analytics use. Stay abreast of industry trends and regulatory changes affecting data analytics. Qualifications: Education: Bachelor s or Master s degree in Data Science, Computer Science, Statistics, Mathematics, or related field. Experience: 12+ years in advanced analytics, data science, data modelling, machine learning, or related fields. 5+ years in leadership roles managing analytics teams and projects. Experience in BFSI, Hi-Tech, Retail, or Healthcare industries preferred. Experience with media data is a plus. Technical Skills: Proficiency in Python, R, SQL. Experience with data visualization tools like Tableau, Power BI. Familiarity with big data platforms (Hadoop, Spark) and cloud services (AWS, GCP, Azure). Strong knowledge of machine learning frameworks and libraries. Soft Skills: Excellent analytical and problem-solving skills. Strong communication and interpersonal abilities. Ability to influence and drive organizational change. Strategic thinker focused on business outcomes. Desirable Expertise: Advanced Analytics Techniques: Descriptive Analytics: Statistical analysis, data visualization. Predictive Analytics: Regression, time series forecasting, classification, market mix modelling. Prescriptive Analytics: Optimization, simulation modelling. Text Analytics: NLP, sentiment analysis. Machine Learning Techniques: Supervised Learning: Linear/logistic regression, decision trees, random forests, gradient boosting, SVMs. Unsupervised Learning: Clustering, PCA, anomaly detection. Reinforcement Learning: Q-learning, deep Q-networks. Generative AI & Large Language Models (Good to Have): Experience with GPT, Gemini, LLAMA, etc. for text generation, summarization, conversational agents. Hyperparameter tuning, prompt engineering, embeddings, fine-tuning. Additional Skills: Proficiency with Tableau or Power BI (advanced visualization). Strong data management, structuring, and harmonization skills.

Position Title: Lead AI/ML Engineer Location: Bengaluru Bellandur (GTP) Employment Type: Full-time Job Summary Synechron is seeking a seasoned Lead AI/ML Engineer to lead cutting-edge initiatives in artificial intelligence and machine learning. This role requires deep technical expertise in AI/ML, including deep learning, NLP, computer vision, and generative AI, combined with strong leadership skills to manage teams and drive innovation. You will collaborate closely with stakeholders to deliver high-impact solutions and play a key role in shaping the company s AI-driven digital transformation. Key Responsibilities Lead end-to-end development of AI/ML solutions across use cases involving machine learning, deep learning, computer vision, NLP, and generative AI. Design and implement scalable models and algorithms, ensuring performance, accuracy, and interpretability. Collaborate with product owners, engineers, and business stakeholders to align AI/ML initiatives with strategic objectives. Mentor and guide data scientists and ML engineers to elevate technical quality and delivery standards. Continuously evaluate emerging AI technologies, frameworks, and methodologies to enhance team capabilities. Contribute to innovation strategy by identifying new business opportunities driven by AI/ML. Ensure data and model governance, ethical AI practices, and responsible deployment of AI solutions. Required Skills & Tools Core Competencies: Strong hands-on experience in machine learning, deep learning, NLP, computer vision, and generative AI. Expertise in Python (primary), with additional knowledge of R, SQL, Java, or C++. Deep understanding of statistics, linear algebra, probability, and algorithm design. Frameworks & Tools: TensorFlow, PyTorch, Keras, OpenCV, Hugging Face, spaCy, Scikit-learn, etc. Data processing tools like Pandas, NumPy, and Spark. Version control and CI/CD tools (e.g., Git, MLflow, Docker, Airflow). Methodologies: Proficient in Agile development and model lifecycle management. Experience with productionizing AI models and deploying them on cloud platforms (AWS, Azure, GCP). Experience 8 12 years of experience in AI/ML, Data Science, or related domains. Minimum 8+ years in a leadership or technical lead role driving successful AI/ML initiatives. Proven record of delivering high-impact AI/ML projects at scale. Experience mentoring and managing AI/ML teams in collaborative, cross-functional environments. Day-to-Day Activities Direct and oversee AI/ML model development, validation, deployment, and monitoring. Review project designs, provide technical oversight, and conduct code/model reviews. Drive research and experimentation to apply novel AI methods to solve real-world problems. Support proposal development and client engagements with AI/ML subject matter expertise. Foster a culture of continuous learning, innovation, and data-driven thinking. Qualifications Bachelor s or Master s degree in Computer Science, Data Science, Artificial Intelligence, or a related field. Relevant certifications (e.g., TensorFlow, AWS Machine Learning, Azure AI Engineer) are a plus. Soft Skills Strong leadership, team management, and mentoring capabilities. Excellent communication and stakeholder engagement skills. Strategic thinking with a passion for innovation and emerging technologies. Ability to thrive in a dynamic, fast-paced environment and lead through ambiguity. Diversity & Inclusion at Synechron Synechron is committed to a diverse and inclusive workplace. Through our Same Difference DEI initiative, we celebrate unique backgrounds and perspectives while fostering a respectful, empowering environment. We welcome applications from candidates of all identities and provide support through mentoring, internal mobility, and flexible work arrangements. Qualification : Bachelors or Masters degree in Computer Science, Data Science, Artificial Intelligence, or a related field.

Data Architect Bangalore, India Location: Bangalore (Bengaluru) Experience: 10 to 15 Years Industry: IT & Data Systems Job Summary: We are seeking an experienced Data Architect with a strong background in designing and implementing enterprise-scale data solutions. The ideal candidate will have expertise in building data lakes, warehouses, and pipelines, with deep knowledge of cloud platforms, data management, and industry best practices. Key Responsibilities: Design, develop, and maintain complex data architectures including data lakes, data warehouses, data marts, and efficient schema design Build and optimize scalable data pipelines for extraction, transformation, and loading (ETL/ELT) processes Apply Agile methodologies in project delivery and collaborate within cross-functional teams Perform data profiling, cleansing, conversion, and ensure high-quality data management for both structured and unstructured data Implement CI/CD and Infrastructure as Code (IaC) practices using tools like GitHub, Jenkins, CloudFormation, and Azure Resource Manager Manage database systems and tools such as PostgreSQL, Oracle, Snowflake, Teradata, MongoDB, Hadoop, and others Utilize data modeling tools like Erwin, Power Designer, and Toad for effective data architecture design Leverage cloud platforms including AWS and Microsoft Azure, with hands-on experience in services like AWS Glue, DMS, Lambda, Azure Data Factory, Synapse, and Data Lake Storage Work with programming and scripting languages including SQL, PL/SQL, Python, Spark, YAML, and JSON Use containerization and automation tools such as Docker, Ansible, and NodeJS for efficient deployment Ensure compliance with cybersecurity principles and frameworks such as NIST Lead data governance initiatives and enforce best practices in data quality and security Preferred Qualifications: ITIL certification and experience with Agile methodology Knowledge of code review and version control best practices, especially in GitHub Familiarity with data science tools and AI/ML frameworks like R, Keras, or TensorFlow Experience with natural language processing (NLP) and machine learning concepts Background in regulated industries, with pharma manufacturing experience highly preferred Exposure to multi-site, global IT projects and manufacturing operations Lead innovative data architecture projects within a dynamic and fast-paced environment Work with cutting-edge cloud technologies and big data ecosystems Collaborate with global teams on impactful enterprise solutions Access to professional growth opportunities in data governance, AI, and cloud technologies

Role: Senior Analyst Data Engineering Location: Bengaluru, Karnataka, India Experience: 3 6 Years Employment Type: Permanent, Full-Time About the Role We are looking for a results-driven Senior Data Engineer to join our high-performing data team in Bengaluru. The ideal candidate will have 3 6 years of experience in data engineering, AI/ML implementation, and working with large-scale databases like Snowflake and Teradata. If you're passionate about driving data-powered insights, building scalable solutions, and applying advanced machine learning and AI techniques, we want to hear from you. Key Responsibilities Design, develop, and implement machine learning models to solve complex business challenges. Apply AI techniques, including generative AI, NLP, and computer vision, to improve analytics capabilities. Use Tableau, Power BI, and other tools to develop insightful, interactive data dashboards. Manage and optimize large datasets using platforms like Snowflake, Teradata, and SQL/NoSQL databases. Collaborate with business and technical teams to translate requirements into robust data engineering solutions. Guide junior data professionals and foster a culture of learning and innovation. Communicate analytical findings clearly to non-technical stakeholders. Stay current with the latest in data science, machine learning, cloud platforms, and big data technologies. Key Skills & Technologies Machine Learning & AI Techniques: Supervised & unsupervised learning, deep learning, neural networks Reinforcement learning, decision trees, random forests, clustering NLP, computer vision, GANs, transfer learning Data Visualization Tools: Tableau, Power BI, Matplotlib, Seaborn, Plotly Programming Languages & Libraries: Python (essential), R, SQL TensorFlow, PyTorch, scikit-learn, pandas, NumPy, Keras, SciPy Databases & Data Management: Snowflake, Teradata, SQL/NoSQL, ETL, data lakes, data warehousing Cloud Platforms: AWS, Azure, Google Cloud Platform (GCP) Big Data Technologies: Apache Spark, Hadoop

Data Scientist - I | Meesho Careers Location: Bangalore, Karnataka | Department: Tech Join Meesho as a Data Scientist and Shape the Future of E-commerce At Meesho, our Data Scientists work on high-impact projects such as fraud detection, inventory optimization, and platform vernacularization, building intelligent systems that enhance the experience of millions of users across thousands of product categories. What You ll Do Design and develop advanced data models and run experiments to derive insights from complex datasets Enhance product usability and identify new growth opportunities through data Analyze reseller behavior to recommend relevant products and increase engagement Design and optimize discount programs to boost reseller sales Identify end-customer preferences to help resellers improve revenue Use analytics to identify supply chain bottlenecks and help suppliers meet SLA requirements Forecast seasonal demand and contribute to strategic planning by modeling key metrics Mentor junior data scientists and help foster a collaborative learning environment Bachelor's or Master s degree in Computer Science, Data Science, or related fields 1 2 years of professional experience as a Data Scientist in a fast-paced, preferably B2C environment Hands-on experience with machine learning, neural networks, and related techniques Proficiency in tools like SQL, Python, R Strong statistical foundation, with expertise in hypothesis testing and model validation Proven ability to conduct and analyze A/B testing Experience working closely with product and tech teams Bonus Points If You Have: Experience in personalization or similar ML problem spaces Familiarity with Big Data technologies such as Apache Spark, Hadoop, or Redshift About Meesho Meesho is India's top-rated e-commerce platform on Glassdoor, known for empowering millions of entrepreneurs across the country. We provide sellers with zero-commission benefits and the lowest shipping costs, serving every serviceable pincode in India. With over 1.75 million registered sellers, Meesho is helping small businesses grow using a unique business model and world-class tech infrastructure. From first-time internet users to seasoned entrepreneurs, our inclusive platform caters to a diverse customer base across India. Our Mission Democratizing Internet Commerce for Everyone Meesho (short for Meri Shop) is dedicated to enabling 100 million small businesses to succeed online, offering them an affordable, reliable, and easy-to-use platform that mirrors local market preferences. Our Culture and Total Rewards We cultivate a performance-driven, people-first culture that values learning, creativity, and excellence. Our "Mantras" guide our daily decisions, from hiring to growth discussions. Meesho Total Rewards includes: Market-leading compensation, including equity-based benefits Holistic wellness through our MeeCare Program: medical insurance, telehealth, wellness events, gym discounts, and more Work-life balance support: generous leave policies, parental benefits, and retirement plans Learning & development assistance and career mobility programs Fun workplace culture with events, personalized gifts, and team engagement activities Join Meesho to work on meaningful data problems, impact millions, and grow your career with a supportive and innovative team. Qualification : Bachelor's or Masters degree in Computer Science, Data Science, or related fields

Lead Data Engineer | ThoughtWorks | Bangalore, India Location: Bangalore, India Employment Type: Full-time, Regular Industry: Information Technology About ThoughtWorks At ThoughtWorks, we're a global technology consultancy that integrates strategy, design, and engineering to drive digital innovation. For over 30 years, we've worked alongside our clients to deliver solutions that challenge the status status quo. With a diverse and inclusive team, we empower each other to grow through shared learning, fostering an environment where innovation thrives. Our commitment to a cultivation culture is key to our success, and we re looking for a Lead Data Engineer to join our Bangalore team and lead transformative projects. Job Overview As a Lead Data Engineer at ThoughtWorks, you will be responsible for designing, developing, and operating modern data architectures that meet client business objectives. You will lead and manage data engineering projects end-to-end, from strategic planning to hands-on coding, ensuring the delivery of scalable and efficient data solutions. Working with cutting-edge technologies, you ll collaborate with stakeholders, clients, and cross-functional teams to implement data-driven strategies that address complex business challenges. Key Responsibilities Project Leadership: Lead and manage data engineering projects from inception to completion, including goal-setting, scope definition, and ensuring on-time delivery in collaboration with cross-functional teams. Data Architecture & Solution Design: Collaborate with clients to design modern data architecture and implement end-to-end solutions that meet key business objectives. Create intricate data processing pipelines to address complex business problems. Stakeholder Collaboration: Work closely with stakeholders to understand business objectives and identify opportunities to leverage data and data quality improvements. Data Modeling & Governance: Develop data models using modern modeling techniques and implement them using appropriate technologies. Ensure compliance with data governance, security, and privacy requirements. Scalable Implementations: Partner with data scientists to design scalable implementations of their models, ensuring the solutions are robust and efficient. Clean, Iterative Code: Write clean, modular code based on TDD (Test-Driven Development) and implement continuous delivery practices to support and operate data pipelines. Technology Guidance: Advise clients on distributed storage and computing technologies, selecting the best options to fit their business needs. Data Quality Strategy: Define and incorporate data quality strategies into daily work processes to ensure high standards and compliance. Job Qualifications Technical Skills: Proven experience in data engineering and system design, with a focus on building Big Data architecture and data pipelines within distributed systems. Deep knowledge of data modeling and hands-on experience with modern data engineering tools and platforms. Strong programming skills, with expertise in building scalable, high-quality data pipelines using languages like Python, Java, Scala, or others. Experience with distributed storage platforms (e.g., Hadoop, Amazon S3, etc.) and distributed processing platforms (e.g., Spark, Flink). Experience working with SQL, NoSQL, data lakes, and other data storage technologies. Familiarity with data visualization techniques and ability to communicate insights effectively across varying audiences. Professional Skills: Stakeholder Management: Strong ability to liaise between clients and other key stakeholders, ensuring trust and buy-in throughout projects. Adaptability & Resilience: Comfortable handling ambiguity and finding innovative solutions to complex challenges. Leadership & Mentorship: Experienced in coaching and mentoring team members, fostering a culture of professional growth and accountability. Risk & Conflict Management: Skilled in managing risks and resolving conflicts, driving projects forward despite challenges. Relationship Building: Natural at cultivating strong relationships with clients, stakeholders, and internal teams to create new opportunities. What You Bring to the Team Leadership: A proven track record in leading high-performance teams and supporting colleagues in their professional development. Curiosity & Innovation: A passion for data and technology and a willingness to continually learn and push the boundaries of what's possible. Collaboration: Ability to work closely with cross-functional teams and stakeholders to design and implement innovative data solutions. At ThoughtWorks, we believe in giving you the autonomy to carve out your unique career path, while providing support through development programs and a vibrant culture of learning. You ll work on exciting projects with a diverse team, solve complex challenges, and make an impact at a global scale. Join ThoughtWorks and be part of a global community of innovators. Together, we turn curiosity into action and creativity into impactful solutions.

Data Engineer (Python/Dataiku/Spark/SAS DI) Location: Bangalore Type of Employment: Full-Time Experience Required: 3 to 5 years Job Overview: We are looking for a passionate Data Engineer with expertise in Python, Dataiku, Spark, or SAS DI to join our dynamic team. In this role, you will collaborate with clients to build data pipelines that support their Data and Analytics journey. As a member of our team, you will be responsible for driving data engineering initiatives, developing and implementing ETL solutions, and optimizing data workflows for efficiency and scalability. Key Responsibilities: Data Engineering Initiatives: Collaborate with clients to define and implement data engineering initiatives based on business needs. ETL Development: Design and develop data integration solutions using tools like Python, Dataiku, Spark, and SAS DI. Develop and maintain automated data pipelines for real-time and batch data ingestions. Performance Tuning: Conduct performance tuning and optimize queries and data models to enhance performance and scalability. Database Design: Design and implement database objects such as tables, views, schemas, and stored procedures. Ensure data quality and consistency across the data warehouse. Collaboration: Work closely with cross-functional teams to translate business requirements into technical solutions and communicate effectively with clients across multiple regions. Troubleshooting: Troubleshoot and resolve data-related issues while ensuring the quality and integrity of the data. Project & Technical Leadership: Act as the technical lead or single point of contact (SPOC) for the client and internal teams, ensuring smooth project delivery. Essential Skills: Data Engineering Tools: Experience with Python, Dataiku, Spark, and SAS DI. SQL Proficiency: Strong SQL experience for extracting data from various databases and performing data analysis. ETL Development: Expertise in designing and developing ETL processes to integrate data from diverse sources into a data mart layer. Data Modeling: Understanding of data modeling concepts, including relational and dimensional modeling. BI Knowledge: Basic understanding of Business Intelligence (BI) tools and analytics needs. Communication Skills: Excellent communication skills to understand business requirements and translate them into technical solutions. Desirable Skills: Data Analysis: Strong analytical skills to perform data analysis, especially for banking products data. BI Tools: Experience with MicroStrategy or Power BI for reporting and dashboarding. Qualifications: 3+ years of experience designing and developing scalable data integration solutions. Familiarity with data analytics and reporting needs. Technical certifications that align with your continuous learning aspirations. Qualities: Strong problem-solving skills with a systematic approach to resolving issues. Ability to influence and implement change confidently. Excellent team player who works effectively in cross-functional teams. Strong communication and presentation skills, capable of interacting with clients and presenting ideas persuasively. Fluent in English, both written and spoken, for effective communication. Opportunity to work with cutting-edge data engineering tools and technologies. Be a key player in building and optimizing data pipelines for clients data and analytics needs. Competitive salary, benefits, and a dynamic, innovative work culture.

Sr. Data Engineer - AWS - Big Data Location:Bangalore Type of Employment: Full-Time Experience Required: 7 to 10 years Job Overview: We are seeking a highly skilled Sr. Data Engineer with expertise in AWS cloud technologies and Big Data to join our Cloud Data Architect Team at Infocepts. In this critical role, you will design and implement robust data solutions using technologies like EMR, Athena, PySpark, AWS Lambda, S3, and other AWS services. The ideal candidate will have a strong foundation in database concepts and SQL and will be responsible for building scalable data pipelines to support high-performance data processing. Key Responsibilities: Technology Assessment and Design: Study the existing technology landscape and evaluate current data integration frameworks. Assist in designing complex Big Data use cases leveraging AWS services. Documentation and Stakeholder Communication: Prepare and maintain comprehensive project documentation, adhering to quality guidelines and schedules. Work closely with Architects and Project Managers to provide accurate estimations, scoping, and scheduling assistance. Clearly communicate design decisions and conduct Proof-of-Concepts to validate new solutions before implementation. Process Improvement and Automation: Identify areas for process automation to improve efficiency and team productivity. Provide expert guidance and troubleshooting support to junior Data Engineers. Training and Knowledge Sharing: Develop and deliver technology-focused training sessions for the team, ensuring continuous knowledge sharing. Share expertise through Expert Knowledge Sharing sessions with Client Stakeholders. Essential Skills: AWS Services Expertise: In-depth knowledge of S3, EC2, EMR, Athena, AWS Glue, and Lambda. Big Data Technologies: Proficiency with Apache Spark, Databricks, and Big Data table formats such as Delta Lake (open-source). Data Warehousing: Strong understanding of data warehousing concepts and architectures. Programming Skills: Advanced programming skills in Python for building data pipelines. SQL Expertise: Strong SQL skills for data transformation, aggregation, and querying large datasets. ETL Workflow Development: Expertise in creating ETL workflows with complex transformations (e.g., SCD, deduplication, aggregation). Orchestration Tools: Familiarity with orchestration tools like Apache Airflow. MPP Databases: Experience with at least one MPP database (e.g., AWS Redshift, Snowflake, SingleStore). Cloud Databases: Exposure to cloud databases like Snowflake or AWS Aurora. Desirable Skills: Cloud Databases: Familiarity with Snowflake, AWS Aurora. Big Data Technologies: Experience with Hadoop and Hive. AWS Certification: Associate or Professional Level AWS Certification. Advanced Knowledge of Big Data Solutions: Exposure to big data tools and frameworks on cloud platforms. Qualifications: Experience: 7+ years of overall IT experience, with 5+ years specifically focused on AWS-related projects. Educational Background: Bachelor's degree in Computer Science, Engineering, or a related field (Master's degree is a plus). Technical Certifications: Demonstrated commitment to continuous learning through certifications or relevant training. Qualities: Strong analytical and problem-solving skills to deep dive into complex technical challenges.

Position: Cloud Data Engineer AWS Big Data Location: Bangalore, India Employment Type: Full-time Experience Required: 5 to 8 years Purpose of the Position: Join the Infocepts Cloud Data Architect Team as a Cloud Data Engineer and help design and implement cutting-edge big data solutions on AWS. You will leverage your expertise in EMR, Athena, PySpark, S3, AWS Lambda, and SQL to develop robust and scalable data platforms. Key Responsibilities: Technology Assessment and Design: Assess existing technology landscape and data integration frameworks. Design complex Big Data use cases using AWS services under guidance of the Architect. Support architectural decision-making by evaluating trade-offs in cost, performance, and durability. Recommend optimizations to existing data infrastructure. Documentation and Stakeholder Communication: Create project documentation adhering to quality and delivery standards. Collaborate closely with Architects and Project Managers for scoping, estimation, and planning. Present design decisions to technical and business stakeholders clearly. Conduct PoCs and design review sessions. Process Improvement and Automation: Identify and suggest opportunities for automation and process enhancements. Mentor junior engineers and support technical problem solving. Training and Knowledge Sharing: Prepare and deliver internal training on AWS and Big Data topics. Lead client knowledge sharing sessions and contribute to case studies. Essential Skills: In-depth experience with AWS services: S3, EC2, EMR, Athena, Glue, Lambda Familiarity with MPP databases like Redshift, Snowflake, or SingleStore Proficiency in Apache Spark and Databricks Strong programming skills in Python Experience building data pipelines using AWS and Databricks Knowledge of Big Data file formats such as Delta Lake Advanced SQL skills for large-scale data manipulation Hands-on experience with Apache Airflow or similar orchestration tools Strong understanding of ETL workflows and data warehousing concepts Desirable Skills: Cloud databases: AWS Aurora, Snowflake Experience with Hadoop and Hive AWS Certifications (Associate or Professional level) are a plus Qualifications: Bachelor s degree in Computer Science, Engineering, or related field (Master s preferred) Overall 5+ years of IT experience with at least 3 years in AWS Big Data projects Ongoing learning and technical certifications are strongly encouraged Key Qualities: Strong problem-solving and analytical thinking Self-driven with a passion for emerging data technologies Excellent communication and client presentation skills Ability to work in cross-functional, agile teams Apply now to be part of a high-impact data transformation team working on large-scale cloud data projects! Qualification : Bachelors degree in Computer Science, Engineering, or related field (Masters preferred)

Azure Data Engineer IBM Consulting Client Innovation Center Location: Bengaluru, India Experience: 7+ Years (Minimum 4+ Years in Azure Technologies) Job Type: Full-Time Education Required: Bachelor s Degree (Master s preferred) Introduction: Join IBM Consulting at our Client Innovation Center in Bengaluru, where we deliver deep technical and industry expertise to a wide range of public and private sector clients. Our delivery centers focus on innovation, agility, and adoption of next-gen technologies to transform businesses locally and globally. Your Role & Responsibilities: Design and develop scalable data engineering solutions on Microsoft Azure. Build, optimize, and manage data pipelines and ETL workflows using tools like Azure Data Factory, ADLS, and Databricks. Write clean, efficient code using PySpark, PL/SQL, and Spark SQL. Integrate and manage data from various sources, both structured and unstructured. Work with SQL, Postgres, Cassandra, and Cosmos DB. Utilize Azure services such as Stream Analytics, SQL DW, Azure Functions, ARM Templates, and Analysis Services. Implement and maintain serverless architectures for modern data solutions. Ensure high-quality data delivery, security, and compliance across systems. Collaborate across teams and contribute to solution design, code reviews, and documentation. Required Skills & Experience: Bachelor s degree in Computer Science, Information Systems, or related technical field. 7+ years total experience in Data Engineering, with 4+ years in Azure-based data projects. Strong hands-on experience with: Azure Data Factory, Data Lake (ADLS), Databricks Programming: Python, PySpark, PL/SQL, Spark SQL Databases: SQL, Postgres, Cassandra, Cosmos DB Solid grasp of data warehousing, relational databases, and cloud-based architectures. Familiarity with version control tools like Git and CI/CD pipelines. Preferred Skills & Qualifications: Master s Degree in a relevant field. Experience with: ARM Templates, Azure Functions, Serverless Architectures Object-oriented scripting languages (Python, Scala, etc.) Excellent problem-solving and communication skills. Ability to work in a fast-paced environment and collaborate effectively with teams and stakeholders. What You ll Get: Work on high-impact, global projects with cutting-edge Azure technologies. Be part of a collaborative and forward-thinking IBM team. Opportunities for professional growth, upskilling, and certifications. A dynamic work culture focused on innovation, agility, and client success. Join us in Bengaluru and be part of IBM s journey in shaping the future of data engineering. Qualification : Bachelors degree in Computer Science, Information Systems, or related technical field.

Job Title: Data Engineer Location: Bengaluru, Karnataka Experience: 5 to 8 Years Work Modality: Full-time (Work from office) Job Description: We are looking for an experienced Data Engineer to join our team and take responsibility for designing, developing, and maintaining scalable ETL/ELT pipelines. This is a full-time position based in Bengaluru, Karnataka, and you will be collaborating with cross-functional teams to define data requirements and ensure data accuracy, consistency, and integrity. Your role will also involve optimizing data workflows, automating processes, and ensuring high availability and reliability of data pipelines. Key Responsibilities: ETL/ELT Pipeline Development: Design, develop, and maintain scalable ETL/ELT pipelines to support data transformation and integration processes. Data Warehouse & Data Lake Optimization: Build and optimize data warehouses, data lakes, and real-time streaming solutions to support large-scale data operations. Collaboration & Data Requirements: Collaborate with cross-functional teams, such as product, data science, and analytics teams, to define data requirements and ensure data accuracy and consistency. Database Structure & Schema Management: Develop and maintain database structures and schemas to ensure efficient data storage and retrieval. Data Workflow Optimization: Optimize data workflows for performance, reliability, and scalability, ensuring the highest level of efficiency. Data Security & Compliance: Implement data security, governance, and compliance best practices to ensure that data is handled securely and meets industry standards. Pipeline Monitoring & Troubleshooting: Monitor, troubleshoot, and improve data pipelines to ensure uptime, reliability, and smooth data processing. Process Automation: Automate data-related processes to improve efficiency and reduce manual intervention, increasing the overall speed of data flow. Required Qualifications: Experience: 5+ years of experience in data engineering or 3-4 years of experience as a Data Engineer. Technical Skills: Strong proficiency in SQL and database management systems such as PostgreSQL, MySQL, SQL Server, etc. Experience with ETL tools such as Pentaho, Talend, Cdata, and SSIS. Exposure to Python, Java, or Scala for data processing is a plus. Experience with big data technologies such as Apache Spark, Hadoop, or Kafka. Familiarity with cloud services (AWS, Azure) and data storage solutions such as S3, Redshift, Snowflake, or BigQuery. Strong knowledge of data modeling, warehousing concepts, and data architecture best practices. Soft Skills: Excellent communication skills with the ability to collaborate effectively across teams. Strong problem-solving skills and the ability to work with large, complex datasets. What We Offer: Competitive Salary: Attractive salary based on experience and expertise. Collaborative Work Environment: Work in a dynamic and fast-paced environment with a team that fosters innovation and collaboration. Growth Opportunities: Opportunities to enhance your skills and career growth in the data engineering field. Comprehensive Benefits: Benefits package designed to support work-life balance and overall employee well-being.

Job Title: Data Scientist Location: Bengaluru Experience: 4+ Years About Linarc: Linarc is revolutionizing the construction industry. As the emerging leader in construction technology, we are redefining how projects are planned, executed, and delivered. Built for general contractors, construction managers, and trade partners, Linarc offers a next-generation platform that delivers unmatched collaboration, automation, and real-time intelligence to construction projects. Our mission is to eliminate inefficiencies, streamline workflows, and drive profitability, empowering teams to deliver projects faster, smarter, and with greater control. Join us in shaping the future of construction tech. If you thrive in a dynamic, fast-growing environment and want to make a significant impact, Linarc is the place for you. This is a full-time position based in Bengaluru. Roles and Responsibilities: Data Analysis & Modeling: Develop and implement machine learning models to address complex business problems. Build predictive models and algorithms to forecast customer behavior, product usage, and business outcomes. Conduct statistical analysis and hypothesis testing to uncover key insights and trends. Data Engineering & Processing: Clean, preprocess, and transform large-scale datasets from multiple sources. Design and optimize data pipelines for both real-time and batch processing. Business Insights & Reporting: Collaborate with business teams to define key performance indicators (KPIs). Develop dashboards and reports to effectively communicate findings and business performance. Present actionable insights and recommendations to stakeholders and leadership teams. Product Improvement: Partner with product managers to integrate data-driven insights into product features. Monitor model performance and continuously improve accuracy and scalability. Run A/B tests to evaluate the impact of product changes and optimize user experience. Research & Development: Stay updated with the latest developments in machine learning, artificial intelligence, and data science. Explore and implement new techniques and tools to drive innovative solutions and improve outcomes. Qualifications: Minimum Experience: 3 years of professional experience in Data Analytics, Data Science, AI, or Machine Learning. Technical Expertise: Deep analytical expertise in applying statistical solutions to business problems. Experience with exploratory data analysis of large datasets using tools like SQL, Hive, Spark, or similar. Proficiency in machine learning and deep learning tools such as Scikit-learn, TensorFlow, PyTorch, and familiarity with techniques like Neural Networks, Gradient Boosting, Logistic Regression, Decision Trees, Random Forests, Support Vector Machines, Clustering, etc. Experience working with visualization tools such as Tableau and Power BI. Business & Innovation: Demonstrated ability to innovate solutions and solve business problems using data science techniques. Desirable Additional Experience: 4+ years of experience in data science or machine learning in a product-focused environment. Experience working with relational or document databases. Proficiency in C, Python, PyTorch, Java, JavaScript, Node.js, React.js, or Vue.js. Strong knowledge of hypothesis testing and regression analysis. Experience in a SaaS product environment. Educational Qualifications: Bachelor s/Master s degree in Computer Science, Data Science, Mathematics, Statistics, or a related field.

Job/Position Summary: Data Engineer Responsibilities: Implement data pipelines that meet design and are efficient, scalable, and maintainable. Implement best practices including proper use of source control, participation in code reviews, data validation and testing. Timely deliveries while working on projects. Act as advisor/mentor and helps junior data engineers in their deliverables. Must Have Skills: Should have experience of at least 4+ years with Data Engineering. Strong experience of design, implementation and fine-tuning big data processing pipelines in production environment. Experience with big tools like Hadoop, Spark, Kafka, Hive, Databricks. Experience in programming at least one of with Python, Java, Scala, Shell Script. Experience with relational SQL and NO SQL databases like PostgresSQL, MYSQL, Cassandra etc. Experience with any data visualization tool (Plotly, Tableau, Power BI, Google Data Studio, Quick sight etc.). Good To Have Skills: Should have Basic Knowledge of CI/CD Pipeline. Experience in working on at least one Cloud (AWS or Azure or GCP). For AWS: - Experience with AWS Cloud services like EC2, S3, EMR, RDS, Athena, Glue, Lambda, EMR. For Azure: -Experience with Azure Cloud services like Azure Blob/Data Lake GEN2, Delta Lake, Databricks, Azure SQL, Azure DevOps, Azure Data Factory, Power BI. For GCP: - Experience with GCP Cloud services Big Query, Cloud Storage bucket, DataProc, Dataflow, Pub Sub, Cloud Function, Data Studio. Sound familiarity in Versioning tools (Git, SVN etc.). Experience Mentoring students is desirable. Knowledge of latest developments in Machine Learning, Deep Learning, Optimization in Automotive domain. Open minded approach to explore multiple algorithms to design optimal solution. History of contribution to articles/blogs/whitepapers etc. in Analytics. History of contribution to Open Source. Requirement: ESSENTIAL SKILLS /COMPETENCIES Data Engineering Hadoop Kafka CI/CD Cloud

Your Impact As a Data Engineer at QuantumBlack, you will collaborate with stakeholders, data scientists, and internal teams to develop and implement impactful data products and solutions. Your key responsibilities will include building and maintaining technical platforms for advanced analytics, designing scalable and reproducible data pipelines for machine learning, and ensuring information security within data environments. You will assess clients' data quality, map data fields to hypotheses, and prepare data for use in analytics models. Additionally, you will contribute to R&D projects, internal asset development, and participate in cross-functional problem-solving sessions with a variety of stakeholders, including C-level executives, to create innovative analytics solutions. You will be based in Gurugram, joining a global data engineering community, and work within cross-functional and Agile project teams alongside project managers, data scientists, machine learning engineers, other data engineers, and industry experts. You will collaborate directly with our clients, ranging from data owners and users to C-level executives. You will be aligned with one of our industry-focused practices: Pharma & Medical Products (PMP) or Global Energy & Materials (GEM). In these practices, you ll work on solving the most critical challenges for our clients in these sectors. PMP focuses on advancing the development and delivery of life-saving medicines and medical treatments, while GEM supports industries like chemicals, steel, mining, and energy to achieve operational excellence. GEMx and PMPx, the assetization arm of these practices, focus on creating reusable digital and analytics assets to support client work. As part of this team, you will help shape impactful solutions for large organizations, developing capabilities for sustained impact. Your Growth In this role, you will contribute to the frameworks and libraries that our teams of Data Scientists and Engineers use to progress from data to meaningful impact. You will have the opportunity to guide global companies through data science solutions, helping them transform and enhance performance across industries including healthcare, automotive, energy, and elite sports. Real-World Impact: You ll gain unique learning and development opportunities globally. Fusing Tech & Leadership: Work with the latest technologies and methodologies, with access to top-tier learning programs. Multidisciplinary Teamwork: Collaborate with data scientists, engineers, project managers, UX designers, and more to enhance performance. Innovative Work Culture: Creativity, passion, and wellness are central to our modern work environment, which includes insightful talks, training sessions, and a focus on work-life balance. Striving for Diversity: We celebrate diversity, with colleagues from over 40 nationalities, appreciating the value that diverse perspectives bring to the workplace. You are a highly collaborative individual who prioritizes impact over agenda. You enjoy learning from colleagues, challenging ideas thoughtfully, and working together to improve processes and solve problems. You believe in iterative change, experimenting with new approaches, and advancing quickly through constant learning and improvement. While we value using the right tech for the right task, our team often leverages technologies such as Python, PySpark, SQL, Airflow, Databricks, Kedro (our open-source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions like AWS, GCP, and Azure. Your Role as a Data Engineer Collaboration: Work with business stakeholders, data scientists, and internal teams to create extraordinary, domain-focused data products (reusable assets) and deliver them to clients. Domain Expertise: Develop deep understanding of client industries and use creative techniques to deliver meaningful impact. Technical Platforms: Build and maintain technical platforms for advanced analytics engagements, spanning both data science and data engineering work. Data Pipelines: Design and implement robust, modular, scalable, deployable, and reproducible data pipelines for machine learning. Data Management: Ensure the security of data environments and compliance with information security standards. Data Wrangling: Assess data quality, map data fields to hypotheses, and prepare data for analytics models. Contribute to R&D: Participate in internal asset development and contribute to R&D projects to drive innovation. Cross-functional Problem Solving: Collaborate with internal teams and clients, including data owners and C-level executives, to create impactful analytics solutions. Your Qualifications and Skills Bachelor s degree in computer science or related field; Master's degree is a plus. 2-5 years of relevant work experience. Proficiency in at least one programming language such as Python, Scala, or Java. Strong experience with distributed processing frameworks (e.g., Spark, Hadoop, EMR) and SQL. Experience in commercial client-facing projects, particularly in close-knit teams. Ability to work with structured, semi-structured, and unstructured data, and identify linkages across disparate data sets. Clear communication skills to explain complex solutions effectively. Understanding of information security principles to ensure compliant handling of client data. Experience with cloud platforms (AWS, Azure, Google Cloud, Databricks) is highly desirable. Experience with CI/CD processes using GitHub Actions, CircleCI, or similar, and end-to-end pipeline development including application deployment is a plus. Qualification : Bachelors degree in computer science or related field; Master's degree is a plus.

As a Manager of the Spark Technical Solutions team, you will lead & manage a team of Technical solution engineers and be responsible for driving deep dive technical solutions for any issues reported by Databricks customers. We expect the manager to resolve challenges with comprehensive technical and customer communication skills. You will assist our customers in their Databricks journey and provide them with the guidance, knowledge, and expertise that they need to realise value and achieve their strategic objectives using our products. The impact you will have: As a manager and member of the leadership team, you will be directly responsible for the management of Technical solution engineers, team leads and operations personnel Responsible for directly monitoring, reporting, and driving improvements to team-level metrics and KPIs, acting as an escalation point with customers and internal teams, and optimising and developing support processes and tools Responsible for working across multiple cross functional teams that include Engineering, product management, sales and customer success; manage Hiring, mentoring and onboarding new support engineers Regularly meet one-on-one with your direct reports, conducting annual reviews and career development discussions throughout the year Be a hands on manager to assist the team members in resolving issues related to Spark core internals, Spark SQL, Structured Streaming, Delta, Lakehouse and other databricks runtime features Manage and drive best practices guidance around Spark runtime performance and usage of Spark core libraries and APIs for custom-built solutions developed by Databricks customers; contribute in the development of tools/automation initiatives Own Engineering JIRA tickets and proactively work to bring quicker resolutions to customer reported issues; participate in creation of knowledge base articles Participate in weekend and weekday on-call rotation and run escalations during databricks runtime outages, incident situations, ability to multitask and plan day 2 day activities and provide escalated level of support for critical customer operational issues, etc What we look for: Min 10-12 years of experience in designing, building, testing, and maintaining Python/Java/Scala/Spark based applications in a typical project delivery and consulting environments with 4+ years working as a Manager 5+ years of hands-on experience in developing and leading any two or more of the Big Data, Hadoop, Spark,Machine Learning, Artificial Intelligence, Streaming, Kafka, Data Science, ElasticSearch related industry use cases at the production scale. Spark experience is mandatory Hands on experience in the performance tuning/troubleshooting of Hive and Spark based applications at production scale. Real time experience in JVM and Memory Management techniques such as Garbage collections, Heap/Thread Dump Analysis is preferred Working and hands-on experience with Data lakes and any SQL-based databases, Data Warehousing/ETL technologies like Informatica, DataStage, Oracle, Teradata, SQL Server, MySQL is preferred Hands-on experience with AWS or Azure or GCP is preferred Experience in implementing CI/CD, Monitoring/alerting for Production Systems Technical lead in design, implementation and support of large scale data and analytics solutions that are highly reliable, flexible, and scalable Experience in leading and managing end-to-end projects and have reported and escalated to top levels Experience in managing and leading teams in an organisation involving multiple reporting lines Strong written and verbal communication skills; very good analytical, organisational, multi-tasking skills About Databricks Databricks is the data and AI company. More than 10,000 organizations worldwide including Comcast, Cond Nast, Grammarly, and over 50% of the Fortune 500 rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark , Delta Lake and MLflow. To learn more, follow Databricks on Twitter,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region, please visithttps://www.mybenefitsnow.com/databricks. Our Commitment to Diversity and Inclusion At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics. Compliance If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.