Business Analyst - Spark Developer Job in Genpact

Responsibilities

• Should have experience working on Spark and SQL modules of Spark extensively
• Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
• Should have experience in analyzing Hive SQL scripts and crafted a solution to implement using Scala
• Should have experience in crafting and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
• Develop Spark scripts by using Scala shell commands as per the requirement
• Develop Scala scripts, UDFs using both Data frames/SQL/Data sets and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop
• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, accurate level of Parallelism and memory tuning
• Expertise with the tools in Hadoop Ecosystem including Hive, HDFS, MapReduce, Sqoop, Spark, Kafka, Yarn
• Excellent knowledge on Hadoop ecosystems such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node
• Very good understanding of Partitions, Bucketing concepts in Hive and crafted both Managed and External tables in Hive to optimize performance.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself
• Develop Hive queries to process the data and generate the data cubes for visualizing
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, and Scala.
• Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data
• Should be involved in building Hive tables, and loading and analyzing data using hive queries
• Proven experience with SQL queries and database tuning
• Strong knowledge of database design and development with previous experience in developing ETL processes, and multifaceted data models
• Respond to & solving support enquiries from users across various groups including Finance, Digital and Operations

Qualifications we seek in you!

Minimum Qualifications

• Programming languages: Java, C, C++, Scala, Python
• Scripting: Shell
• Operating systems: Linux, Unix, Windows
• RDBMS/No SQL: SQL Server, MySQL, Oracle 11g, Azure, HBase
• Hadoop Ecosystem: Spark Scala, Spark Python, Map Reduce, Hive, HBase, HDFS, Sqoop, Pig, , Zookeeper, Kafka, Spark streaming, Oozie

Preferred qualifications

• Analytical thinking and problem solving abilities.
• Good Presentation Skills