Key Responsibilities
Ability to design, build and unit test applications on Spark framework on Python.
Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
Develop and execute data pipeline testing processes and validate business rules and policies.
Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation.
Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
Create and maintain integration and regression testing framework on Jenkins integrated with Bit Bucket and/or GIT repositories.
Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings.
Work collaboratively with onsite and offshore team.
Develop & review technical documentation for artifacts delivered.
Ability to solve complex data-driven scenarios and triage towards defects and production issues.
Ability to learn-unlearn-relearn concepts with an open and analytical mindset.
Participate in code release and production deployment.
Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment.
Preferred Qualifications
BE/B.Tech/ B.Sc. in Computer Science/ Statistics from an accredited college or university.
Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications.
Expertise in handling complex large-scale Big Data environments preferably (20Tb+).
Minimum 3 years of experience in the following: HIVE, YARN, HDFS.
Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
Ability to build abstracted, modularized reusable code components.
Prior experience on ETL tools preferably Informatica PowerCenter is advantageous.
Able to quickly adapt and learn.
Able to jump into an ambiguous situation and take the lead on resolution.
Able to communicate and coordinate across various teams.
Are comfortable tackling new challenges and new ways of working
Are ready to move from traditional methods and adapt into agile ones
Comfortable challenging your peers and leadership team.
Can prove yourself quickly and decisively.
Excellent communication skills and Good Customer Centricity.
Strong Target & High Solution Orientation.
Key job details
About Virtusa
Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.
Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.
Virtusa is an Equal Opportunity Employer. All applicants will receive fair and impartial treatment without regard to race, color, religion, sex, national origin, ancestry, age, legally protected physical or mental disability, protected veteran status, status in the U.S. uniformed services, sexual orientation, gender identity or expression, marital status, genetic information or on any other basis which is protected under applicable federal, state or local law.
Applicants may be required to attend interviews in person or by video conference. In addition, candidates may be required to present their current state or government-issued ID during each interview. All candidates must be authorized to work in the USA.
Learn more
Awards and recognition
Syncing your Virtusa Information
Syncing Complete
Exporting your personal Virtusa brand items
Export Complete
Check your downloads folder for files and implementation instructions.
Assets are now available in your profile for future editing and use.
Loading your existing headshots
Please select any one of the images below
Downloading Secure Assets
Signing out