Attention prospective job seekers! Beware of fraudulent offers Read more

Big data engineer

Hyderabad, Andhra Pradesh, India
Big data engineer - CREQ189284
  • Position: Big data engineer
  • Primary: big data concepts, aws, pyspark
  • Location: HYD
  • Create Trigger based automation framework for Data Migration
    Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same
    Integrate CDMS framework with Lake and Data bridge API
    Data migration from S3 Managed to Hadoop On prem
    Jobs for Daily and Bulk loads
    Test support for AVRO to test lake features
    Test support for Compression types like LZO, .ENC to test lake features
    ABINITIO integration: Build feature to create operation trigger for ABI pipeline
    Movement to new datacenter -SQL server migration
    Carlstadt to Ashburn (DR switchover)

    Develop and maintain data platforms using Python.
    Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity.
    Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs .
    Implement and manage agents for monitoring, logging, and automation within AWS environments.
    Handling migration from PySpark to AWS.
    (Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc.
    Must have experience with SQL database programming SQL performance tuning relational model analysis.
    Good knowledge in developing UNIX scripts Oracle SQLPLSQL.
    Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging.
    Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management.
    Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs.
Primary Location
Hyderabad, Andhra Pradesh, India
Job Type
Primary Skills
Big Data, Python, Spark
Years of Experience
  • Education: Any degree or equivalent
  • Experience: 6+ years
Job Posting

Join Virtusa


Please enter a valid email address to begin your application.

Thank you for verifying your email. Please proceed with the steps below to apply.

We only accept the following file extensions: .pdf, .docx or .doc
Maximum file size: 1 MB
File name must not include special characters or spaces (e.g. “name_resume.pdf”)

Please attach your CV/Resume, ensure it is in the correct format and smaller than 1MB.
We only accept the following file extensions:

Thank you. You already have an active account with Virtusa's hiring system. Please login to our portal to proceed with your application or apply for more opportunities.

LoginClick to Login

About Virtusa

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 30,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

Learn more