Data Engineer (GCP) ID56375

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

WHAT YOU WILL DO
• Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP;
• Develop and manage lakehouse layers and Delta Lake workflows using BigQuery and Dataproc;
• Collaborate with stakeholders across data engineering, compliance, and business teams;
• Design and implement pipelines to acquire, normalise, transform, and release large volumes of financial data;
• Design and implement bitemporal data models on BigQuery for regulatory-grade time-series datasets;
• Build and maintain testing frameworks for data pipelines and transformation logic;
• Own end-to-end solutions including ingestion pipelines, QA workflows, correction management, and audit trails;
• Contribute to shared platform services in a collaborative environment;
• Support implementation of AI solutions including data ingestion, anomaly detection, and semantic search using Vertex AI.

MUST HAVES
• 6–8 years of experience in data engineering;
• Proficiency in Python for data pipelines, transformation logic, and automation;
• Proficiency in SQL with hands-on experience in BigQuery including partitioning, clustering, and time-series queries;
• Experience with Cloud Composer (Apache Airflow) for pipeline orchestration;
• Working knowledge of Dataproc (Apache Spark) for batch ingestion and incremental processing;
• Experience with AI-assisted development tools such as GitHub Copilot or similar;
• Experience with Git version control and collaboration workflows;
• Familiarity with REST APIs for integrations;
• Familiarity with GCP technologies (Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, VPC Service Controls);
• Understanding of financial data concepts related to equities and& other asset classes;
• Upper-intermediate English level.

NICE TO HAVES
• Knowledge of data libraries such as pandas or PySpark;
• Experience with columnar storage and time-series analytics tools such as ClickHouse;
• Familiarity with Dataplex for data governance and lineage;
• Understanding of Change Data Capture (CDC) using Datastream;
• Understanding of bitemporal data modeling concepts;
• Knowledge of financial reference data such as equities, fixed income, or corporate actions;
• Experience with BigQuery cost management techniques;
• Experience with CI/CD pipelines and Terraform for infrastructure as code;
• Exposure to LLMs and& Agentic AI using Vertex AI for data-related use cases.

PERKS AND BENEFITS
• Remote work & Local connection: Work where you feel most productive and connect with your team in periodic meet-ups to strengthen your network and connect with other top experts.
• Legal presence in India: We ensure full local compliance with a structured, secure work environment tailored to Indian regulations.
• Competitive Compensation in INR:  Fair compensation in INR  with dedicated budgets for your personal growth, education, and wellness.
• Innovative Projects: Leverage the latest tech and create cutting-edge solutions for world-recognized clients and the hottest startups.