Note: The job is a remote job and is open to candidates in USA. Alpha Consulting Corp. is a Fortune 500 technology, engineering, and science solutions leader. They are seeking a Data Automation Engineer to design and implement AI-driven automation solutions across AWS and Azure environments, focusing on building scalable data pipelines and automations for analytics and reporting.
Responsibilities
- Design and maintain data pipelines in AWS using S3, RDS/SQL Server, Glue, Lambda, EMR, DynamoDB, and Step Functions
- Develop ETL/ELT processes to move data from multiple data systems including DynamoDB → SQL Server (AWS) and between AWS ↔ Azure SQL systems
- Integrate AWS Connect CRM data into the enterprise data pipeline for analytics and operational reporting
- Engineer, enhance ingestion pipelines with Apache Spark, Flume, Kafka for real-time and batch processing into Apache Solr, AWS Open Search platforms
- Leverage Generative AI services and Frameworks (AWS Bedrock, Amazon Q, Azure OpenAI, Hugging Face, LangChain) to:
- Create automated processes for vector generation and embeddings from unstructured data
- Automate data quality checks, metadata tagging, and lineage tracking
- Enhance ingestion/ETL with LLM-assisted transformation and anomaly detection
- Build conversational BI interfaces that allow natural language access to Solr and SQL data
- Develop AI-powered copilots for pipeline monitoring and automated troubleshooting
- Implement SQL Server stored procedures, indexing, query optimization, profiling, and execution plan tuning to maximize performance
- Apply CI/CD best practices using GitHub, Jenkins, or Azure DevOps for both data pipelines and GenAI model integration
- Ensure security and compliance through IAM, KMS encryption, VPC isolation, RBAC, and firewalls
- Support Agile DevOps processes with sprint-based delivery of pipeline and AI-enabled features
Skills
- BS in Computer Science or related field with 2+ years of data engineering, automation experiences
- Hands-on experience with SQL, SSIS, Python, Spark, Bash, Power shell, AWS/Azure CLIs
- Experience with AWS services like S3, RDS/SQL Server, Glue, Lambda, EMR, DynamoDB
- Familiarity with Apache Flume, Kafka, Solr for large-scale data ingestion and search
- Familiarity with LLM, Gen AI frameworks using AWS Bedrock, Azure OpenAI or open source platform, tools
- Experience with integrating REST API calls in data pipelines and workflows
- Familiarity with JIRA, GitHub / Azure DevOps / Jenkins for SDLC and CI/CD automation
- Strong troubleshooting and performance optimization skills in SQL, Spark or other data engineering solutions
- Experience operationalizing Generative AI (GenAI Ops) pipelines, including model deployment, monitoring, retraining, and lifecycle management for LLMs and AI-enabled data workflows
- Good communication and presentation skills
- Ability to obtain Federal government Public Trust clearance
- Certifications: AWS Data Engineer, AWS AI/ML Specialty, Azure AI Engineer, Databricks certified Data Engineer
- Experience implementing RAG pipelines, embeddings, and vector search with Solr, OpenSearch, FAISS, Pinecone, or Pgvector/SQL server vector types
- Experience with GenAI powered coding tools such as Claude Code, OpenAI Codex, VS Code
- Experience with multi-cloud data integration (AWS ↔ Azure SQL)
- Familiarity with Client BizTalk and SSIS for SQL Server ETL workflows
- Knowledge of data lineage/governance tools (Purview, Unity Catalog, AWS Glue Catalog)
- Familiarity with Infrastructure-as-Code (Terraform/CloudFormation, Bicep) for automated deployments
- Experience with compliance frameworks (FedRAMP, PCI-DSS, HIPAA)
Company Overview
Alpha Consulting Corp. has been exceeding expectations in the IT, pharmaceutical, and clinical staffing business since 1994. It was founded in 1994, and is headquartered in East Brunswick, New Jersey, USA, with a workforce of 201-500 employees. Its website is http://alphaconsulting.com.