Data Engineering Manager

Overview

The Data Engineering Manager is responsible for
• Design and build a metadata-driven file ingestion framework on Databricks aligned with medallion architecture
• Lead onboarding of datasets using pattern-based ingestion (not one-off pipelines)
• Define and enforce data contracts including schema, keys, and data quality rules
• Implement scalable ingestion patterns supporting header / no-header / schema evolution scenarios
• Drive table-driven configuration model to eliminate dependency on static YAML-based onboarding
• Integrate ingestion framework with governance tools (e.g., DataHub) for lineage, discovery, and ownership
• Ensure strong data quality, auditability, and observability through centralized logging and control tables
• Collaborate with business and source system teams to understand data semantics and define keys
• Establish best practices for file ingestion, schema management, and incremental processing
• Lead vendor teams to ensure delivery aligns with target architecture and platform standards

Education
• BLD - Bachelor's Level Degree in computer science required
• 15 years of experience in a complex computing environment may be considered in lieu of degree.

Certification/Licensure
• Databricks Certification (Preferred)

Experience
• 8+ years in data engineering with strong experience in modern data platforms (Azure preferred)
• Hands-on expertise in Databricks and PySpark for large-scale data processing
• Strong understanding of medallion architecture (Bronze / Silver / Gold)
• Experience designing metadata-driven or configuration-driven data pipelines
• Deep knowledge of data ingestion patterns (batch, file-based, incremental loads)
• Strong experience with data modeling concepts (keys, SCD, merge strategies)
• Experience implementing data quality frameworks and validation rules
• Proficiency in SQL and distributed data processing concepts
• Experience with Azure Data Lake Storage (ADLS) and file-based ingestion patterns
• Familiarity with CI/CD, Git, and deployment practices in data engineering
Preferred Experience
• Experience with data catalog and governance tools (DataHub, Collibra, Alation, etc.)
• Exposure to customer 360 / MDM / CRM data integration (Salesforce, Dynamics, etc.)
• Experience working in healthcare or regulated environments (HIPAA, PHI handling)
• Familiarity with Unity Catalog and data access governance models
• Experience integrating data platforms with workflow tools like ServiceNow for access management
• Exposure to event-driven ingestion patterns (file triggers, streaming, etc.)
• Experience working in large-scale migration or modernization programs ,enterprise data platforms
• Experience enabling self-service data access and governance
• Familiarity with API-driven data integration patterns

We provide market-competitive compensation packages, inclusive of base pay, incentives, and benefits. The base pay rate for Full Time employment is:$116,729.60-$216,777.60. Additional compensation may be available for this role such as shift differentials, standby/on-call, overtime, premiums, extra shift incentives, or bonus opportunities.