Note: The job is a remote job and is open to candidates in USA. Digital Remedy is a performance media partner for agencies, brands, and media companies. As the Sr. Principal Data Engineer, you will architect the next-generation data platform, ensuring scalable and resilient integrations while overseeing data lakehouse architecture and managing production excellence.
Responsibilities
- Architectural Leadership: Design and oversee the migration/optimization of our data lakehouse architecture using Databricks
- Production Excellence: Serve as the escalation point for the platform, leveraging a proven record of handling production issues to ensure 24/7 reliability of mission-critical pipelines
- Strategic Capacity Planning: Work directly with the SVP of Tech to translate product roadmaps into technical requirements. This requires the ability to multi-task across various high-priority workstreams without losing sight of architectural integrity
- Infrastructure as Code: Drive the adoption of Databricks Asset Bundles (DABs) to standardize deployments across dev, staging, and production
- Mentorship & Governance: Provide high-level guidance to engineering squads in India and the US, defining schemas and governance models for BigQuery and Databricks
- Work with engineers across multiple geographies and time zones
Skills
- Expert-Level Databricks: Minimum 4+ years of hands-on experience specifically within the Databricks ecosystem (Delta Lake, Unity Catalog, Photon)
- The Stack: Deep proficiency in Spark (PySpark/Scala), Python, and SQL. Experience with GCP (BigQuery) is a major plus
- Scale: Proven experience managing petabyte-scale datasets and high-concurrency pipelines (AdTech experience preferred)
- Independent Execution: Must be a self-starter capable of taking a high-level concept from the SVP and driving it to completion with minimal supervision
- Experience: 10+ years in Data Engineering, with at least 3 years in a Principal or Architect capacity
- Incident Management: A history of successfully diagnosing and resolving complex, large-scale production bottlenecks and data outages
- Context Switching: A demonstrated ability to multi-task, balancing long-term architectural R&D with immediate business requests and partner integrations
- FinOps & Communication: Strong understanding of optimizing Databricks/BigQuery costs and the ability to communicate technical trade-offs to non-technical stakeholders
Benefits
- Participation in a retirement plan
- Quarterly and annual incentive plan bonuses
- Life and disability insurance
- Health, dental and vision plans
- Flexible spending accounts
- 15 paid vacation days
- 2 paid personal days
- 9 paid holidays
- 40 hours of paid sick leave
- Parental leave
- Employee stock purchase plan
Company Overview
Digital Remedy is a digital media solutions company leading the tech-enabled marketing space. It was founded in 2000, and is headquartered in New York, New York, USA, with a workforce of 51-200 employees. Its website is http://www.digitalremedy.com/.