Cloud Infrastructure Architect (Databricks)

This is a remote position.

Role Objective

Lead the architectural design and automation of a Databricks 2.0 ecosystem, focusing on scalable Infrastructure-as-Code (IaaC), multi-environment governance, and decentralized CI/CD orchestration.

Core Responsibilities

• Infrastructure Design: Architect Databricks 2.0 environments (Dev/UAT/Prod) using Terraform, managing networking, compute quotas, and cluster policies.
  


• Security & Governance: Define RBAC frameworks and Unity Catalog strategies. Manage Microsoft Entra ID integrations, Service Principals, and automated key rotations.
  


• CI/CD Orchestration: Build automated pipelines for promoting jobs, notebooks, and workflows, enabling independent team deployments via approval-gate processes.
  


• Environment Parity: Design automated data refresh workflows to sync production data to lower environments for consistent testing.
  

Technical Stack

• Platform: Databricks 2.0 (Unity Catalog, Workflows, DLT).
  


• IaC/DevOps: Terraform, GitHub Actions/Azure DevOps, Git.
  


• Identity: Microsoft Entra ID (Service Principals/RBAC).
  


• Data: Delta Lake, SQL, Python.
  

Key Requirements

• Architectural Expertise: Proven track record in designing secure, scalable Data Lakehouses.
  


• Automation-First Mindset: Experience replacing manual setups with robust, version-controlled infrastructure.
  


• Leadership: Ability to bridge the gap between Infrastructure (Infra) and Data Engineering (DE) teams.