Senior Data Engineer

Position Summary

We are seeking a Senior Data Engineer with specialized expertise in data streaming technologies to join our data team. This role focuses on building and maintaining high-performance data streaming architectures that enable real-time data processing and analytics. The ideal candidate will have deep experience with Apache Kafka, AWS Managed Streaming for Apache Kafka (MSK), Amazon Kinesis, and related streaming technologies in cloud environments.

Role Focus

A Senior Data Engineer at Effectual is primarily responsible for building and maintaining the streaming data architecture that enables real-time data processing and analytics. This involves constructing robust data streaming pipelines that transform and transport data from various sources in real-time, ensuring data flows efficiently through streaming systems for immediate analysis and operational decision-making. You will focus on the efficient and secure management of streaming data systems, ensuring that data is processed, stored, and made available for real-time analytics and downstream applications.

Essential Duties and Responsibilities

Streaming Data Architecture & Pipeline Development

Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis

Develop real-time data pipelines that handle high-volume, high-velocity data streams

Implement event-driven architectures and microservices patterns for streaming data processing

Create and optimize data streaming topologies for complex event processing scenarios

Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms

Kafka & MSK Management

Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments

Implement Kafka Connect pipelines for streaming data integration

Design optimal Kafka topic partitioning strategies and replication configurations

Monitor and optimize Kafka cluster performance, throughput, and latency

Implement Kafka security configurations including SSL/TLS, SASL, and ACLs

Manage Kafka Schema Registry for data serialization and evolution

Kinesis & AWS Streaming Services

Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions

Configure Kinesis Analytics applications for real-time stream processing

Optimize Kinesis shard management and auto-scaling configurations

Implement Kinesis data retention and archival strategies

Integrate Kinesis with other AWS services for comprehensive streaming solutions

Data Processing & Analytics

Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda

Implement complex event processing (CEP) patterns for real-time analytics

Build streaming ETL pipelines that transform data in motion

Create real-time aggregations, windowing operations, and stateful stream processing

Optimize streaming query performance and resource utilization

Integration & Data Flow Management

Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases

Implement data lineage and monitoring for streaming data pipelines

Create automated data quality checks and validation for streaming data

Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution

Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements

DevOps & Infrastructure Management

Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation

Automate deployment and management of streaming infrastructure through CI/CD pipelines

Monitor streaming system health, performance metrics, and alerting

Implement disaster recovery and high availability strategies for streaming systems

Stay current with emerging trends in streaming technologies and cloud-native solutions

Team Collaboration and Project Management

Collaborate with data architects, data scientists, and application teams on streaming data requirements

Support rigorous project governance through daily progress reviews and time tracking

Provide technical leadership and mentorship to junior data engineers

Communicate complex streaming concepts to technical and non-technical stakeholders

Operate with transparency and responsiveness to support high-performing teams

Skills and Experience

Required Experience

7+ years of experience in the data engineering field with significant streaming data specialization

Bachelor's degree in Computer Science, Engineering, or related STEM field

Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools

Proven experience with AWS MSK and Amazon Kinesis services in production environments

Strong background in real-time data processing and stream analytics

Technical Proficiencies

Streaming Technologies: Apache Kafka, Kafka Connect, Kafka Streams, Amazon MSK, Amazon Kinesis (Data Streams, Data Firehose, Analytics)

Programming Languages: Proficient in Python, Java, and Scala for streaming applications

Stream Processing Frameworks: Apache Spark Streaming, Apache Flink, AWS Lambda for stream processing

Data Serialization: Experience with Avro, Protocol Buffers, JSON, and schema registry management

Big Data Technologies: Hadoop ecosystem, Apache Spark, distributed computing concepts

Database Technologies: SQL and NoSQL databases, data warehousing solutions, time-series databases

Cloud and Infrastructure Skills

AWS Services: Deep knowledge of AWS streaming and analytics services (MSK, Kinesis, Lambda, EMR, Glue)

Containerization: Docker and Kubernetes for streaming application deployment

Infrastructure as Code: Terraform, CloudFormation for streaming infrastructure automation

Monitoring: CloudWatch, Prometheus, Grafana for streaming system observability

Security: Implementation of streaming data security, encryption, and access controls

Development and Operations Skills

Expert use of code versioning tools such as GitHub

Expert knowledge of Agile methodologies and delivery practices

Experience with CI/CD pipelines for streaming data applications

Understanding of data APIs, REST services, and microservices architectures

Leadership Competencies

Leadership & Team Management

Risk Management and mitigation strategies for streaming systems

Conflict Resolution

Strategic Planning & Leadership for data streaming initiatives

Resource Management and capacity planning

Change Management for streaming technology adoption

Target Certifications

Core AWS Certifications

AWS Data Engineer Associate (required)

AWS Solutions Architect Professional (preferred)

AWS Developer Professional (recommended)

Streaming-Specific Certifications

Confluent Certified Administrator for Apache Kafka (highly recommended)

Confluent Certified Developer for Apache Kafka (preferred)

Additional Valuable Certifications

AWS Big Data Specialty (if available in current form)

AWS Security Specialist

Certified Associate Data Analyst with Python

Certified Professional Python Programmer Level 1

Databricks Data Engineer Professional

Programming Certifications

Certified Associate Python Programmer

Java or Scala certification (Oracle Certified Professional)

Preferred Qualifications

Experience with Apache Flink for advanced stream processing

Knowledge of Apache Pulsar as an alternative messaging system

Experience with event sourcing and CQRS patterns

Understanding of Apache Airflow for batch and streaming workflow orchestration

Experience with ksqlDB for stream processing using SQL

Background in financial services, IoT, or other real-time data intensive industries

Experience with multi-cloud streaming architectures

Knowledge of Apache NiFi for data flow automation

Performance Metrics

Streaming pipeline uptime and reliability (99.9%+ availability)

Data processing latency and throughput optimization

Cost optimization of streaming infrastructure

Successful real-time analytics implementations

Team productivity and knowledge transfer effectiveness

Company Offered Benefits

Full-time employees are eligible to participate in our employee benefit programs:

Medical, dental, and vision health insurances,

Short term disability, long term disability and life insurances,

401k with Company match

Paid time off (PTO) (120 hours PTO that accrue over one year)

Paid time off for major holidays (14 days per year)

These and any other employee benefit offerings are subject to management’s discretion and may change at any time.

Physical Demands and Work Environment

The work is generally performed in an office environment. Physical demands include sitting, keyboarding, verbal communication, written communication. Employees are occasionally required to stand; walk; reach with hands and arms; climb or balance; and stoop, kneel, crouch, or crawl. The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this position. Reasonable accommodation may be made to enable individuals with disabilities to perform the functions.

Salary Range for this position: $150,000-$180,000

CA ID: IT10000478

"Salary ranges provided are for informational purposes only and may vary depending on factors such as experience, qualifications, and geographic location. The final salary offer will be determined based on the candidate's skills and alignment with the role requirements."

This job description may not be inclusive of all assigned duties, responsibilities, or aspects of the job described, and may be amended anytime at the sole discretion of the Employer. Duties and responsibilities are subject to possible modification to reasonably accommodate individuals with disabilities. To perform this job successfully, the incumbents will possess the skills, aptitudes, and abilities to perform each duty proficiently. This document does not create an employment contract, implied or otherwise, other than an “at will” relationship. Effectual Inc. is an EEO employer and does not discriminate on the basis of any protected classification in its hiring, promoting, or any other job-related opportunity.