Senior Data Engineer

Отговорности

• Design and implement robust batch and streaming pipelines that ingest, transform, and deliver data to lakehouse and warehouse layers
• Define and maintain scalable data models and schemas; lead decisions on normalization, denormalization, and partitioning strategies
• Architect and build metadata‑driven and contract‑driven workflows for schema evolution, validation, and data quality
• Develop and optimize Python services, APIs, and utilities that support data ingestion, orchestration, observability, and platform automation
• Work with Airflow and similar orchestrators to design resilient, modular DAGs and manage complex dependencies
• Lead the development of distributed processing jobs using Spark or other frameworks; tune performance and resource usage
• Collaborate on infrastructure: Docker and Kubernetes deployments, CI/CD pipelines, and cloud/on‑prem integration
• Set coding standards, implement testing strategies, and establish monitoring and incident‑response practices
• Mentor engineers, perform code reviews, and provide guidance on system design and implementation choices

Изисквания

• Python - 5+ years of experience building production‑grade services, APIs, data pipelines, and libraries
• SQL - 5+ years of expertise in complex joins, window functions, performance tuning, and analytical modeling
• Deep experience with streaming systems such as Kafka, Pub/Sub, or Kinesis, including design of topics, consumer groups, and exactly‑once semantics
• Proficient with Airflow or a comparable orchestrator at scale: dynamic DAGs, custom operators, error handling, and multi‑tenant setups
• Extensive hands‑on knowledge of DBT or similar tools for transformations, modeling, and testing
• Strong experience with Docker and Kubernetes, including deployment patterns, Helm/manifest management, and resource tuning
• Solid understanding of distributed data processing (Spark/Flink/Beam) and how to optimize jobs for large datasets
• Proven ability to design data models, handle schema evolution, and implement data quality and observability frameworks
• Experience building and maintaining CI/CD pipelines and working with Git‑based workflows
• Excellent communication skills in English; able to collaborate across engineering, analytics, and product teams

Nice to Have

• Experience with ClickHouse, PostgreSQL, or other analytical/operational databases at scale
• Familiarity with lakehouse architectures, object storage formats (Parquet, Iceberg, Delta), and partitioning strategies
• Knowledge of Infrastructure as Code tools like Terraform or Helm; exposure to GitOps
• Experience with Flink, Pulsar, or other streaming analytics frameworks
• Background in building internal data platform products (catalogs, quality services, lineage tools)
• Exposure to multi‑region or hybrid cloud architectures and compliance challenges