Lead the engineering and technical strategy for a scalable multitenant data integration platform leveraging Apache Airflow and GCP technologies
Job Description
- Lead engineering delivery and technical strategy for a multitenant data integration platform
- Build and maintain productiongrade ETLELT pipelines orchestrated with Apache Airflow Utilize container platforms such as GKE and Cloud Run
- Apply strong software engineering skills in Python APIs configuration management testing and performance tuning Implement CICD pipelines release controls automated quality gates and InfrastructureasCode Terraform
- Design for reliability including retries idempotency deadletter patterns and operational resilience Translate complex technical concepts to diverse audiences and align crossteam efforts Work with GCP BigQuery for largescale data warehousing and analytics
- Architect cloud solutions using GCP Cloud Architecture best practices
- Desirable Experience with data lineagemetadata management tools multitenant architectures data governance and vendor system integrations
Roles and Responsibilities
- Own and drive the technical roadmap aligning engineering with business priorities and operational goals
- Lead solution design for new integrations and platform enhancements ensuring maintainability and scalability
- Establish and enforce engineering standards for coding testing documentation observability and security
- Contribute handson to critical ETLs framework components design spikes and performance optimizations
- Design and develop robust Airflow ETL pipelines with effective scheduling dependency management and error handling
- Develop onboarding patterns for new source and vendor systems ensuring reliability and traceability
- Drive data quality controls including validation reconciliation and exception handling
- Enhance the multitenant architecture focusing on tenant isolation configuration metadatadriven pipelines and runtime scalability
- Optimize platform performance cost and reliability across environments with capacity planning and SLAs
- Improve automation for build deployment and release processes with CICD best practices and selfservice tooling
- Implement comprehensive data lineage and observability with logging metrics ing and dashboards
- Partner with stakeholders including product owners architects vendors and governance teams to manage requirements and delivery
- Lead agile ceremonies remove blockers and maintain delivery momentum
- Mentor engineers through reviews and coaching to foster continuous improvement and capability growth
Skills
Mandatory Skills : Apache Airflow, ETL Concepts, GCP BigQuery, GCP Cloud Architecture, Python
