Our client is seeking a highly skilled and experienced Azure Data Factory (ADF) Architect to provide consulting and advisory services on ADF pipeline design, implementation, and optimization. The ideal candidate will have a deep understanding of Azure cloud services, data integration, and ETL/ELT processes, with a proven track record of designing scalable and efficient data pipelines. This role involves collaborating with stakeholders to deliver best practices, innovative solutions, and technical guidance for ADF projects.
Key Responsibilities
- Lead consulting and advisory on Azure Data Factory (ADF) pipeline design and implementation, ensuring alignment with business-as-usual (BAU) operations.
- Architect and implement robust ADF pipelines with a focus on API integration, authentication, and end-to-end security best practices.
- Collaborate with stakeholders to translate business requirements into scalable ADF solutions, optimizing for maintainability and operational efficiency.
- Provide expert guidance on ADF features, troubleshooting, and continuous improvement of data workflows in production environments.
- Mentor teams on secure ADF pipeline development, API management, and compliance with enterprise security standards.
Required Qualifications
Technical Expertise:
- Strong expertise in Azure Data Factory, including pipeline design, data flow, and integration runtime configuration.
- Proficiency in Azure cloud services such as Azure Data Lake, Azure Synapse Analytics, Azure SQL Database, and Azure Key Vault.
- Hands-on experience with ETL/ELT processes, data transformation, and data integration patterns.
Programming and Scripting:
- Proficiency in SQL, Python, or other scripting languages for data manipulation and automation.
- Experience with REST APIs and integrating ADF with external systems.
Architectural Knowledge:
- Deep understanding of cloud-based data architecture, including data lakes, data warehouses, and real-time data processing.
- Familiarity with DevOps practices, CI/CD pipelines, and Infrastructure as Code (IaC) using tools like Azure DevOps, ARM templates, or Terraform.
Soft Skills:
- Strong communication and interpersonal skills to engage with both technical and non-technical stakeholders.
- Problem-solving mindset with the ability to provide innovative solutions to complex challenges.
- Leadership and mentoring abilities to guide teams and ensure project success.
Preferred Qualifications
- Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Microsoft Certified: Azure Solutions Architect Expert.
- Experience with other data integration tools like SSIS, Informatica, or Talend (optional but beneficial).
- Knowledge of big data technologies such as Databricks.
