TQUSI01000_6073 - Data Engineer

Key Skills & Competencies

Industry specific skills

•    Preferably background from Banking

•    Experience working with Data Products 

Personal skills 

Communicative skills

•    Presentation skills and the ability to adapt communication to the target group

•    Very good communication and presentation skills with proven ability to collaborate with cross-functional teams

Leadership and collaboration skills

•    Ability to collaborate and work effectively in teams

Problem solving and analysis

•    Critical thinking and ability to identify core problems

•    Decision-making under uncertainty

Personal qualities

•    Independence and ability to work with little supervision

•    Flexibility and adaptability to new situations

Technical skills 

Azure Databricks with Unity Catalog

•    Expertise in Azure Databricks: Extensive experience with Azure Databricks for data engineering, data science, and analytics solutions.

•    Unity Catalog: Proficiency in using Unity Catalog for comprehensive data governance, access control, and lineage tracking across multiple environments.

Advanced Python Skills

•    Advanced proficiency in Python for data engineering tasks, including designing and implementing ETL/ELT pipelines and developing reusable, modular code for efficient data transformations.

•    Familiarity with Python libraries like Pandas, NumPy, PySpark,

SQL Expertise

•    Strong SQL skills to design complex queries and optimize database performance.

•    Experience working with Azure databases like Azure SQL Database, Synapse Analytics, and Data Lake for querying, processing, and managing large datasets.

IDMC Data Integration 

•    Mapping design in Cloud Data Integration (CDI) — Source, Target, Expression, Aggregator, Joiner, Lookup, Filter, Router, Sorter, Rank transformations

·    Mapping Task and Task Flow configuration and scheduling

·    Parameterization (in/out parameters, parameter files) for reusable, environment-agnostic pipelines

·    Working with Mapplets and reusable transformations

·    Runtime environments — Secure Agent setup, agent groups, and troubleshooting agent connectivity

Git and CI/CD Expertise

•    Version Control: Advanced experience with Git for managing code repositories, versioning, and collaborating in multi-developer environments.

•    CI/CD Pipelines: Skilled in designing and implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions.

Cloud Data Architecture

•    In-depth knowledge of Azure Cloud architecture, including services like Azure Data Lake and Blob Storage.

•    Hands-on experience in designing and deploying scalable, cloud-native data pipelines for large datasets.

Technical Proficiencies

·    Languages: Python, SQL,

·    Tools & Platforms: Azure Databricks, Unity Catalog, Data Lake, Azure SQL Database, Git, Azure DevOps, 

·    Version Control: Git

·    CI/CD Tools: Azure DevOps

·    ETL: IDMC Data Integration 

Experience Summary

• 5+ years of hands-on experience in delivering end-to-end data solutions using Azure technologies.

• Expertise in Azure Databricks with Unity Catalog for large-scale data processing and governance.

• Advanced skills in Python and SQL for efficient data transformation and analysis.

• Strong experience CI/CD pipelines, and working in Git-based environments for efficient deployment.

• Driving best practices in code quality, data architecture, and DevOps.