Responsible for designing, developing, and maintaining data processing pipelines using Databricks platform. Working closely with data engineers and data scientists to implement data solutions that meet business requirements, to help client with their cloud migration journey.
It includes the following responsibilities:
Designing and developing data pipelines using Databricks platform.
Writing efficient and optimized code in languages such as Python, Scala, or SQL.
Collaborating with data engineers to ensure data quality and integrity.
Implementing data transformations and aggregations to support analytics and reporting.
Working with data scientists to deploy machine learning models on Databricks.
Troubleshooting and resolving issues related to data pipelines and Databricks environment.
Optimizing performance and scalability of Databricks jobs.
Documenting technical specifications and maintaining code repositories.
Keeping up-to-date with the latest Databricks features and best practices.
Participating in code reviews and providing feedback to improve code quality.
He/she should have a strong understanding of distributed computing concepts and experience with cloud platforms such as Azure. They should also possess good problem-solving skills and be able to work in a collaborative team environment.
