Role Overview:

The Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and infrastructure, with a focus on leveraging Databricks and Data Lake technologies. This role collaborates with data scientists, analysts, and business teams to ensure the eƯicient processing, storage, and accessibility of data for analytics. The Data Engineer will play a key role in transforming raw data into valuable, structured formats and optimizing data workflows to support data-driven decisions across the organization.

Key Responsibilities:

• Design, implement, and manage scalable data pipelines for ingestion, processing, and storage, utilizing Databricks and Data Lake technologies.

• Partner with data scientists, analysts, and business stakeholders to understand data needs and ensure solutions meet analytical requirements.

• Integrate multiple data sources (e.g., databases, APIs, cloud storage) into central repositories such as Data Lakes, ensuring data is easily accessible and optimized for performance.

• Develop and automate ETL (Extract, Transform, Load) processes using Databricks and other big data tools. • Architect and optimize Data Lake infrastructures for both structured and unstructured data storage and processing.

• Apply data validation, transformation, and cleansing processes to ensure consistent and high-quality data across all pipelines.

• Automate data workflows to ensure seamless data processing and fast access for analytics and machine learning.

• Work with cloud platforms like AWS, Google Cloud, and Azure to build scalable, cloud-based data solutions.

• Ensure compliance with data privacy, security, and regulatory standards in data handling and processing.

• Troubleshoot, monitor, and optimize data pipelines for high availability, performance, and error resolution.

• Document data pipeline architectures, best practices, and processes to foster knowledge-sharing and cross-team collaboration.

Core Competencies:

• Databricks Expertise: Solid experience using Databricks to build data pipelines, optimize processing performance, and work with Spark for large-scale data processing.

• Data Lake Knowledge: Strong understanding of Data Lake architecture and best practices for managing structured and unstructured data.

• Programming Skills: Proficient in Python, Java, or Scala for building and automating data workflows.

• Cloud Platform Experience: Skilled in working with cloud platforms (AWS, Azure, Google Cloud).

• ETL Automation: Expertise in automating ETL processes using platforms like Apache Airflow and Databricks workflows.

• Data Integration: Ability to integrate and streamline data from multiple sources into a cohesive pipeline for analytics.

• People Management Skills: Proven ability to eƯectively lead, motivate, and develop a team. This includes setting clear goals, providing constructive feedback, resolving conflicts, and fostering a positive work environment

• Collaboration Skills: Strong team player with the ability to communicate complex technical concepts to both technical and non-technical stakeholders.

Required Skills:

• Proven experience as a Data Engineer, specifically with Databricks and Data Lake technologies.

• Expertise in SQL for managing and querying large datasets.

• Hands-on experience with Databricks, Apache Spark, and related big data tools.

• Proficiency in cloud platforms (AWS, Azure, Google Cloud).

• Familiarity with ETL automation tools like Apache Airflow and Databricks workflows.

• Strong programming skills in Python, Java, or Scala for data processing and pipeline development.

• Excellent problem-solving skills, with the ability to troubleshoot and resolve complex data engineering challenges.

Please refer to job description.