Senior Data Engineer

Key Responsibilities Data Pipeline Development: Design, develop, and maintain robust data pipelines using ETL and ELT methodologies to process and integrate data from various sources into a data lake, a central data warehouse, operational data stores, analytical data marts and various application interfaces. Streaming Data Processing: Implement and manage real-time data streaming solutions utilising Kafka, Debezium, Kafka Connect. Workflow Orchestration: Build, schedule and maintain custom workflows using Apache Airflow to ensure timely and accurate data processing and delivery. Database Management: Work with a variety of database technologies, including relational databases (MySQL, PostgreSQL), NoSQL databases (MongoDB) and analytical/big data systems (Redshift, BigQuery, SingleStore). Infrastructure as Code: Employ tools like Terraform, Kubernetes, and Helm to manage and provision infrastructure efficiently. CI/CD Implementation: Develop and maintain continuous integration and deployment pipelines to streamline development processes. Testing and Quality Assurance: Conduct unit and integration testing to ensure high code quality, data integrity and system reliability. Collaboration: Engage with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data needs and deliver solutions. Documentation: Maintain clear and comprehensive documentation of data processes, workflows and systems. Monitoring and Support: Monitoring system performance and addressing faults and failures in production systems as part of an on-call rotation **** Skills and Experience Educational Background: Bachelor's degree in Computer Science, Engineering or a related field. Programming Skills: Proficiency in Python is essential. JavaScript and Scala development experience is advantageous. Data Engineering Experience: Minimum of 3 years of experience in data engineering roles, with a focus on building and managing data pipelines. Software Engineering Experience: Minimum of 2 years experience in software and/or application development roles (can be concurrent with data engineering experience) Streaming Technologies: Hands-on experience with Kafka, Debezium, and Kafka Connect. Data pipeline orchestration tools: Proficiency in a data pipeline orchestration tool or suitable workflow orchestration tool like Apache Airflow (preferred), Databricks, Dagster or Airbyte. Database Expertise: Strong understanding and hands-on experience working with various database technologies, including MySQL, PostgreSQL, MongoDB and Redshift (BigQuery and SingleStore advantageous) Infrastructure Tools: Experience with Terraform, Kubernetes, and Helm for infrastructure management. Cloud Computing: Solid knowledge of cloud computing concepts, with experience in AWS services being advantageous. SQL Proficiency: Ability to write complex SQL queries across different dialects. Testing Practices: Familiarity with unit and integration testing methodologies. CI/CD Pipelines: Experience in setting up and maintaining CI/CD pipelines. Data Science Tools: Exposure to analytical systems and basic data science tooling. Familiarity with basic machine learning and analytical modelling concepts advantageous. BI Reporting Platforms: Exposure to self-service reporting tools like Tableau, Looker and DOMO. Soft Skills Communication: Good verbal and written communication skills, with the ability to convey complex technical concepts to non-technical stakeholders. Team Collaboration: Demonstrated ability to work collaboratively within a team and across departments. Adaptability: Comfortable working in a fast-paced environment with changing priorities, technologies and tooling. Life-long learners will do well here. Problem-Solving: Strong analytical and problem-solving skills. Company Core Values Transparency: We encourage open sharing of work, seeking feedback and having honest conversations promptly. Clear Communication: We simplify ideas, communicate directly and confirm understanding to ensure clarity. Kindness: We value positive vibes, generosity and empathy, both within the team and with customers. High Standards: We insist on delivering consistent excellence, taking ownership and striving to be domain experts. Pursue Growth and Learning: We view every situation as a learning opportunity, encouraging experimentation and continuous improvement. Embrace the Mission: We find joy in our work and recognise the company's broader impact.