The Senior Data Engineer will lead the development and optimization of our data infrastructure, playing a critical role in shaping the future of our data architecture. They will be responsible for designing and implementing complex data solutions, mentoring team members, and collaborating with stakeholders to drive innovative data initiatives. This position requires extensive experience in data engineering, a deep understanding of advanced data technologies, and the ability to provide strategic guidance.

Key Responsibilities

• Architectural Leadership: Lead the design and implementation of scalable, high-performance data architectures.
• Provide strategic input into the evolution of the company's data infrastructure and technology stack.
• Advanced Data Modelling: Design and implement advanced data models to meet complex business requirements. Optimize data storage and retrieval mechanisms for optimal performance.
• Big Data and Advanced Analytics: Champion the use of advanced analytics and big data technologies to derive valuable insights.
• Work on advanced analytics solutions, including machine learning pipelines and predictive modelling.
• Performance Optimization: Oversee the optimization of data pipelines, databases, and ETL processes for maximum efficiency. Conduct in-depth performance tuning and troubleshooting of complex data workflows.
• Team Leadership and Mentorship: Lead a team of data engineers, providing technical guidance and mentorship. Foster a culture of continuous learning and collaboration within the data engineering team.
• Cross-functional Collaboration: Collaborate with data scientists, analysts, software developers, and other stakeholders to understand business needs and deliver comprehensive data solutions.
• Act as a key liaison between the data engineering team and other business units.
• Data Governance and Compliance: Establish and enforce data governance policies, ensuring data quality, integrity, and compliance with regulatory requirements.
• Stay abreast of industry best practices and emerging trends in data governance.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• 10+ years of experience in data engineering with a focus on complex data solutions.
• Expert proficiency in programming languages such as Python, SQL, and advanced scripting languages.
• In-depth knowledge of data warehousing concepts, big data technologies, and cloud platforms (e.g., AWS, Azure, Google Cloud).
• Proven experience in leading and mentoring a team of data engineers.
• Strong analytical and problem-solving skills with a strategic mindset.
• Excellent communication and collaboration skills.

Preferred Skills

• Certifications in relevant data engineering, cloud, or advanced analytics technologies.
• Experience with real-time data processing and streaming technologies.
• Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with data security and privacy considerations.

Tech Stack

Core Tech Skills

• SQL
• Python
• Spark (PySpark & SQL)
• Hadoop

Specific Tech Skills

• AWS S3
• AWS Lambda
• AWS Redshift (Serverless, including Redshift Spectrum)
• AWS EMR
• AWS Step Functions
• AWS EventBridge Rules & Pipes
• AWS Glue
• AWS SQS
• AWS SNS
• AWS API Gateway
• AWS Athena
• AWS VPC
• AWS KMS
• AWS Secrets Manager
• io tables (Open table format)
• Unity Catalog (Open Catalog of Tables, Views, Functions, Models)
• Dynamo DB

Developer/Productivity Tools

• AWS Cloud Development Kit (CDK)
• Docker (for Dev Containers)
• Azure DevOps
• Xray
• JIRA
• Confluence (Wiki)
• Draw.io