Sobre esta vaga de Data Engineer na Kogan
What you'll do:
Scalable Pipeline Development: Design and maintain ETL/ELT pipelines capable of handling 10M+ daily events and large-scale data transfers across our platforms.
Data Modeling: Develop and optimize data models in environments like BigQuery or Snowflake to ensure high performance for both analytics and ML training sets with optimal cost
Support ML Workflows: Build the underlying features and data inputs required for Machine Learning models
Develop and refine ML models for practical business use cases, such as customer sentiment, churn prediction or demand forecasting
MLOps Integration: Establish and maintain MLOps pipelines to help automate the deployment and monitoring of models in production.
System Integration: Work with internal APIs and third-party tools to ingest data efficiently while maintaining strict data integrity.
Governance & Quality: Implement best practices for data quality, security, and documentation to ensure our data remains a "source of truth."
Development according to software engineering best practices (Git, CI/CD, trunk based development, tests)
AI Collaboration: Contribute to experiments with AI and LLMs to assess how they can be practically applied to solve business problems.
What you'll need:
Strong SQL Foundations: Solid experience writing and optimizing SQL for commercial-scale products (e.g., handling millions of rows and complex joins efficiently).
Pipeline Orchestration: Proven experience using tools like Airflow, dbt, or AWS Glue to manage and monitor production-grade data workflows.
Python Proficiency: Strong Python skills for data transformation, scripting and interacting with various data sources.
ML Engineering Exposure: Practical experience building the data infrastructure that supports machine learning, including data preprocessing and model deployment pipelines. Experience with machine learning models development
Cloud Experience: Hands-on experience with cloud data platforms, with a strong preference for GCP.
Software Best Practices: Familiarity with Git, CI/CD, and basic containerization (Docker) to ensure code quality and deployment reliability.
Problem-Solving Mindset: A practical approach to engineering that balances the need for speed with long-term system stability.
Experience with event streaming (e.g., Kafka, Kinesis) for real-time data needs.
Exposure to ML platforms and tools such as SageMaker, Vertex AI, or Databricks.
Familiarity with BI and visualization tools like Looker or Tableau.
An interest in eCommerce dynamics and customer behavior analytics.
