Position Purpose: To be part of an existing team to create, maintain, support and improve upon existing internal production applications to speed up delivery of products to our customers. This will also include the implementation of new business rules and functionality the business deems required.
Key Responsibilities:
- Develop technical solutions that help achieve business and customer goals. Be able to provide clear feedback to technical and project teams upon encountering technological limitations.
- Be part of the ongoing development and support of any new solutions, as well as existing legacy production processing systems to agreed estimates and schedules.
- A demonstrated focus on continuous improvement and process optimisation of new and existing systems.
- Work with project management to assist in work breakdowns and estimates as required.
- Ensure documentation is created and maintained with a view to enabling ongoing development support (and where possible improvement) as well as handing over ongoing processing to the existing production team.
- Ensure ongoing code quality, data quality checking, error reporting and correction occurs.
Essential Subject Skills and Experience:
- 2+ years of E2E In-depth Databricks experience on a real project(s) that have gone to production
- Databricks Genie, AI/BI Dashboards and Analytics reporting.
- Databricks pipelines, warehouse, jobs.
- Databricks Platform set up and admin, cost reduction.
- Databricks ELT, CDC, Streaming - general ETL and ELT and ETL/ELT scaling. Databricks Ingestion tools, e.g., SQL Server Connect
- Development of Databricks applications, ‘App’ and ‘One’ experience.
- Strong experience with Delta Lake, Lakehouse and Medallion architectures, Data architecture
- Meta data catalogs and data governance - Databricks Unity catalog.
- Strong data modelling/data expertise.
- Understanding of AI and Ml.
- General computing experience, AWS, S3, CI/CD, Git, SDLCs, Agile/Iterative.
- Excellent communication and time management skills
- Strong analytical and problem-solving skills
- Be flexible/adaptable and can embrace change
Desirable Skills and Experience:
- DB/Spark performance and optimization.
- Strong Apache Spark and pyspark - created Spark clusters, Spark (and Kafka) Streaming, Data pipelines/orchestration.
- Strong Python and Dash development experience.
- Advance SQL, with materialized views, Common Table Expressions.
- Statistics, Machine Learning, data mining, Experimental design.
- MLFlow, AI training/testing/evaluation and its orchestration.
- GenAI, LLMs, Prompt Engineering, RAG, Fine Tuning, Agents.
- AI ways of working, e.g. CRISP-DM, TDSP etc.
