Location:

Shift:

Scheduled Weekly Hours:

Worker Type:

Job Summary:

Job Duties:

Responsibilities

• Data Pipelines: Develop, test, and deploy batch (Spark) and streaming (Flink/Spark Structured Streaming) jobs on OpenShift
• Message & Stream: Create and manage Kafka topics, configure connectors, monitor consumer lag, troubleshoot message delivery
• Lakehouse & Storage: Create and maintain Iceberg tables (schema changes, partitioning, maintenance); operate MinIO for data lake workloads
• Query & Serving: Write SQL transformations (dbt), build Trino queries for federated analytics, manage materialized views on StarRocks/ClickHouse
• Orchestration: Author and maintain Airflow/Dagster DAGs; troubleshoot pipeline failures and data quality issues
• Observability: Build Grafana dashboards and Prometheus alerts for pipeline health, data freshness, and quality metrics
• Kubernetes Operations: Write basic Helm charts, manage ConfigMaps/Secrets, troubleshoot pod failures under guidance
• CI/CD: Maintain CI pipelines for data jobs, contribute to ArgoCD application definitions
• Write clear documentation: runbooks, pipeline specs, onboarding guides
• Participate in code reviews and agile ceremonies

Required Skills & Experience

• 3 – 6 years in data engineering, software engineering, or DevOps
• Working knowledge of Kubernetes: Pods, Deployments, Services, ConfigMaps, basic Helm usage
• Practical Spark experience: PySpark or Spark SQL batch ETL, reading/writing to object storage and Iceberg tables
• Basic Kafka experience: producing and consuming messages, understanding partitions and offsets
• Understanding of data lake/lakehouse concepts: columnar formats (Parquet/ORC), open table formats (Iceberg/Delta Lake), partitioning, schema evolution
• Experience with an S3-compatible object store (MinIO, AWS S3, Ceph)
• Solid SQL skills; proficient in Python
• Comfortable with Git, Docker, CI/CD, and Linux shell scripting

Nice to Have

• Exposure to Trino or Presto for federated queries
• Familiarity with a real-time OLAP engine: StarRocks, ClickHouse, or Doris
• Experience with dbt for data transformation
• Workflow orchestration: Airflow or Dagster
• Interest in data observability, data mesh, and data quality frameworks

Company Introduction:

ITD SZ

港交所科技（深圳）有限公司，是2016年12月28日于深圳市前海自贸区成立的外商独资企业。

作为港交所的技术子公司，港交所科技（深圳）有限公司主要是为集团及其附属公司提供计算机软件、计算机硬件、信息系统、云存储、云计算、物联网和计算机网络的开发、技术服务、技术咨询、技术转让；经济信息咨询、企业管理咨询、商务信息咨询、商业信息咨询、信息系统设计、集成、运行维护；数据库管理、大数据分析；以承接服务外包方式提供系统应用管理和维护、信息技术支持管理、数据处理等信息技术和业务流程外包服务。