The Position

The Data Engineering team is a vital component of OPSWAT's technology division, responsible for building and maintaining the foundational data infrastructure that supports the entire organization. The team's mission is to create a scalable, reliable, and secure data platform on Microsoft Azure, enabling data-driven decision-making across all business units. This involves designing and managing data warehouses and data lakes, developing robust data integration pipelines, and ensuring the quality and accessibility of data assets. A key strategic initiative for the team is the development and implementation of a Master Data Management (MDM) layer, which will play a critical role in enhancing data consistency and accuracy across various systems and applications.  

What You Will be Doing

• Design and Maintenance of Azure Data Warehouse/Data Lake:
  • Ensuring the performance, reliability, and security of the data platform, requiring proactive monitoring and optimization.
  • Evaluate and implementing strategies to optimize the data warehouse and data lake for cost-effectiveness and efficiency, aligning with best practices for Azure data management 
  • Development of ETL/ELT pipelines on Azure, utilizing tools such as Azure Data Factory and Azure Databricks, to ingest and transform data from a variety of sources.
  • Implementing data quality rules and processes to ensure the accuracy and reliability of the data within the platform
  • Optimize data storage & computing resources for cost efficiency
  • Proactively identify pipeline bottlenecks and performance issues
• Development and Implementation of Master Data Management (MDM) Layer: A key responsibility is to research and recommend suitable MDM tools and technologies available within the Azure ecosystem. Our goal is to establish a "single source of truth" for critical data entities, ensuring consistency and quality across all connected systems.
• Collaboration and Communication: The individual will work closely with product teams, sales and revenue operations, customer experience, and other stakeholders across the organization to understand their data requirements and deliver effective solutions that meet their needs. This requires the ability to explain complex technical concepts in a layman terms to non-technical audiences effectively.
• Risk management: ensure data warehouse availability & resiliency (during service outage, network issues, unexpected high loads, external attacks, disaster recovery, etc.)
• Define effective and measurable KPIs for the team and individual team members
• Team resource management & Key Man Dependency mitigations
• Strategic planning (at least 3 quarters in advance on team roadmap and vision)
• Study new technologies & industry trends and propose appropriate solutions for company’s need

What We Need from You 

• Bachelor's degree in Computer Science, Data Engineering, or a related field.
• Minimum of 3 years of proven experience in data warehouse/data lake design, implementation. This experience should include hands-on work with core Azure data services such as Azure Data Lake, Azure Synapse Analytics, and Azure SQL Database.  
• Extensive experience with ETL/ELT tools within the Azure ecosystem, including Azure Data Factory and Azure Databricks.  
• A strong understanding of database architecture principles, encompassing both relational databases (e.g., Azure SQL Database, Azure Synapse Analytics) and the concepts of NoSQL databases for handling unstructured data. 
• Experience in integrating data from multiple SaaS applications, including Salesforce, Hubspot, Netsuite, HRIS (e.g., Workday, ADP), and ZoomInfo.  
• Experience in researching and implementing Master Data Management (MDM) solutions.
• Strong problem-solving, analytical, and troubleshooting skills are essential for this role.

It Would be Nice if You Had

• Good written and verbal communication skills, including the ability to collaborate effectively with cross-functional teams, are required.
• Experience with real-time data streaming technologies such as Azure Event Hubs or Kafka.
• Experience with DevOps practices and tools for automating data pipelines, such as Azure DevOps and CI/CD pipelines.
• Experience working in a Scrum Agile team.

Company Introduction

Our mission is to enable every organization to be climate conscious and positive with the world’s best climate software and data products. Persefoni is creating an all-in-one platform that allows organizations to measure, analyze, and reduce their Enterprise Carbon Footprint. Our goal is to provide our customers unprecedented visibility and insights into the impact their organization has on the environment. Leveraging the latest breakthroughs in data science and software, our technology will empower teams and leaders to mobilize their organizations to continuously improve their greenhouse gas emissions metrics.

About The Role (What you will get to do)

The Data Engineer will be a key member to assist in building out our web services and backend persistence layer. Successful candidates should have a minimum of three years of recent professional programming in positions requiring the skills listed below with an emphasis on data integrity and data pipelines. Our project entails implementing our pre-approved development targets and developing a robust and reusable code framework to deliver a variety of new features across our product lines according to our preferred architecture design and best practices. Our backend stack is a combination of Golang, Redis, MySQL, and Snowflake while following the Clean Architecture Design pattern. APIs are documented and managed using the Open API framework and maintained in ReadMe. Our data pipelines for data loading and machine learning pipelines are managed via Databricks while our data transformation occurs with DBT against Snowflake within AWS cloud services. All development work is managed via JIRA tracked sprints, Github Git-flow branch management, and infrastructure is managed in AWS with monitoring in Datadog.

Responsibilities (What we expect)

• Collaborate on the design of data models, technical architecture, data flows, schemas, and API contracts.
• Relay updates on existing work via Jira ticket status, push well documented pull requests for features, and collaborate through review and comments on your fellow developer pull requests.
• Building scalable and robust data flows using Databricks, DBT and Snowflake.
• Add/Update to our backend persistence model in MySQL by writing SQL based migration scripts.
• Write/Update data test using native DBT tests and frameworks like Great Expectations to ensure adequate test coverage of our code base.

Qualifications (What you bring to the role)

• At least 3+ years experience building/managing data pipelines within Databricks.
• At least 3+ years experience transforming data in Snowflake with tools like DBT.
• At least 3+ years of experience specifically building data and ML pipelines
• At least 3 years of experience writing SQL against normalized data structures.
• Proficiency with code debugging and uncovering performance issues utilizing developer tooling.
• Proficiency writing data tests.
• ExperienceProficiency in English language is highly preferred. 

How You'll Know You’re on the Right Track

• Within your first day, you should have been able to clone the repository and install the necessary tooling to build and run our projects in cloud based IDEs such as Databricks and DBT.
• Within your first week, you will be familiarizing yourself with the code base and coordinating work on development tickets as requested.
• Within your first month, you should be successfully operating within the development team, submitting PRs, providing feedback on team member PRs, and advancing your assigned projects with your contributions.
• Within your first year, you should be contributing to the data and software engineering teams to provide your personal insights as to how to improve our processes and architectures. 

Job Type:

Full-time position located in Vietnam governed by Vietnam labour regulations, with the Persefoni entity being established in Vietnam.   

Interview Process

• Step 1: Initial screen with talent acquisition
• Step 2: Technical Interview
• Step 3: Interview with Vietnam Country Director
• Step 2: Interview with Engineering Leadership team
• Step 4: Offer extended
• Step 5: Background check and onboarding

Persefoni is an Equal Opportunity employer

We are committed to valuing and respecting people and organizations of all backgrounds. We proudly bring this to life by fostering a culture of innovation, creativity, diversity of thought, and inclusion.

We strive for each of our team members to be able to show up for work every day as their genuine selves. Similar to the reverence given to Earth’s biodiversity, we recognize the vast potential that exists when all of the facets of diversity within our team are appreciated and illuminated. This policy extends to all aspects of our employment practices.