Nicholas Lu is a goal-oriented engineer focused on ML systems and distributed architecture. He works across the MLOps lifecycle, building and maintaining data and model pipelines using Airflow, Spark, Kafka, and Kubernetes. His experience includes deploying containerised workloads with Docker, managing infrastructure via Terraform, and implementing CI/CD with Jenkins, GitLab, and Bitbucket Pipelines. He supports site reliability through monitoring, automation, and incident response in cloud environments (AWS, GCP, Azure, Alibaba Cloud). He codes in Python, Java, SQL, and Bash, and works extensively with Linux and open-source tools.
Development of EKS/GKE cluster on terraform as well as deployment of Helm via gitops method.
Debug and development of training pipeline and distributed training on GPU nodes with GPU
slicing to job level. Addition of monitoring and alerting tools for model and job for faster and more
automated model development.
Develop containerized applications with kubernetes with Helm charts and KOPS for
infrastructure defined as code. Applications like image classification streaming video analytics
pass through . Leveraging AWS EKS and classical EC2 with a kubernetes environment .
Implement hadoop storage for fellow data scientists . Perform architecture review and
documentation for customer use cases. Manage implementation of Hadoop/Kafka/Airflow into the
team stack. Wrote a terraform script for building a sandbox environment which involves EKS
cluster, EC2, VPC and Cloudwatch.
Pipeline from GCS and MySQL via Federated Query.
Implementation of Databricks cluster via IaC, Cronjob ETL Application-state on ArgoCD. Github
action on Bigquery with data integrity check and data drift detection. BigQuery data model with
DBT and Argoworkflow. Deployment and development of POCs with GCP services which
shortens the time of delivery and compatibility with other GCP ecosystems. Successfully deploy
and deliver first ETL pipeline with automated test, semantic versioning and pre-commit checks.
Develop royalty processing with AWS EMR hadoop and manage ETL scripts in the team from
database to nested/ cloud storage. Develop documentation and guide BI team members of three.
Performs redshift automation and integration with Matillion ETL involving Cassandra offload to
S3 and Redshift and tableau report scheduling. Performed Mainframe AS400 DB2 to on-prem
SQL Server migration plan which was the decommissioning plan of AS400 platform.
