At Synoptic Data PBC, I had the opportunity to lead various data engineering projects and contribute to the development of robust data ingest pipelines. Here are some key aspects of my experience:
Lead data ingest development: I took charge of both maintaining the existing infrastructure and spearheading the development of new data ingest pipelines. This involved designing efficient and scalable solutions to ensure seamless data acquisition from diverse sources.
Performance optimization: I played a crucial role in improving the performance, scalability, and reliability of data processing pipelines. By carefully analyzing and optimizing the algorithms and identifying bottlenecks, I significantly enhanced runtime efficiency and reduced resource consumption.
Backend development: I actively contributed to backend development projects focused on data processing, storage, and delivery. Leveraging my Python skills, I developed robust code modules and implemented efficient data manipulation techniques to facilitate seamless data flow within the system.
Monitoring and observability: I recognized the importance of monitoring production systems to ensure smooth operation and identify potential issues. By enhancing the monitoring infrastructure, I provided comprehensive insights into the state of processing pipelines and datasets, benefiting both internal teams and external applications.
Collaboration and teamwork: Working closely with data scientists, product developers, and the business team, I fostered effective collaboration to deliver high-quality solutions. This collaborative environment allowed for the seamless integration of data engineering practices with the broader goals and requirements of the organization.
Expertise in Python, AWS, AWS Lambda, EC2 servers, etc.: Throughout my work, I leveraged my proficiency in Python programming to build robust data engineering solutions. Additionally, I utilized various AWS services such as AWS Lambda, EC2 servers, and other relevant tools to ensure the scalability, reliability, and security of the data processing pipelines.
Docker Container Management: Managed a Docker Container for the World Meteorological Organization WIS2.0 project. I'm Proficient in managing Docker containers, orchestrating deployments with Docker Compose, implementing Infrastructure as Code (IaC) principles, monitoring and troubleshooting containerized applications, and integrating Docker containers into CI/CD pipelines.