Heryanto is the Data Engineering Lead at Kudo, Indonesia’s most prominent Online-to-Offline agent network and digital inventory platform that is committed in empowering small and medium retailers to move forward, be more productive in improving welfare, and contribute to the nation’s prosperity through technology.
He earned his Bachelor of Computer Science from Bina Nusantara University and have since then dedicated his professional experience to different aspects of the data field.
Prior to Kudo, Heryanto has more than 6 years experience in data, from data warehouse modeling to distributed data processing. He also has built several data teams in big companies, including Sinarmas and Kompas Gramedia. To support his career in data, Heryanto also earned a certification and officially became a Certified Google Data Engineer.
By his experiences in big companies, currently he is bringing the best practice and taking Data Engineering team in Kudo to the next level, with the aim to build and develop a strong team to be able to build fully automated data pipeline and processing, ensure zero downtime in every component of the services and to serve as the trustworthy foundation for decision-making.
The Journey To Fully Automated Streaming Data Pipeline
In my experience as data team, most of the work spend on preparing the data. We have to do manual work every time there are new data sources added or changes. And we have to wait for couple hours to wait for fresh data to be able to consume. With all the pain we've endured, we decided to build data pipeline that can automatically do all the work for us. It is gonna be a long journey and in here we would like to share our attempt so far.
Key points: - Kudo Introduction - Data Team in Kudo - What we do before fully automated streaming data pipeline? - How we make this possible? (Services, Architecture, Collaboration, etc) - Data career in Kudo