Today, our technology team here at True Analytics is launching our very own engineering blog. This comes after almost four years of mostly staying under the radar with very little official online presence to speak of. Mainly it was because everyone in the team was just focused on quietly doing the work. And with almost every other engineering organization in the world having their own blog, it seemed at first that there’s not much new material we can add to that collective.
But we now know that is far from the truth. While there are indeed many similarities in what these blogs talk about, every technical team still deals with their own unique challenges set within the specific context of their particular business, ecosystem, and circumstance. And the truly successful teams are those that learn to take a nuanced approach in solving them.
So this, our blog, is where we aim to talk about what we’ve learned and will still learn in our own little corner of the tech world. By doing so, we hope to start conversations and learn together with other engineering organizations within Thailand, the region, and elsewhere. But first, a little bit about our story.
How It Started
True Analytics is a business unit under True Digital Group which, in turn, is the digital transformation arm of True Corporation. Our technology team within True Analytics provides all manner of engineering capabilities to our internal data science, product, and commercial teams, and our external partners, and clients.
I joined as employee #6 within the tech team at the start of 2018. Aside from three of us who’ve been around the block a few times, the rest of the team were very young, just one or two jobs removed from university. So there we were, (mostly) fresh-faced, with big ambitions, and very eager to get the party started.
We also had an army of consultants. The original plan was for the consulting team to deliver the initial big data platform while we built up the team and our internal capabilities. Their design and a tool with a fancy UI looked really good on paper. But when faced with the truly huge volume of data that a telco understandably generates, critical components of our pipelines simply couldn’t hold up in production. Big batch jobs were taking so long to complete or just crashing completely. An important streaming process had a backlog measured in days. Multiple attempted fixes – server tuning, code refactors, even throwing additional nodes at it – didn’t do much. Worst of all, the project was already two months behind schedule.
With no fix in sight and the business delay lengthening, our small team started experimenting with alternative approaches. A quick prototype using just straightforward Spark (batch and streaming) demonstrated that it can successfully handle the same problematic pipelines with even less resources required.
We decided to go with this approach and aligned with our executive leadership who were extremely supportive even when told of such a late pivot. After six weeks of massive rewrites and reimplementation (and a full four months past the original go-live date), our data platform finally started running in production. We learned some hard but valuable lessons during those early days and we took them to heart. We’ve been learning more ever since.
How It’s Going
Four years. Our small squad of eight has grown into a tribe of 40+ talented people across multiple disciplines and functions – software and data engineering, architecture, application development, QA, infrastructure and DevOps. Most of our young first-joiners who braved those exciting first few months are still with us, themselves veterans by now. They drive the many initiatives we have, lead teams, and act as mentors for others.
We’ve built both internal tools and external products (some of which are listed here: https://www.truedigital.com/true-analytics). We are actively evolving our technologies (e.g. migrating our Hadoop cluster to a K8s data platform, designing for hybrid on-prem/cloud systems) and continuously upgrading our skills and discipline (MLOps/MLE). And all these, we always strive to do within the same supportive engineering culture and mindset of experimentation.
We are hiring
If working on interesting problems in software, big data and analytics is something you’re passionate about, do check out our jobs page on LinkedIn. Or just reach out to us for a chat.