Today’s guest post comes from Dale Humby, CTO of Nomanini, an enterprise payments platform provider based in South Africa that enables transactions in the cash-based retail sector. With Google Cloud Platform, Nomanini has seen a 20 percent boost in productivity since developers can focus on rolling out new features instead of focusing on infrastructure. Read how Humby and his team are enabling the delivery of essential services in far-flung locations.
About 50 percent of Africa’s population, or half a billion people, live on less than $1 per day. Many people don’t have reliable access to electricity and telecommunications. Nomanini and our partners are helping change this by providing access to these essential services. Our network of merchants, equipped with point of sale terminals and a financial backend powered by Google Cloud Platform, allows us to distribute pre-paid airtime and electricity to far-flung, rural villages in an affordable and reliable way.
Vendors in Maputo
Our custom built, ruggedized point of sale terminals are known for their speed and reliability – and our financial processing backend has to be just as reliable. Merchants use Nomanini’s platform to make a living by getting commissions from their sales. Even a single incorrect sale or a few minutes of downtime costs merchants customers and money.
Many people choose Google App Engine because it automatically scales to hundreds of servers and beyond. While this is important to us as we grow exponentially, we initially chose App Engine for its reliability: Downtime puts merchants’ livelihoods at risk. We make extensive use of Task Queues for all processing, and Datastore, GCP’s high-availability NoSQL database, as our financial transaction store. In addition, all data is streamed in real-time to BigQuery for customer analytics and reporting. Nightly reconciliation jobs export data from Datastore to Google Cloud Storage for long-term backup.
The financial technology space is highly competitive. We have to continually improve our product or get left behind. Over the past four years Nomanini has built an engineering process that allows us to continually innovate, while simultaneously ensuring our product is (almost) bug free. Our team has doubled from three developers to six, while the number of deployments to production has increased from one per month to more than six per day – representing an increase in development velocity of 120 times. To do this we’ve been using Kanban, a production methodology developed by Toyota, and more recently adopted by the software industry as a way to remain agile.
Team size and deploys to production per month since 2011
By continually improving our development process, we can build better products for our customers and provide updates that help them grow their own returns, like rolling out firmware with new products for our clients to sell and improving battery life.
Once code for the embedded system has been committed, it’s pushed to our cloud-hosted Mercurial source code repository. The continuous integration server checks out the code, builds all the artifacts, including executable binaries for the point of sale terminals, and uploads them to Google Cloud Storage. We run a service on App Engine that manages our build pipeline, tracking each commit through CI and test, Alpha, Beta and Stable phases.
Release pipeline: New versions in CI at top, stable version in production below
Production terminals are subscribed to either the Beta or Stable channels, and when a new version of the application is available they download the binaries directly from Cloud Storage over the GSM network, then install and upgrade automatically. Using Cloud Storage means we don’t have to manage our own FTP or HTTP file server, reducing operational complexity and cost and increasing reliability of the upgrade service.
Terminals collect diagnostic information and stream it to BigQuery. We run statistical tests comparing metrics between versions of code, and have automated alerts when code in Beta deviates significantly from the Stable version, indicating a potential issue with the new version. (For more details, I’ve written a blog post on Monitoring distributed systems with BigQuery and R)
We make extensive use of Google Cloud Monitoring to view performance and business metrics on TV’s around the office, and receive SMS and email alerts if Cloud Monitoring detects issues. All server logs are streamed to BigQuery where we can investigate issues, store logs for auditing and run analyses, for example, to highlight areas for performance improvement.
Cloud Platform offers a cohesive set of services that would be difficult to build with our small team, and almost impossible to host as reliably and securely as Google. By building on top of Cloud Platform, we can release features for production within a few hours of completing development, delighting our customers and giving us a significant competitive advantage. But what truly drives our product innovation is the knowledge that the better our products, the easier it is for people in remote rural villages to switch the lights on or get in touch with a loved one who’s far away
How Nomanini scaled from releases per day with GCP