Scaling a football simulation game to 8 million users
Golden Manager is a football game for the web with an ambitious monetization strategy. Backed by F.C. Barcelona player Gerard Piqué, it started out as an MVP developed by a third-party agency to get it off the ground quickly.
Although useful at the beginning, the MVP showed quite a few problems very soon. On one hand development had slowed down to a crawl due to poor code quality, but the biggest problem was its architecture and design, which made it unable to scale to the demands of a hungry fandom. Every time Piqué tweeted about the game, thousands of fans came running to sign up, crashing the entire system. On top of that, their monetized features were too slow to be engaging to users, who often walked away in frustration.
That's when we came in. Once we stabilized the platform, and without stopping feature development at any time, we overhauled the whole system. We took the self-hosted monolithic Rails application, deployed with much fear once a week, to a rock-solid, multi-region AWS deployment that handled 8 million registered accounts, with thousands of concurrent users at any given time.
We built a multi-world, multi-region production system with millions of users and thousands of concurrent players
To improve maintainability, we also ported the monolith to a well documented JSON API with a thick Ember client, which was paramount to the later development of iOS and Android clients.
We also rewrote their exclusive revenue generator, an auction system, from a slow feature with very low engagement to a real-time system that players loved using, all while setting business and production metrics to measure every step of the project.
Aside from working closely with the whole C-level as technology advisors, by the time we left we had hired a team of 10 developers, to whom we handed over a rock-solid production system and a whole set of best practices and guidelines to aid further development.
The challenge of too much success
Having reached the right market early, and thanks to the strong brand that is Gerard Piqué, there was no shortage of paying customers for Golden Manager. Every ad and every social media campaign brought in thousands of users eager to play, and ready to spend real money on their virtual auction system, where users compete to acquire the most valuable football players in the game's world.
The self-hosted Rails monolith was trying its best to serve the increasing load, but with no in-house development team, and serious structural challenges in the architecture, it was far from stable. Deploys were infrequent and fearful of rocking the boat too much. One of the major challenges was the sign up and log in service, which handled users from all over the world and crashed consistently with every advertisement campaign, and even with Piqué's tweets about the game.
At the core of the game, the virtual auction system was the exclusive revenue generating feature. However, it was not anywhere near real-time, and too unreliable for users to enjoy it, so it had very low usage overall. On top of all this, the feature backlog kept growing, and there were no metrics in place to assess the health of the product.
"By the end, we had migrated everything to a multi-region AWS deployment. We also split the playable world in regions, load-balanced after login, to offer a customized experience to users in different parts of the globe"
Scaling with the right architecture
After stabilizing the platform through refactoring and low-hanging fruit, we began reworking the most critical aspects of the system, while setting up business and system metrics throughout. By the end, we had migrated everything to a multi-region AWS deployment. We also split the playable world in regions, load-balanced after login, to offer a customized experience to users in different parts of the globe.
One of the most important structural changes was gradually porting the Rails monolith to a JSON API with an Ember app in the frontend, page by page as to avoid a fearsome big bang rewrite and continue to deliver value —we successfully delivered dozens of features while making this kind of structural changes to the whole system.
Incidentally, decoupling the application with an API was paramount to developing native mobile applications later, when we worked hand in hand with the iOS and Android developers to tailor and optimize certain endpoints for mobile.
We finally rewrote the virtual auction system entirely into a real-time application with Websockets, Ember and Ruby, and soon the revenue started flowing in with highly engaged paying customers and great retention, competing with each other to acquire the best football players.
To tie everything together, we tracked every action in the system and built an ETL to ship it to AWS Redshift, where their Business Intelligence team could analyze it to drive decisions about the product going forward. We also set up data pipelines for the sales team to get an end-to-end view of the conversion funnel throughout the gaming experience
Building a productive team from scratch
When we started working with Golden Manager, there was no in-house team, as they still relied in the original agency that built the MVP to do bug fixes and feature development. We helped the CTO build a competent team from scratch by interviewing dozens of candidates, looking for the best fit.
Eventually we had hired and trained almost a dozen developers on our best practices, and the new architecture that we put in place. After we left, this productive team kept shipping features at a rapid pace, deploying multiple times a day without fear, and with a pervasive metrics-driven culture, where features and metrics go hand in hand.
An engaging game with millions of users and a healthy, productive team
By the end of the project, Golden Manager was far from where it had started.
It now had a multi-world, multi-region production system with millions of users, and thousands of concurrent players at any given time —supported by a productive team built from scratch, deploying multiple times per day and guided by pervasive metrics.
"We helped the CTO build a competent team from scratch by interviewing dozens of candidates, looking for the best fit"
- Data engineering
- Software Architecture & Development
When we handed over the project, we had put a team of 10 in place, trained in our best practices, shipping code daily on a solid production system handling both the steady influx of new paying customers and thousands of concurrent users playing the game.
- Ruby on Rails