Author: Chris Wyllie


how to deliver app services to scale

Things you should consider before A/B Testing

A/B testing.  It’s something we’ve probably all heard of, perhaps even used, but it can often be misunderstood and misused as well (even when approached with the best of intentions!). Today we’re going to be taking a look at what A/B testing can be used for, some of the potential challenges you’ll face along the way, and some of the other strategies you can employ to achieve similar outcomes.

Consider the Purpose of your A/B Test and how long it might take

The purpose of A/B testing is to determine which variant (“A” or “B”) of something (think app landing page, signup flow, etc.) is better than the other by running experiments. In these experiments, some users are shown variant A, and others are shown variant B. Conversion rates are then measured for each variant and can be compared to determine which one performs better.

Conceptually, it’s reasonably simple and usually quite easy to implement technically, but the real issues that can be encountered with A/B testing are with experiment design and interpretation of the results. The first big issue with experiment design is the sample size.

To consider a result from A/B testing to be conclusive, you should be aiming for statistical significance in the measured outcomes. However, if you don’t have a suitably large userbase, it may take a long time to achieve statistical significance in practice because you lack the traffic to achieve the necessary statistical margin more quickly. The recent example of Optimizely shutting down their free tier to focus on their enterprise product line certainly suggests that small sample sizes really can make A/B testing tricky.

Another consideration with the experiment design is the time period it will run for. How do you measure when the experiment will be considered complete? If you are running it to the point of statistical significance then this may be many months. During this time, how do you approach changes to the feature you are testing? You effectively have to stick with the variants you designed up front until the test concludes or you risk invalidating the outcomes by changing the variants as the test is running.

With this in mind, conducting meaningful A/B tests can be harder than it sounds when you’re building out a mobile app and trying to respond to the market quickly. Modern agile development practices are at odds with locking in long-running test variants before making decisions for the next changes.

What are the alternatives to A/B testing?

Following the above, we would suggest you explore some of these options as an alternative (or as a supplement) to more traditional A/B testing:

  • Agile development with short release cycles, frequently combing your backlog to improve the app rapidly
  • Conducting user testing at the design stage to iron out any issues early on
  • Using an ASO tool to stay on top of search rankings, ratings & reviews, and the competition
  • Integrating a crash reporting tool to alert you to errors as they occur so you can respond before your users notice
  • Staying on top of retention stats and key conversion metrics to see what impact your changes are having in real time
  • Using automated push campaigns to re-engage users as they drop out
  • Integrating a user feedback tool to gather actionable insights from real users

Adopting a proactive stance to change, whilst engaging your users in conversation about what works for them, and doesn’t


Whilst A/B testing might be a good choice for your project, it is worth checking your experiment design for soundness and considering how you might add other tools and techniques to your process to help you adapt to change more quickly.

If you’re currently looking for tools that can help you on this journey, we’d love to show you our Mobile Application Performance Management (mAPM) offering, which includes conversion analytics, App Store Optimization (ASO), Crash Reporting and more, all in one place from a single SDK. If you would rather just test take a look yourself then you also have the option of a no obligation 30 Day Free Trial where we have made a demo App available for you to play around with.

Rise of the Machine Learning

machine learning apps

One item that featured at both of this year’s big developer conferences, but yet seems to have flown under the radar, is Machine Learning. While Machine Learning was probably the main theme emerging from the Google IO Keynote, the new Core ML toolkit from Apple was a relatively low-key announcement during its much-hyped WWDC keynote.

So, what is Machine Learning And Why Should You Care?

Machine Learning is an area of computer science that allows the development of models that are trained on known datasets, in order to solve problems on data they’ve never seen before. The idea being that the model can be ‘taught’ to identify patterns or correlations in data, and respond with a ‘best guess’ for new data. The model gets better as it is trained on larger and larger volumes of data.

The practical applications of Machine Learning are incredibly wide-ranging, from image machine-learning-appsclassification and object recognition, to Natural Language Processing (NLP) and speech recognition. As the field has advanced over the past years, it is starting to reach a tipping point where it is leaving its academic roots and becoming more and more accessible to developers for everyday tasks.

This is reflected in the release of the new TensorFlow Lite engine from Google, and the Core ML toolkit from Apple. And this is why you should care… both of these technologies allow running a pre-trained Machine Learning model in an app on an Android or Apple smartphone!

This is a huge step towards commoditizing Machine Learning and allowing it to be exploited for common tasks in everyday situations, such as telling you if what you’re about to eat is hotdog or not. The TensorFlow library is already used by AirBnb, Uber, Snapchat and Twitter and both Apple and Google’s documentation sites are full of practical examples and state-of-the-art pre-trained models for you start playing with and tailoring to your problem space. It is important to note that neither of these toolkits allow for training the models on-device, there is a good reason for this. Training of ML models is typically an expensive, iterative process, which needs some serious computing grunt to get the best result.

Until now, one of the best ways to train a model was by using a GPU instance from somewhere like AWS. However, now Google are getting in on the game with the announcement of their Cloud Tensor Processing Unit (TPU) offering. The Cloud TPUs are Google Compute Engine (GCE) instances with hardware specifically designed to quickly train Machine Learning models. They are intended to easily integrate with Google’s open-source TensorFlow toolkit, and should offer a decent advantage over simply using a GPU instance to train your models on.

While Apple doesn’t offer their own hardware to train models on, they have been reasonably astute by supporting tools that convert existing models built with TensorFlow to something that will run on Core ML. In fact, the conversion tools also support models built with other popular toolkits, such as Keras, and Caffe, among others.

Another area where Apple really seem to shine is with performance. During their keynote, Apple quoted a 600x performance gain with their Core ML toolkit on the latest iPhone versus running the same model on a Google Pixel or Galaxy S8.

Machine Learning: Hype vs. Reality

As with every reasonably new technology, there is a degree of hype surrounding Machine Learning. However, even taken with a pinch of salt, this technology definitely seems like one to watch. It seems clear that Machine Learning is going to become a technology that more and more developers will take advantage of as the biggest players in the industry throw some serious energy at making it accessible to everyone.

There’s also never been a better time to learn about ML with the fantastically practical course available from Coupling the knowledge form this with the ease of use of TensorFlow and Core ML’s pre-trained models, you should be well on your way to utilizing ML in your upcoming app projects.

Ready to join with the machines? Get started today with a FREE trial from Kumulos – from push notifications, to app analytics to our industry leading mBaaS platform – we’ve got everything you need to terminate your next app project (sorry – had to squeeze that in there somewhere).

Got an idea for an app but not sure how to get it done? Don’t worry, Kumulos has you covered.