Apple Create ML vs Kaggle is it really so good?

During recent WWDC Apple presented their newest tool called Create ML. As a ML enthusiast i was really impressed on what i’ve seen on dedicated session (703), so i thought it would be worth to investigate how powerful it really is.

1. Create ML in a nutshell (you can skip if you watched WWDC presentation).

Create ML is a new framework designed to help developers easily and quickly build machine learning models using Swift and Xcode. As for now it is able to work with 3 types of data: images, text and tabular data. The workflow is really simple, you just need to deliver set of training data, write few lines of code and magic happens, small CoreML model is ready to be used in your app. Apple promises to significantly reduce time of training, which in case of 10,000 images should be counted in minutes (as always Apple avoids providing details). Training is so fast, because (in case of image classifier) Apple uses Transfer Learning. We don’t know what Neural Network is base, but during the presentation Inception-v3 was mentioned as an example of state of the art solution for image classification.

2. WWDC Demo presentation in a nutshell (you can skip if you watched WWDC presentation).

During WWDC, presenters showed how easily we can build different models, first was image classifier, built using only couple lines of code:

Second example was very similar, Text Classifier deciding if text is positive or negative, as in image case, only couple lines of code were enough to train it:

As for me, the most interesting demo was the last one, presenting work with tabular data. In this case Create ML is using new data format, called MLDataTable, at first look it’s just a wrapper for normal data matrix, luckily has built-in CSV and JSON parser.

In case of tabular data Create ML offers 2 paths: easy one and a little bit more sophisticated. At first take a look at the simplest way to find house price formula, it looks nearly the same as both previous examples:

Tabular data problems might be resolved (depending on case) using either MLRegressor for regression problems and MLClassifier for classification. But that’s not all what is really interesting, while working on tabular data, you can manually decide what method of classification/regression will be used, as Create ML support couple most commonly used in industry:

3. Create ML in real life problems

Demo apps are always as simple as possible, it makes them easy to understand and that’s ok. However in case of Machine Learning, fact that something is working with simple example, doesn’t really implies that it will work with a more sophisticated one. Create ML target users are developers, not ML experts or data scientists, so i decided to behave like an average ML newbie, don’t deeply analyze data, just put sets into Create ML and let it do the magic. I decided to use Kaggle competitions as a benchmark, and then compare quality of model with what Kagglers achieved, i also wanted to check how fast Create ML is. In lot of cases, key to the sucessfull model is data understanding and preprocessing, if developer will just import CSV and run MLRegressor expecting high quality results for sophisticated problem… he might be really disappointed with effect. Because of that i decided to choose examples, which are close to real life problems but not very complex, as this would be pointless without data preprocessing.

My Macbook Pro config as a reference:

I started with testing Image Classifier, for testing i used “Plant Seedlings Classification” (https://www.kaggle.com/c/plant-seedlings-classification), which goal is to determine the species of a seedling from an image. Dataset is significantly bigger than the one presented on the demo, it contains over 4300 images, i run the code using only 2 lines of code, result is visible below:
I was really impressed with the result, it took 5 minutes on a typical developer laptop to train a pretty good model, using 1.7GB of training data. Is there any better way to build Proof of Concept model and put into working iOS/macOS app to verify an idea? I don’t think so. Apple, you did a great job, really. What is also interesting to notice, is that Create ML is automatically creating validation set from the one provided as a training, so developer doesn’t have to split this data manually, one small step to simplify process.

Ok, not everyone is going to build “Not Hotdog” app, so let’s take a look at another very common case in machine learning, “Tabular Data” and regression/classification problems. In this case i started with “Getting Started Titanic Survivals” (https://www.kaggle.com/c/titanic), goal of this competition is to determine if particular passenger will survive collision with iceberg based on data like age, sex, type of ticket, deck, etc.

I downloaded CSV file, created MLClassifier, pressed play and…
Ok, it’s not working, Create ML needs perfectly clean data. That’s not a big problem, i can remove rows with empty columns using couple lines of Python, however that reduces amount of training data from 891 examples to only 183, but at least it’s working. During next run xCode informed me that: “Feature \’Ticket\’ is not of type (numeric, string, array, or dictionary).\n” so i dropped this column and finally got working model:
Once again, i am impressed with the result, data was relatively easy, but it’s close to what lot of developers might face in typical apps. Create ML is working, it’s fast and efficient.

Second attempt was “Restaurant Revenue Prediction” competition (https://www.kaggle.com/c/restaurant-revenue-prediction/data), where goal is to estimate revenue of the restaurant in a given year, using date of opening, city, type of restaurant etc. Same as in case of Titanic, i just downloaded the data and run playground:
In this case Create ML decided to use Boosted Tree Regressor and as a result MSE is even lower than the one which won competition 3 years ago. Of course the reason for that might be the fact that leaderboards are created based on data which is not public, but still, Create ML was able to build very good, probably even comparable with winning model in couple seconds. This is what most of developers need.

4. Pros and Cons of Create ML

Pros:
  • currently it’s the easiest way to build ML models for iOS/macOS, especially for Proof Of Concept apps, or those working with relatively simple data
  • Create ML opens ML world to thousands of iOS/macOS developers
  • is very efficient and easy to use
Cons:
  • very few ways to tune and debug final model
  • no public knowledge what model is used to classify images
  • no data preprocessing tools (but that’s obvious, Apple target are not data scientists, but software developers)

5. My opinion

Looks like Create ML is really a great tool, especially for developers, who didn’t work with Machine Learning at all, it’s fast, pretty efficient and extremely easy to start with. If you are an iOS/macOS developer, just give it a try.

6. Open question

Is problem of big ML models for image classification just hidden by the fact that all weights are stored within iOS/macOS and Create ML builds CoreML model containing only weights for last, retrained layer of (Inception-V3)?

P.S.
If you are going to get a job as Machine Learning Engineer be sure to check my post about the most popular questions asked on ML interview: http://theaigeek.com/typical-machine-learning-interview-questions/

Leave a Reply

Your email address will not be published. Required fields are marked *