Multi label vs. Multi class
An experiment in loss functions on the same dataset.
- Experiment intro:
- Tank classifier models:
- Bulding a model using multi class loss
- Bulding a model using multi label loss
- Addtional validation (out of train scope images)
- Comparison summary
Following Jeremy Howard's suggestion in the Practical Deep Learning for Coders course we will run an experiment with two different loss functions on the same data set. In this experiment, we will take a simple multi class model and retrain it as multi-label classification. we will see if:
- We can make it work effectively with images that don't contain any tanks.
- Try an image with two different kinds of tanks.
- Check whether the accuracy on the single-label dataset is impacted using multi-label classification.
Both models are trained to classify between three classes: Merkava mk4 tank, M1 Abrams tank or a water tank
tank_types = 'merkava mk4','M1 Abrams','water'
path = Path('tanks')
#downloading 150 images to labeled directories
if not path.exists():
path.mkdir()
for o in tank_types:
dest = (path/o)
dest.mkdir(exist_ok=True)
urls = search_images_ddg(f'{o} tank', max_images=150)
download_images(dest, urls=urls)
#deleting not working files
fns = get_image_files(path)
failed = verify_images(fns)
failed.map(Path.unlink);
tanks = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parent_label,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls = tanks.dataloaders(path)
dls.valid.show_batch(max_n=5, nrows=1)
learnMC = cnn_learner(dls, resnet18, metrics=error_rate)
learnMC.fine_tune(4) #fastai already chooses multi class loss becuase od the CategoryBlock
Review top 10 loss images:
def parentlabel(x):
return [x.parent.name] # as get_y recieve a list
tanks2 = DataBlock(
blocks=(ImageBlock, MultiCategoryBlock),
#MultiCategoryBlock(add_na=True)
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parentlabel,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls2 = tanks2.dataloaders(path)
dls2.valid.show_batch(nrows=1, ncols=5)
learnML = cnn_learner(dls2, resnet18, metrics=accuracy_multi)
#defaults for the accuracy multi are: threshold=0.5, Sigmod=True
learnML.fine_tune(4)
Addjusting the threshhold (as the curve is smooth we shuold nit be worried about overfitting):
preds, targs = learnML.get_preds()
xs = torch.linspace(0.05,0.95,29)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);
learnML.metrics = partial(accuracy_multi, thresh=0.85)
learnML.validate()
Review top 10 loss images:
The first model is binary and forces to select a single class based on any image input. Its softmax encourages a single class-selection even more. While the second model is multi-label and predicts each class separately, it may be better at alerting images outside the scope of the class or two classes within one image.
Now let's see if one model can perform better on out-of-scope images even though they were both trained on the same dataset:
Trying to predict an image with both Merkava and Abrams tanks in it:
twotanks = PILImage.create('/content/mek_abs_3.jpg')
twotanks.show()
Multi class prediction:
learnMC.predict(twotanks)
Multi labal prediction:
learnML.predict(twotanks)
Both models are quite certain on the same class which is only Abrams tank.
bicycle = PILImage.create('/content/bycycle_nature.jpg')
bicycle.show()
Multi class prediction:
learnMC.predict(bicycle)
Multi labal prediction:
learnML.predict(bicycle)
Both models are quite certain on the same class which is a water tank.
piano = PILImage.create('/content/Grand-Piano.jpg')
piano.show()
Multi class prediction:
learnMC.predict('/content/Grand-Piano.jpg')
Multi labal prediction:
learnML.predict('/content/Grand-Piano.jpg')
Both models are quite certain on the same class which is the Abrams tank.
Although we used two different loss functions which are designed for a different purpose, based on the tests, it turns out both models are performing quite similar:
-
Metrics - both models are around a 7% error rate (actually adjusting the multi label threshold increases its accuracy but I believe the difference is more connected to process randomness and different metric functions than the models).
-
Top loses - looks like the models are having problems on the same images.
-
Out of scope train images - small random check, the multi label model didn't show any advantage in identity the images were out of scope or label two different images as we might expect.
The bottom line proves that the main factor is the data set, the model will be able to learn as good as the examples he learn during training and as we didn't train the multi label model with the relevant examples, in this case, it couldn't out perform the multi class model.