IFT6266

Le tableau présenté ici est celui demandé le 26 avril en classe. Tous les modèles sont de même capacité avec tous les mêmes hyper-paramètres. Seules quelques détails sont changés et sont expliqués dans le tableau.

3 hidden layers, 1000 units by layer, around 12 millions exemples seen for each layers in the pre-train, pre-train learning rate  of 0.01, decreasing finetune learning rate starting from 0.1 and is divided by 2 on each epoch of P07 (or 100 times NIST), version theano: c02f8559e0f2+, corruption level of the DA 0,2, validation set used by the model is always NIST. Calculations were made on GPU. The model's directory PATH is /data/lisa/data/ift6266h10/experiments_SDA/

Note that a pre-train on P07 and PNIST, especially PNIST is great for the test error on NIST. Note also that a second finetune on NIST makes the validation error to go down, but test error goes up. This might be an indicator of an over-fit or maybe the test distribution is more like P07 and PNIST than NIST.

The choice of the final non-linearity (sigmoid or softmax) does not seems to be an important factor for the performance, but the inside non-linearity (tanh or sigmoid) seems to have a greater effect. Obviously, the effect of the inside non-linearity is not the same with NIST , P07 and PNIST. WIth NIST, the sigmoid non-linearity seems to be the better choice. On the other side, with P07 and PNIST, tanh seems to be from far the better choice.

The error rate on the test set for NIST is higher when a second finetune on NIST is done after a first one on P07 or PNIST. This situation is strange, but it is possible to seen that it is the inverse for the apriori error rate (the error rate when the model know the class of the image) on the test set of NIST. When a second finetune is done on NIST, the apriori error rate on the NIST test set goes down. The second finetune is good for making a discrimination in a class, but not between class. Obviously, I do not have an hypothesis about this strange behaviour yet.

During the training process, the validation set used was always NIST, but in regards to the results in the table we can see a problem. A low validatoin error rate on NIST is not predictive of a low test error rate. As an experiment, we tried to calculate the final validation error on P07 and PNIST too for all the models. WIth these tests, we can see that PNIST is more representative of the NIST test set. A low validation error rate on PNIST is predictive of a low test error on NIST.

Please also note that the ift6266 version is for information only. The exact code ran is in the model's directory reported in the table. The code may differ a little from the repository.