The vernier scale works on the principle of using alignment of line segments displaced by a small amount to make fine measurements. Human eye can easily detect this alignment of lines which is the main fact that drives a vernier.
A vernier scale has a main scale and a vernier scale. The main scale has the normal resolution with a least count of 1 mm. The vernier scale is attached to the main scale which can slide on it and has graduations that are spaced by the same 1 mm only but are slightly displaced with respect to the marks on the main scale. The displacement is the key here.
When the vernier scale is closed, that is, it is making a 0 measurement, you will see that the zeros of the main scale and the vernier scale coincide but the first mm mark on the vernier is 1/10th mm short of the first mm mark of the main scale. The second mm mark of the vernier is 2/10th of an mm short of the corresponding main scale mark. Similarly the third is 3/10th short, fourth is 4/10th short until the ninth mark which is 9/10th of an mm short. The 10th mark is 10/10th = 1 mm short of the corresponding mark of the main scale and therefore aligns with the previous main scale reading which is 9 (10-1) mm.
Now say you need to measure 5 mm. The zero of the vernier scale will move 5 mm forward and align with the 5 mm mark on the main scale. You will simply take the reading as 5 mm.
Now let us suppose we need to measure a length which is 5.4 mm. You will slide the vernier scale to the requisite length. The zero of the vernier scale will be slightly ahead of the 5 mm mark of the main scale. Thus 5 mm becomes the main scale reading. The amount by which the zero of the vernier is ahead of the 5 mm mark is 0.4 mm which is 4/10th of an mm.
The first mark after the zero on vernier scale which was earlier short by 1/10th of an mm will now march ahead by a net distance of 3/10th ( 4/10 – 1/10) of an mm of the corresponding mark on the main scale and will still be misaligned. Similarly, the second one will march ahead by a net distance of 2/10th ( 4/10 – 2/10) of an mm. The third one will be ahead by 1/10th (4/10-3/10) of an mm. However, the fourth one which was previously short by 4/10th of an mm will now march ahead by 4/10th of an mm and come into alignment with the main scale mark.
This coincident mark can be seen by the naked eye and can be recorded easily. Thus we say that the fourth mark of the vernier coincides with a main scale mark and thus the vernier scale reading is 4*1/10 = 0.4 mm. hence the total length measurement becomes 5 + 0.4 = 5.4 mm.