iMac 27 RAM slots for 24GB upgrade

I only bought one pair of 8gb sticks from crucial for now (16GB total). When I opened the compartment in the back of my iMac, I noticed that the 8GB (2x4GB) from Apple are in slots 2 and 4 (if you were to label them 1-4, from top to bottom). I assumed that by mixing the sizes (4GB sticks and 8GB sticks), I should have the same sizes next to each other, so I moved the 4GB stick from slot 2 to slot 3 and inserted the two 8GB sticks to slots 1 and 2. Is this correct or should I have left the 4GB stick where it was?

FYI, the iMac reads 24GB and if you go to 'Apple->About this Mac->More Info->Memory', the 8GB pair is on the left hand side and the 4GB pair is on the right hand side.

Any additional information is much appreciated!