Logarithms were invented in Scotland in 1614; and they held the solution to the problem of 'dividing' one ratio by another.
We were interested in how many times one ratio goes into another, that is how many times it must be 'added' to itself – really multiplied by itself – to give the other ratio. That is,
Ax = B,
where A and B are ratios, and we are to find x. If we take logarithms of both sides:
x log A = log B.
So, x = log B / log A.
What that means is that we can 'divide' ratios by (really) dividing their logarithms. (We can also 'add' ratios by adding their logarithms, and 'subtract' ratios by subtracting their logarithms: you might like to think about why this works.) The exact number of tones (9:8) in an octave (2:1) is given by log(2/1)/log(9/8), which is about 5.88.
Mercator did calculations like this in his notebooks, and in his later English manuscripts. Thus he was able to work out the exact relationship between different musical intervals: 5.88 tones in an octave, 3.44 tones in a fifth, and so on.
To get an approximation that could be used for dividing up the octave to make a useable scale, he didn't divide the octave into twelve. Instead he tried to find a division of the octave that would give a good approximation for the fourth and fifth within it. He found that the ratio of the octave to the fifth is about 1.710, which is close to 53/31. So he proposed dividing the octave into 53 parts, of which a perfect fifth would be 31. This is a closer approximation than the equal tempered scale we saw above, where the fifth is made to equal 7/12 of an octave.
Mercator went through the whole of one of the popular tuning systems in use in the seventeenth century – the meantone temperament – and showed how each of its notes could be approximated by one of the 53 steps in his octave. Better and better approximations could be found, but only by dividing the octave into more and more small parts, which would be impractical for real musical instruments.
Mercator proved his point, even if it was only in manuscripts that circulated privately, that ratios could still be useful for studying music, and indeed that musical problems could be a stimulus for new developments in the handling of ratios. It is a shame that he didn't publish his musical work – there is a good deal more to it than I have written about here – he was certainly planning to at one stage, and it seems as though the plague was to blame for disrupting that plan. He re-used some of the same ideas in the 1660s to provide a commentary on some new musical theories that were causing a stir at the Royal Society: but again, nothing was published.
Only one copy is now left of Mercator's prospectus, the Rationes mathematicae, published in Copenhagen in 1653. It's in Paris, and I've never seen the actual book. But these days it's rather easier to get things across the English Channel, and the Bibliothèque Nationale mailed page images to me on a CD. It's a pity that Mercator didn’t have that option.