Really? Well I thought I'd check by starting a new game, earning exactly $200 and then subsequently dying in sequence giving the following figures.
$200
$180 (200*10%=20 200-20=180)
$162 (180*10%=18 180-18=162)
$146 (162*10%=16.2, 16 to the nearest $, 162-16=146)
$132 (146*10%=14.6, it appears GTA always rounds down on this basis as it has minused $14 from $146)
$119 (132*10%=13.2, rounded down making $13)
$108 (119*10%=11.9, supporting the rounding down theory as $11 has been subtracted as opposed to 12)
$98 (108*10%=10.8, confirming the rounding theory as $10 has been subtracted instead of 11)
It is clear from these results that the answer is most definately not 1/12th, it is unclear how you arrived at such a figure, though it may be possible that the rounding down basis with GTA applies has thrown off however you tested it or it's back to school for some maths. It is also possible that the $10,000 ceiling has skewed your answer should you have been dealing with an original figure of over $100,000. However testing more then once would easily have displayed any error made, which is why no test should be completed only once as this leads to misleading and/or incorrect data caused by a simple lack of input in order to get the desired output. This is especially true of statistical calculations particularly when dealing with rounded figures in this way.
To go further to fully prove the answer, by confirming the $10,000 limit, I loaded my complete save and did the exact same test but of course starting with much more money, fortunately my starting value was a rounded hundred for easier checking.
$1,261,800
$1,251,800 (-$10,000 from original, expressed as a percentage this works out at ~0.7925%)
$1,241,800 (-$10,000 again, expressed as a percentage ~0.7988%, note changed % when compared to the original, confirming no one fraction is true, simple maths)
$1,231,800 (-$10,000 again, you can see where this is going by now, ~0.8053% way off 10% let alone 1/12th which is ~8.3%, recurring 3)
This confirms that a $10,000 is the maximum that can be lost, and that the % (and therefore fraction) is ever changing once $100,000 is passed. This also confirms my original answer of 10% to a maximum of $10,000, and we can now add that to be precise it is 10%, rounded down to the nearest whole number and to a maximum of $10,000.
Lesson to be learnt, thoroughly check your figures and do your research especially when more then one person is suggesting the same answer which you believe to be incorrect, that's a sign you should at least double check. There's nothing more embarassing then getting your own question wrong.
Class dismissed.