At 8/31/10 11:43 AM, Archawn wrote:
Anyone know why Flash thinks the following is correct?
12346 * .1 = 1234.6000000000001
Similar problems are causing my rounding functions to give me similar results for certain numbers.
decimal numbers are stored similar to binary numbers, they're still sums of powers of 2, just negative powers of 2 (2^-1 = 1/2, 2^-2 = 1/4, etc) [floating numbers have an exponent too, not important for now]
hence, .1 would be stored as a sum of powers of 2
1/16 + 1/32 + 1/256 + 1/512 + 1/4096 + 1/8192 + 1/65536 + 1/131072 = 0.09999847412109375
in binary:
.0001100110011001100110011...
you aren't gonna get to .1 with this sequence with a finite number of bits (just like how 1/3 is .33333... in base-10)