Alex Rivera | Logout

How to deal with Number precision in Actionscript?

Asked 2009-03-11T00:15:24.480
18

I have BigDecimal objects serialized with BlazeDS to Actionscript. Once they hit Actionscript as Number objects, they have values like:

140475.32 turns into 140475.31999999999998

How do I deal with this? The problem is that if I use a NumberFormatter with precision of 2, then the value is truncated to 140475.31. Any ideas?

Edit
Report

3 Answers

33

This is my generic solution for the problem (I have blogged about this here):

var toFixed:Function = function(number:Number, factor:int) {
  return Math.round(number * factor)/factor;
}

For example:

trace(toFixed(0.12345678, 10)); //0.1
  • Multiply 0.12345678 by 10; that gives us 1.2345678.
  • When we round 1.2345678, we get 1.0,
  • and finally, 1.0 divided by 10 equals 0.1.

Another example:

trace(toFixed(1.7302394309234435, 10000)); //1.7302
  • Multiply 1.7302394309234435 by 10000; that gives us 17302.394309234435.
  • When we round 17302.394309234435 we get 17302,
  • and finally, 17302 divided by 10000 equals 1.7302.


Edit

Based on the anonymous answer below, there is a nice simplification for the parameter on the method that makes the precision much more intuitive. e.g:

var setPrecision:Function = function(number:Number, precision:int) {
 precision = Math.pow(10, precision);
 return Math.round(number * precision)/precision;
}

var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on

N.B. I added this here just in case anyone sees this as the answer and doesn't scroll down...

answered 2009-03-11T00:47:48.363
1

GraniteDS 2.2 has BigDecimal, BigInteger and Long implementations in ActionScript3, serialization options between Java / Flex for these types, and even code generation tools options in order to generate AS3 big numbers variables for the corresponding Java ones.

See more here: http://www.graniteds.org/confluence/display/DOC22/2.+Big+Number+Implementations.

answered 2010-08-10T13:51:55.277
1

Surprisingly the round function in MS Excel gives us different values then you have presented above. For example in Excel

Round(143,355;2) = 143,36

So my workaround for Excel round is like:

public function setPrecision(number:Number, precision:int):Number {
precision = Math.pow(10, precision);

const excelFactor : Number = 0.00000001;

number += excelFactor;

return (Math.round(number * precision)/precision);
}
answered 2012-10-19T16:37:33.560

Your Answer