Questions tagged [precision]

194

votes

36

answers

Unexpected results when working with very big integers on interpreted languages

asked 2013-08-04T18:46:25.513

13

votes

2

answers

C++ calculating more precise than double or long double

c++ precision

asked 2013-02-01T00:18:59.070

12

votes

5

answers

How to use and verify quad precision in gfortran?

fortran precision

asked 2012-07-09T16:22:55.880

21

votes

2

answers

GNU Octave, round a number to units precision

linux matrix rounding precision octave

asked 2012-07-08T19:33:16.603

15

votes

3

answers

SQL set floating point precision

sql floating-point precision truncate

asked 2012-06-20T19:19:58.987

15

votes

2

answers

Why is -freciprocal-math unsafe in GCC?

c math gcc optimization precision

asked 2012-05-21T03:18:33.780

10

votes

3

answers

calculating a function in matlab with very small values

matlab precision exponential epsilon

asked 2012-04-04T00:43:10.613

11

votes

4

answers

Precision in C floats

c floating-point precision

asked 2012-03-19T06:09:30.187

15

votes

4

answers

Determine MAX Decimal Scale Used on a Column

sql sql-server t-sql decimal precision

asked 2012-01-23T20:26:37.583

32

votes

2

answers

Why do bean validation Min/Max constraints not support the double type?

java constraints precision bean-validation

asked 2011-09-14T20:49:31.417

12

votes

5

answers

Converting 8 bytes of little-endian binary into a double precision float

java parsing floating-point precision endianness

asked 2011-06-24T16:58:55.070

12

votes

2

answers

Instability in DeleteDuplicates and Tally

wolfram-mathematica duplicates precision

asked 2011-05-29T09:22:19.267

19

votes

5

answers

Why do I need 17 significant digits (and not 16) to represent a double?

c floating-point precision ieee-754 floating-accuracy

asked 2011-05-25T00:11:12.563

90

votes

9

answers

Is there any way to get current time in nanoseconds using JavaScript?

javascript time precision time-precision

asked 2011-05-14T15:29:57.770

20

votes

4

answers

Dealing with very small numbers in R

r precision underflow

asked 2011-04-27T10:26:51.563

18

votes

5

answers

losing precision converting from java BigDecimal to double

java floating-point double precision bigdecimal

asked 2011-04-21T20:45:07.690

14

votes

3

answers

MATLAB precision

matlab floating-point precision

asked 2011-04-06T15:29:02.960

9

votes

4

answers

SQL Server truncates decimal points of a newly created field in a view

sql-server t-sql precision

asked 2011-03-22T00:24:19.957

34

votes

5

answers

How can I make sure a float will always be rounded up with PHP?

php floating-point rounding precision ceil

asked 2011-02-23T16:18:13.787

523

votes

14

answers

JavaScript displaying a float to 2 decimal places

javascript floating-point precision

asked 2010-07-02T03:21:24.700

9

votes

3

answers

32-bit versus 64-bit floating-point performance

performance floating-point precision

asked 2010-06-29T01:40:38.620

19

votes

3

answers

Why does a C# System.Decimal remember trailing zeros?

c# decimal precision

asked 2010-06-08T11:17:02.827

10

votes

2

answers

double minus double giving precision problems

c# double precision

asked 2010-04-30T02:09:41.467

26

votes

5

answers

C# float to decimal conversion

c# floating-point precision

asked 2010-04-07T18:12:13.370

160

votes

4

answers

Controlling number of decimal digits in print output in R

r precision digits r-faq output-formatting

asked 2010-02-18T09:48:37.263

10

votes

3

answers

Why is this bearing calculation so inacurate?

java gis precision double-precision

asked 2010-02-09T22:11:46.923

14

votes

5

answers

Java BigDecimal trigonometric methods

java precision trigonometry bigdecimal

asked 2010-01-31T21:50:59.993

63

votes

5

answers

Set the display precision of a float in Ruby

ruby floating-point precision

asked 2009-12-19T19:49:51.543

11

votes

3

answers

Limit the precision on std::cout of default values in boost::options_description

c++ precision cout boost-program-options

asked 2009-11-14T17:22:10.400

1044

votes

7

answers

decimal vs double! - Which one should I use and when?

c# double decimal precision currency

asked 2009-07-22T14:37:54.860

11

votes

3

answers

Is there any way to get the repeating decimal section of a fraction in Python?

python decimal precision

asked 2009-07-12T21:35:10.737

91

votes

8

answers

What range of numbers can be represented in 16-, 32-, and 64-bit IEEE-754 systems?

floating-point precision numeric ieee-754

asked 2009-05-16T14:37:41.893

211

votes

11

answers

What's the difference between a single precision and double precision floating point operation?

floating-point precision processor operations

asked 2009-04-29T06:30:19.177

25

votes

4

answers

Representing integers in doubles

c math floating-point precision ieee-754

asked 2009-04-17T06:00:44.010

75

votes

5

answers

Why is 24.0000 not equal to 24.0000 in MATLAB?

matlab floating-point precision

asked 2009-03-26T16:10:38.863

17

votes

5

answers

"new BigDecimal(13.3D)" results in imprecise "13.3000000000000007105.."?

java precision bigdecimal

asked 2009-01-20T10:36:26.307

14

votes

10

answers

2.9999999999999999 >> .5?

javascript bit-manipulation precision

asked 2008-10-06T02:42:28.497

56

votes

10

answers

Are doubles faster than floats in C#?

c# performance floating-point precision

asked 2008-10-01T17:58:46.337

Questions tagged precision