Floating Point Rounding Issues

Perl

Consider the following program:

$x=12345678901.234567000;
$y=($x-int($x))*1000000000;
printf("%f->%f\n",$x,$y);

Here's what it prints:

12345678901.234568->234567642.211914

I was expecting:

12345678901.234567->234567000

This appears to be some sort of rounding issue in Perl. Adding a new package seems to help:

use bignum;
$x=12345678901.234567000;
$y=($x-int($x))*1000000000;
printf("%f->%f\n",$x,$y);
print $x;

prints:

12345678901.234568->234567000
12345678901.234567

Notice how the second line is correct. The variable is fine, but printf is having some issues.

Note that if the variable is being converted from a string there are still issues. Consider the following:

use bignum;
$x="12345678901->234567000";
$y=($x-int($x))*1000000000;
printf("%f:%f\n",$x,$y);

prints:

12345678901.234568->234567642.211914

So use +0 to cast to a number:

$x="12345678901.234567000";
$x+=0;#Cast to a number
$y=($x-int($x))*1000000000;
printf("%f->%f\n",$x,$y);

prints:

12345678901.234568->234567000.000000

MySQL

The same rounding issue also appears in MySQL. Consider the following:

SELECT ('12345678901.234567'-FLOOR('12345678901.234567'))*1000000000;

prints:

234567642.211914

Take off the single quotes and all is well:

SELECT (12345678901.234567-FLOOR(12345678901.234567))*1000000000;

prints:

234567000.000000

It appears that the string to number conversion is broken in MySQL, so you have to define it explicitly. Unfortunately, you can't just cast to a double:

SELECT (CAST('12345678901.234567' as double)-FLOOR(CAST('12345678901.234567' as double)))*1000000000;

prints:

SQL Error (1064): You have an error in your SQL syntax;

So, it's necessary to use a decimal, but that fails too:

SELECT (CAST('12345678901.234567' as decimal)-FLOOR(CAST('12345678901.234567' as decimal)))*1000000000;

prints:

SQL Error (1064): You have an error in your SQL syntax;

Nice. Well, I guess I need to specify the width explicitly.

SELECT (CAST('12345678901.234567' as decimal(65,30))-FLOOR(CAST('12345678901.234567' as decimal(65,30))))*1000000000;

prints:

234567000.000000000000000000000000000000

PostgreSQL

The same thing happens in PostgreSQL:

SELECT ('12345678901.234567'-FLOOR('12345678901.234567'))*1000000000;

prints:

234567642.211914

And with float casts:

SELECT ('12345678901.234567'::float-FLOOR('12345678901.234567'::float))*1000000000;

prints:

234567642.211914

But, decimal works:

SELECT ('12345678901.234567'::decimal-FLOOR('12345678901.234567'::decimal))*1000000000;

234567000

No votes yet