www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.conv.to!string(float) Is too heavy

reply Hipreme <msnmancini hotmail.com> writes:
I have wrote a simple hello world using std.conv:


```
import std.conv:to;
import std.stdio;

export void MAIN(float b)
{
	writeln("Hello World " ~ to!string(b));
}
```

Building that with dmd app.d -lib creates 500KB lib
With `-g`, it goes up to 800KB

Using ldc, the build time goes up by 2 times.

if you simply change from `float b` to `int b`, the lib goes down 
from 500KB to 60KB.
The build times decreases by almost 10x (must check), but I can 
feel that anyway.

Using std.conv greatly degrades a modularized workflow, by 
increasing a lot of build time and binary size. I needed 
implementing my own to! function to get a much better build time 
for my program.
Oct 12 2021
next sibling parent IGotD- <nise nise.com> writes:
On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.


I've seen similar effects, not only with std.conv but with other 
modules as well. You just use a small part of one module but the 
compiler imports a lot of code.

It would be very interesting to know what happens underneath and 
how we can reduce the code bloat.

We should investigate what happens and we can use your example as 
case study.
Oct 12 2021
prev sibling next sibling parent reply user1234 <user1234 12.de> writes:
On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.


You compare apples and oranges. Convertion from integer types to 
string is trivial, converting float types to string is not.

That being said there's some room for improvment. `writeln` of a 
`float` will take the same path as `format(%f, v)`, which 
contains the code to format float in every possible way, as the 
specifier is not known at compile time, and finally only a small 
part of the bloat generated is executed.

This is bit strange, as for writeln, the specifier is known at 
compile time.


 The build times decreases by almost 10x (must check), but I can 
 feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.
Oct 12 2021
parent reply Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 12 October 2021 at 16:04:49 UTC, user1234 wrote:

 On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.


 You compare apples and oranges. Convertion from integer types 
 to string is trivial, converting float types to string is not.

 That being said there's some room for improvment. `writeln` of 
 a `float` will take the same path as `format(%f, v)`, which 
 contains the code to format float in every possible way, as the 
 specifier is not known at compile time, and finally only a 
 small part of the bloat generated is executed.

 This is bit strange, as for writeln, the specifier is known at 
 compile time.


 The build times decreases by almost 10x (must check), but I 
 can feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.






Lot of people has implemented float to string conversion without 
requiring 500KB.
I really don't think I need all that 'runtime speed' nor 'ultra 
precision', it would be great if we had some compiler flag to 
stop this template bloating thing for we can choose for build 
speed instead.

Yea, writeln does cause 500KB requirement for `floats` and 
`struct`s too (without float members). I guess it is much more 
template hell than any other thing.

Yea, I forced it for not inlining my number, because, that's the 
point, most of the time you won't be inlining a number.

I have like, more than 8 modules, some depending on each other, 
this build time can increase a LOT linking time and the compile 
time. Usually, the modularized workflow should make things faster 
and reuse already compiled code. If the stdlib itself weights so 
much, the modularized workflow will start to feel useless
Oct 12 2021
parent reply IGotD- <nise nise.com> writes:
On Tuesday, 12 October 2021 at 18:06:53 UTC, Hipreme wrote:

 Lot of people has implemented float to string conversion 
 without requiring 500KB.


Floats to string is of course more complicated that integers but 
500KB is ridiculous. It should rather be something like 10-20 KB.

It's likely it is a combination of that the optimizer/linker is 
unable to remove unused functions. Also that imports usually 
imports more stuff, even if it is just a tiny function from 
another import.

It's like a box of cables, you just want one cable but when you 
pull out one cable you get the entire ball of cables.
Oct 12 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Oct 12, 2021 at 07:08:44PM +0000, IGotD- via Digitalmars-d wrote:

 On Tuesday, 12 October 2021 at 18:06:53 UTC, Hipreme wrote:

 
 Lot of people has implemented float to string conversion without
 requiring 500KB.


 
 Floats to string is of course more complicated that integers but 500KB
 is ridiculous. It should rather be something like 10-20 KB.
 
 It's likely it is a combination of that the optimizer/linker is unable
 to remove unused functions. Also that imports usually imports more
 stuff, even if it is just a tiny function from another import.
 
 It's like a box of cables, you just want one cable but when you pull
 out one cable you get the entire ball of cables.


It'd be nice to untangle that ball of cables so that less cables get
pulled out when you tug at one (so to speak). :-D  We talked about
pay-as-you-go modularization of Phobos at least a few times, but so far,
that ball of cables persist.


T

-- 
Never ascribe to malice that which is adequately explained by incompetence. --
Napoleon Bonaparte
Oct 12 2021
parent reply Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 12 October 2021 at 19:50:51 UTC, H. S. Teoh wrote:

 On Tue, Oct 12, 2021 at 07:08:44PM +0000, IGotD- via 
 Digitalmars-d wrote:

 On Tuesday, 12 October 2021 at 18:06:53 UTC, Hipreme wrote:

 
 Lot of people has implemented float to string conversion 
 without requiring 500KB.


 
 Floats to string is of course more complicated that integers 
 but 500KB is ridiculous. It should rather be something like 
 10-20 KB.
 
 It's likely it is a combination of that the optimizer/linker 
 is unable to remove unused functions. Also that imports 
 usually imports more stuff, even if it is just a tiny function 
 from another import.
 
 It's like a box of cables, you just want one cable but when 
 you pull out one cable you get the entire ball of cables.


 It'd be nice to untangle that ball of cables so that less 
 cables get pulled out when you tug at one (so to speak). :-D  
 We talked about pay-as-you-go modularization of Phobos at least 
 a few times, but so far, that ball of cables persist.


 T


I do hope that this gets noted by the decision people, having to 
reimplement stdlib is the last thing I wanted right now, but 
500KB for string to float conversion per module, (without debug 
symbols ) is pretty unacceptable.
Oct 12 2021
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Oct 12, 2021 at 10:15:10PM +0000, Hipreme via Digitalmars-d wrote:
[...]
[...]

 I do hope that this gets noted by the decision people, having to
 reimplement stdlib is the last thing I wanted right now, but 500KB for
 string to float conversion per module, (without debug symbols ) is
 pretty unacceptable.


[...]

*Per module*?? That shouldn't be the case.  You must be doing something
wrong...  Are you using dmd or ldc?  You might want to look into various
options for eliminating duplicate template instantiations, like
-linkonce-templates, or LTO, or some such.  No matter how bad the
string-to-float bloat is (and I agree it's bad), it should not be adding
bloat *per module*.

Possibly one thing to watch out for is using compile-time format
strings, i.e., `writeln!"abc %s def"(...)` instead of runtime format
strings `writeln("abc %s def", ...)`.  The former has its uses, such as
catching bad format strings at compile-time; but it may potentially be
the cause of a lot of template bloat if you have a lot of different
format strings.


T

-- 
Being able to learn is a great learning; being able to unlearn is a greater
learning.
Oct 12 2021
parent reply Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 12 October 2021 at 22:58:51 UTC, H. S. Teoh wrote:

 On Tue, Oct 12, 2021 at 10:15:10PM +0000, Hipreme via 
 Digitalmars-d wrote:
 [...]
 [...]

 I do hope that this gets noted by the decision people, having 
 to reimplement stdlib is the last thing I wanted right now, 
 but 500KB for string to float conversion per module, (without 
 debug symbols ) is pretty unacceptable.


 [...]

 *Per module*?? That shouldn't be the case.  You must be doing 
 something wrong...  Are you using dmd or ldc?  You might want 
 to look into various options for eliminating duplicate template 
 instantiations, like -linkonce-templates, or LTO, or some such. 
  No matter how bad the string-to-float bloat is (and I agree 
 it's bad), it should not be adding bloat *per module*.

 Possibly one thing to watch out for is using compile-time 
 format strings, i.e., `writeln!"abc %s def"(...)` instead of 
 runtime format strings `writeln("abc %s def", ...)`.  The 
 former has its uses, such as catching bad format strings at 
 compile-time; but it may potentially be the cause of a lot of 
 template bloat if you have a lot of different format strings.


 T



Humm I say module for representing libs, I have splitted my 
project in various libs, whose I call modules, so, it needs to 
link to each lib I'm using. Although I have implemented most I 
need right now, the current problem is std.array.

std.array increases 500Kb by only including it, so, for keeping 
my build times okay I have been basically ditching phobos...
Oct 13 2021
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 14, 2021 at 12:04:52AM +0000, Hipreme via Digitalmars-d wrote:
[...]

 Humm I say module for representing libs, I have splitted my project in
 various libs, whose I call modules, so, it needs to link to each lib
 I'm using.


Hmm, I'd have thought LTO or -linkonce-templates ought to drop the
redundant templates once you link the libs.  Is there any reason why it
isn't?



 Although I have implemented most I need right now, the current problem
 is std.array.
 
 std.array increases 500Kb by only including it, so, for keeping my
 build times okay I have been basically ditching phobos...


If I were you I'd report a bug.  The mere act of importing std.array
shouldn't pull in a whole bunch of dead weight if you don't even use
anything.  There used to be more Phobos modules with this problem, but a
while back there was some effort to fix this.  If std.array still has
this problem it should get looked into.


T

-- 
Once the bikeshed is up for painting, the rainbow won't suffice. -- Andrei
Alexandrescu
Oct 13 2021
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.
 The build times decreases by almost 10x (must check), but I can 
 feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.


`writeln` itself even for plain strings is already quite heavy.
I'd recommend using printf.

I also have a realtively lightweight `dtoa` function.
it's here:
https://raw.githubusercontent.com/UplinkCoder/fpconv/master/src/fpconv_ctfe.d
Oct 12 2021
parent Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 12 October 2021 at 16:05:19 UTC, Stefan Koch wrote:

 On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.
 The build times decreases by almost 10x (must check), but I 
 can feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.


 `writeln` itself even for plain strings is already quite heavy.
 I'd recommend using printf.

 I also have a realtively lightweight `dtoa` function.
 it's here:
 https://raw.githubusercontent.com/UplinkCoder/fpconv/master/src/fpconv_ctfe.d


I'm not defending writeln (specially because in my code it causes 
1500KB increase, but I could not reproduce in a simple hello 
world that increase), but the problem seems that std.conv for 
floats seems to consistently do that
Oct 12 2021
prev sibling parent reply russhy <russhy gmail.com> writes:
On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.
 The build times decreases by almost 10x (must check), but I can 
 feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.



try to build with:

```json
"dflags-ldc": [
     "-linkonce-templates",
],
```


i'd personally suggest using snprintf, but depending on your 
usecase that might not be perfect

```d
char[32] tmp = 0;
snprintf(tmp.ptr, tmp.length, "Hello World %f", value);
```
Oct 12 2021
next sibling parent Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 12 October 2021 at 17:35:31 UTC, russhy wrote:

 On Tuesday, 12 October 2021 at 14:47:21 UTC, Hipreme wrote:

 I have wrote a simple hello world using std.conv:


 ```
 import std.conv:to;
 import std.stdio;

 export void MAIN(float b)
 {
 	writeln("Hello World " ~ to!string(b));
 }
 ```

 Building that with dmd app.d -lib creates 500KB lib
 With `-g`, it goes up to 800KB

 Using ldc, the build time goes up by 2 times.

 if you simply change from `float b` to `int b`, the lib goes 
 down from 500KB to 60KB.
 The build times decreases by almost 10x (must check), but I 
 can feel that anyway.

 Using std.conv greatly degrades a modularized workflow, by 
 increasing a lot of build time and binary size. I needed 
 implementing my own to! function to get a much better build 
 time for my program.



 try to build with:

 ```json
 "dflags-ldc": [
     "-linkonce-templates",
 ],
 ```


 i'd personally suggest using snprintf, but depending on your 
 usecase that might not be perfect

 ```d
 char[32] tmp = 0;
 snprintf(tmp.ptr, tmp.length, "Hello World %f", value);
 ```



snprintf can't be called at compile time, so, it can miss a lot.
That flag didn't seem to do something
Oct 12 2021
prev sibling parent Ogi <ogion.art gmail.com> writes:
On Tuesday, 12 October 2021 at 17:35:31 UTC, russhy wrote:

 i'd personally suggest using snprintf, but depending on your 
 usecase that might not be perfect

 ```d
 char[32] tmp = 0;
 snprintf(tmp.ptr, tmp.length, "Hello World %f", value);
 ```


Be aware of [the bug with 
`snprintf`](https://issues.dlang.org/show_bug.cgi?id=22266) when 
compiling for Windows.
Oct 14 2021