• Martin Nowak (18/18) Sep 26 2013 Running dmd with huge page backed malloc resulted in about 10%
• Jonathan M Davis (5/8) Sep 26 2013 IIRC, Leandro made his concurrent GC use environment variables to set st...
• Martin Nowak (5/8) Oct 01 2013 Speaking about the concurrent GC, huge pages incur a huge penalty for
• luminousone (8/26) Sep 26 2013 I have played around with huge pages a bit, my understanding from
• Martin Nowak (7/13) Oct 01 2013 The main benefit should come from heavily reducing the amount of TLB
• Rainer Schuetze (10/27) Sep 27 2013 An application that needs to allocate ore than a few MB is currently
• Martin Nowak (3/13) Oct 01 2013 Env variables make sense.

"Martin Nowak" <code dawg.eu> writes:

Running dmd with huge page backed malloc resulted in about 10% 
faster compilations.
This looks very promising though other tests with some D GC 
microbenchmarks did not benefit from huge pages.
You can read more about this here.
http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

I'm interested in adding a runtime parameter that would advise 
the GC to preferably use huge pages as backing memory.
There are a number of other interesting runtime parameters, like 
max heap size, GC grow policies, turning on statistics, so I 
wonders how to make them available.
Java seems to use -XX:+UseLargePages and -Xmx for the maximum 
heap size.
GHC uses a nice scheme where runtime parameters passed via 
command line are embraced by +RTS arg1 arg2 -RTS. They also 
require to link-in support for this.
http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
Any good ideas for druntime?

"Jonathan M Davis" <jmdavisProg gmx.com> writes:

On Friday, September 27, 2013 02:31:21 Martin Nowak wrote:
 I'm interested in adding a runtime parameter that would advise
 the GC to preferably use huge pages as backing memory.

[snip]
 Any good ideas for druntime?

IIRC, Leandro made his concurrent GC use environment variables to set stuff 
like this.

- Jonathan M Davis

Martin Nowak <code dawg.eu> writes:

On 09/27/2013 02:54 AM, Jonathan M Davis wrote:
 IIRC, Leandro made his concurrent GC use environment variables to set stuff
 like this.

 - Jonathan M Davis

Speaking about the concurrent GC, huge pages incur a huge penalty for 
COW because the virtual memory granularity goes from 4Kb to 2Mb, i.e. 
500x more copying when the parent process modifies a page while the 
forked process performs GC.

"luminousone" <rd.hunt gmail.com> writes:

On Friday, 27 September 2013 at 00:31:23 UTC, Martin Nowak wrote:
 Running dmd with huge page backed malloc resulted in about 10% 
 faster compilations.
 This looks very promising though other tests with some D GC 
 microbenchmarks did not benefit from huge pages.
 You can read more about this here.
 http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

 I'm interested in adding a runtime parameter that would advise 
 the GC to preferably use huge pages as backing memory.
 There are a number of other interesting runtime parameters, 
 like max heap size, GC grow policies, turning on statistics, so 
 I wonders how to make them available.
 Java seems to use -XX:+UseLargePages and -Xmx for the maximum 
 heap size.
 GHC uses a nice scheme where runtime parameters passed via 
 command line are embraced by +RTS arg1 arg2 -RTS. They also 
 require to link-in support for this.
 http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
 Any good ideas for druntime?

I have played around with huge pages a bit, my understanding from 
reading about them, and use(ROAM diamond tree nodes in huge page 
memory), is that they need to be used carefully, many processors 
only have a small number of entries for huge pages in their 
lookup tables.

The best use of huge pages seems to be from code built around 
them, not sure if it entirely makes sense to put them in gc,

Martin Nowak <code dawg.eu> writes:

On 09/27/2013 04:01 AM, luminousone wrote:
 I have played around with huge pages a bit, my understanding from
 reading about them, and use(ROAM diamond tree nodes in huge page
 memory), is that they need to be used carefully, many processors only
 have a small number of entries for huge pages in their lookup tables.

When out of huge pages it would fallback to normal mmap.

 The best use of huge pages seems to be from code built around them, not
 sure if it entirely makes sense to put them in gc,

The main benefit should come from heavily reducing the amount of TLB 
entries in the cache.
So if your program suffers from many cache misses due to TLB lookups
that would help. And the freed cache can also be used for other stuff 
instead.

Rainer Schuetze <r.sagitario gmx.de> writes:

On 27.09.2013 02:31, Martin Nowak wrote:
 Running dmd with huge page backed malloc resulted in about 10% faster
 compilations.
 This looks very promising though other tests with some D GC
 microbenchmarks did not benefit from huge pages.
 You can read more about this here.
 http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

 I'm interested in adding a runtime parameter that would advise the GC to
 preferably use huge pages as backing memory.
 There are a number of other interesting runtime parameters, like max
 heap size, GC grow policies, turning on statistics, so I wonders how to
 make them available.

An application that needs to allocate ore than a few MB is currently 
often slowed down considerably by early collections of the GC. Being 
able to use larger values for initial memory and pool size increments 
would help a lot here.

 Java seems to use -XX:+UseLargePages and -Xmx for the maximum heap size.
 GHC uses a nice scheme where runtime parameters passed via command line
 are embraced by +RTS arg1 arg2 -RTS. They also require to link-in
 support for this.
 http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
 Any good ideas for druntime?

For the precise GC I moved the configuration variable(s) to its own 
module: https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d
This module allows configuration via environment variables, but you can 
change the defaults by adding a file with module declaration "gc.config" 
and other settings.

Martin Nowak <code dawg.eu> writes:

On 09/27/2013 03:33 PM, Rainer Schuetze wrote:
 An application that needs to allocate ore than a few MB is currently
 often slowed down considerably by early collections of the GC. Being
 able to use larger values for initial memory and pool size increments
 would help a lot here.

Yep, that's the main use case.

 For the precise GC I moved the configuration variable(s) to its own
 module:
 https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d
 This module allows configuration via environment variables, but you can
 change the defaults by adding a file with module declaration "gc.config"
 and other settings.

Env variables make sense.