www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Getting # Physical CPUs

reply dsimcha <dsimcha yahoo.com> writes:
When I do parallel programming in D, it's nice to have a sane default for the
number of threads.  The ideal number is the number of cores that can see the
current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
to provide nPhysialCPUs or something similar, which would allow the total
number of physical CPUs in the system to be determined.  Is there any easy way
to determine this in D2?
Jul 14 2010
next sibling parent reply Don <nospam nospam.com> writes:
dsimcha wrote:
 When I do parallel programming in D, it's nice to have a sane default for the
 number of threads.  The ideal number is the number of cores that can see the
 current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
 to provide nPhysialCPUs or something similar, which would allow the total
 number of physical CPUs in the system to be determined.  Is there any easy way
 to determine this in D2?
coresPerCPU is a misnomer. It's actually the total number of cores. Will fix.
Jul 14 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 When I do parallel programming in D, it's nice to have a sane default for the
 number of threads.  The ideal number is the number of cores that can see the
 current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
 to provide nPhysialCPUs or something similar, which would allow the total
 number of physical CPUs in the system to be determined.  Is there any easy way
 to determine this in D2?
coresPerCPU is a misnomer. It's actually the total number of cores. Will fix.
Ok, then maybe this is a bug and belongs in bugzilla. Here's a little test program I wrote: import core.cpuid, std.stdio; void main() { writeln("Cores: ", coresPerCPU, " Threads: ", threadsPerCPU); } This prints the following on the machine in question: Cores: 1 Threads: 2 Here's a cat/proc/cpuinfo on the same machine: processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7321.79 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 4 siblings : 2 core id : 4 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7314.82 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 3 siblings : 2 core id : 3 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.15 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 7 siblings : 2 core id : 7 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.01 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 4 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.01 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 5 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 4 siblings : 2 core id : 4 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.07 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 6 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 3 siblings : 2 core id : 3 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.13 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management: processor : 7 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.825 cache size : 1024 KB physical id : 7 siblings : 2 core id : 7 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cid cx16 xtpr bogomips : 7315.03 clflush size : 64 cache_alignment : 128 address sizes : 40 bits physical, 48 bits virtual power management:
Jul 14 2010
parent reply Don <nospam nospam.com> writes:
dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 When I do parallel programming in D, it's nice to have a sane default for the
 number of threads.  The ideal number is the number of cores that can see the
 current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
 to provide nPhysialCPUs or something similar, which would allow the total
 number of physical CPUs in the system to be determined.  Is there any easy way
 to determine this in D2?
coresPerCPU is a misnomer. It's actually the total number of cores. Will fix.
Ok, then maybe this is a bug and belongs in bugzilla. Here's a little test program I wrote:
[snip] Thanks, that's definitely a bug. The code in core.cpuid has not been tested on the most recent CPUs (Intel added a totally new method) and their documentation is quite convoluted. It's hard to get it right without an actual machine.
Jul 14 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Don (nospam nospam.com)'s article
 [snip]
 Thanks, that's definitely a bug. The code in core.cpuid has not been
 tested on the most recent CPUs (Intel added a totally new method) and
 their documentation is quite convoluted. It's hard to get it right
 without an actual machine.
Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Jul 14 2010
parent reply Don <nospam nospam.com> writes:
dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 [snip]
 Thanks, that's definitely a bug. The code in core.cpuid has not been
 tested on the most recent CPUs (Intel added a totally new method) and
 their documentation is quite convoluted. It's hard to get it right
 without an actual machine.
Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Please check if the latest druntime commit fixes this.
Jul 15 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 [snip]
 Thanks, that's definitely a bug. The code in core.cpuid has not been
 tested on the most recent CPUs (Intel added a totally new method) and
 their documentation is quite convoluted. It's hard to get it right
 without an actual machine.
Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Please check if the latest druntime commit fixes this.
Thanks. Unfortunately I can't test this because the Linux build script for Phobos is broken on my machine in some inscrutable way. Frankly, my success rate at building stuff from other people's make files is well under 50%. Make is just a horrible technology that needs to die a horrible death. We should be eating our own dogfood and using rdmd for build scripts. Here's the error message I'm getting. I know basically nothing about make except that it's a build system and that it almost never works, so I can't even begin to debug this. Here's the error message I've been getting, on a freshly unpacked 2.047 directory on some ancient Linux distro that my sys admin insists on using: $ make -flinux.mak make --no-print-directory -f OS=posix BUILD=release make[1]: OS=posix: No such file or directory make[1]: *** No rule to make target `OS=posix'. Stop. make: *** [release] Error 2 And here's the error I get when I try on a different machine w/ a more modern distro (this one is probably due to lack of 64 bit libs): $ make -flinux.mak make --no-print-directory -f linux.mak OS=posix BUILD=release cc -c -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o cc -c -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o cc -c -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o In file included from /usr/include/features.h:378, from /usr/include/string.h:26, from etc/c/zlib/zutil.h:23, from etc/c/zlib/crc32.c:29: /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1 make: *** [release] Error 2
Jul 15 2010
next sibling parent reply Don <nospam nospam.com> writes:
dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 [snip]
 Thanks, that's definitely a bug. The code in core.cpuid has not been
 tested on the most recent CPUs (Intel added a totally new method) and
 their documentation is quite convoluted. It's hard to get it right
 without an actual machine.
Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Please check if the latest druntime commit fixes this.
Thanks. Unfortunately I can't test this because the Linux build script for Phobos is broken on my machine in some inscrutable way. Frankly, my success rate at building stuff from other people's make files is well under 50%. Make is just a horrible technology that needs to die a horrible death. We should be eating our own dogfood and using rdmd for build scripts.
I agree. In this case, core.cpuid is completely stand-alone. So you could just copy it into another directory and change the module statement.
Jul 15 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 dsimcha wrote:
 == Quote from Don (nospam nospam.com)'s article
 [snip]
 Thanks, that's definitely a bug. The code in core.cpuid has not been
 tested on the most recent CPUs (Intel added a totally new method) and
 their documentation is quite convoluted. It's hard to get it right
 without an actual machine.
Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Please check if the latest druntime commit fixes this.
Thanks. Unfortunately I can't test this because the Linux build script for Phobos is broken on my machine in some inscrutable way. Frankly, my success rate at building stuff from other people's make files is well under 50%. Make is just a horrible technology that needs to die a horrible death. We should be eating our own dogfood and using rdmd for build scripts.
I agree. In this case, core.cpuid is completely stand-alone. So you could just copy it into another directory and change the module statement.
Great idea. Unfortunately that still doesn't fix it. I get different wrong information on some machines, but it's still wrong, and the specific example I posted to Bugzilla hasn't changed at all.
Jul 15 2010
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
As a side note, why is there both a std.cpuid and a core.cpuid?

Does std use core? In that case, why not import std.cpuid?
Jul 15 2010
parent Don <nospam nospam.com> writes:
Philippe Sigaud wrote:
 As a side note, why is there both a std.cpuid and a core.cpuid?
 
 Does std use core? 
No. In that case, why not import std.cpuid?
 
 
std.cpuid is deprecated.
Jul 15 2010
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
dsimcha wrote:
 Here's the error message I'm getting.  I know basically nothing about make
except
 that it's a build system and that it almost never works, so I can't even begin
to
 debug this.  Here's the error message I've been getting, on a freshly unpacked
 2.047 directory on some ancient  Linux distro that my sys admin insists on
using:
 
 $ make -flinux.mak
 make --no-print-directory -f  OS=posix BUILD=release
 make[1]: OS=posix: No such file or directory
 make[1]: *** No rule to make target `OS=posix'.  Stop.
 make: *** [release] Error 2
The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error.
 And here's the error I get when I try on a different machine w/ a more modern
 distro (this one is probably due to lack of 64 bit libs):
 
 $ make -flinux.mak
 make --no-print-directory -f linux.mak OS=posix BUILD=release
 cc -c  -m32 -O3 etc/c/zlib/adler32.c
-ogenerated/posix/release/etc/c/zlib/adler32.o
 cc -c  -m32 -O3 etc/c/zlib/compress.c
-ogenerated/posix/release/etc/c/zlib/compress.o
 cc -c  -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o
 In file included from /usr/include/features.h:378,
                  from /usr/include/string.h:26,
                  from etc/c/zlib/zutil.h:23,
                  from etc/c/zlib/crc32.c:29:
 /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
 make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1
 make: *** [release] Error 2
This is most likely because you have not gotten the 32 bit dev system installed on your 64 bit system (it usually is not installed by the default linux install). Which one do you have? BTW, using a script rather than make wouldn't have helped you with the second issue.
Jul 15 2010
next sibling parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound2 digitalmars.com)'s article
 And here's the error I get when I try on a different machine w/ a more modern
 distro (this one is probably due to lack of 64 bit libs):

 $ make -flinux.mak
 make --no-print-directory -f linux.mak OS=posix BUILD=release
 cc -c  -m32 -O3 etc/c/zlib/adler32.c
-ogenerated/posix/release/etc/c/zlib/adler32.o
 cc -c  -m32 -O3 etc/c/zlib/compress.c
-ogenerated/posix/release/etc/c/zlib/compress.o
 cc -c  -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o
 In file included from /usr/include/features.h:378,
                  from /usr/include/string.h:26,
                  from etc/c/zlib/zutil.h:23,
                  from etc/c/zlib/crc32.c:29:
 /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
 make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1
 make: *** [release] Error 2
This is most likely because you have not gotten the 32 bit dev system installed on your 64 bit system (it usually is not installed by the default linux install). Which one do you have? BTW, using a script rather than make wouldn't have helped you with the second issue.
Yeah, in my original most I meant to say it was probably due to lack of 32-bit libs, which I don't have root privileges to install on the machine in question. It can see the same file systems via NFS as some machines that do have 32-bit libs installed, so my usual kludge is to always use a machine w/ 32-bit libs for building. Even so, I only figured this out b/c I was previously aware of the problem. The point isn't that make failed here, it's that it failed with an absolutely inscrutable error message. I wouldn't have even been able to begin guessing what was wrong if I didn't already know about the 32-bit lib issue.
Jul 15 2010
prev sibling next sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
Walter Bright, el 15 de julio a las 11:40 me escribiste:
 dsimcha wrote:
Here's the error message I'm getting.  I know basically nothing about make
except
that it's a build system and that it almost never works, so I can't even begin
to
debug this.  Here's the error message I've been getting, on a freshly unpacked
2.047 directory on some ancient  Linux distro that my sys admin insists on
using:

$ make -flinux.mak
make --no-print-directory -f  OS=posix BUILD=release
make[1]: OS=posix: No such file or directory
make[1]: *** No rule to make target `OS=posix'.  Stop.
make: *** [release] Error 2
The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error.
This is even standard POSIX: http://www.opengroup.org/onlinepubs/009695399/utilities/make.html (see the OPERANDS section) -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- - Que hacés, ratita? - Espero un ratito...
Jul 15 2010
prev sibling next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Walter Bright Wrote:

 
 $ make -flinux.mak
 make --no-print-directory -f  OS=posix BUILD=release
 make[1]: OS=posix: No such file or directory
 make[1]: *** No rule to make target `OS=posix'.  Stop.
 make: *** [release] Error 2
The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error.
Looks to me like a macro isn't being set. In the first output line it has the argument '-f ' there are two spaces after requesting to look in a file. So the file name it sees is OS=posix
Jul 15 2010
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 15 Jul 2010 14:40:57 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 dsimcha wrote:
 Here's the error message I'm getting.  I know basically nothing about  
 make except
 that it's a build system and that it almost never works, so I can't  
 even begin to
 debug this.  Here's the error message I've been getting, on a freshly  
 unpacked
 2.047 directory on some ancient  Linux distro that my sys admin insists  
 on using:
  $ make -flinux.mak
 make --no-print-directory -f  OS=posix BUILD=release
 make[1]: OS=posix: No such file or directory
 make[1]: *** No rule to make target `OS=posix'.  Stop.
 make: *** [release] Error 2
The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error.
The issue is the -f is passed a blank argument. Probably in linux.mak, it's calling something like this: make --no-print-directory -f $(SUB_MAKE_FILE) OS=$(OS) BUILD=release or something like that, and SUB_MAKE_FILE isn't being set for some reason. I think make is interpreting that OS=posix as a makefile to use. -Steve
Jul 16 2010
prev sibling parent reply eris <jvburnes gmail.com> writes:
dsimcha Wrote:

 When I do parallel programming in D, it's nice to have a sane default for the
 number of threads.  The ideal number is the number of cores that can see the
 current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
 to provide nPhysialCPUs or something similar, which would allow the total
 number of physical CPUs in the system to be determined.  Is there any easy way
 to determine this in D2?
dsimcha, This is a relatively difficult problem in general to do portably due to hardware differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner. I came to the conclusion that the people most motivated to keep up to date on a portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc. The HWLoc C libraries are written by the open-mpi project here: http://www.open-mpi.org/projects/hwloc/ I created an interface to their C library called HWLoc.d and I haven't published it anywhere. I just needed it for my multi-core Dendrite system. Give me an email you want it sent to and it's yours. I could post it here, but it would be a lot of noise in the discussion. I'll probably post it on dsource later. My license on all of my Dendrite code is zlib (which means you can do almost anything except claim that you wrote the original) and the open-mpi C libraries are new BSD, so have at it. You'll obviously have to download and install the hwloc c libraries also. Here's some output from "hwhello.d" on a 2 socket system: *** Objects at level 0 CPU Set: 00000003 Index 0: System(1002MB) *** Objects at level 1 CPU Set: 00000001 CPU Set: 00000002 *** Objects at level 2 CPU Set: 00000001 Index 0: L2(2048KB) CPU Set: 00000002 Index 1: L2(2048KB) *** Objects at level 3 CPU Set: 00000001 Index 0: L1(16KB) CPU Set: 00000002 Index 1: L1(16KB) *** Objects at level 4 CPU Set: 00000001 CPU Set: 00000002 *** Objects at level 5 CPU Set: 00000001 CPU Set: 00000002 *** Printing overall tree System(1002MB) L2(2048KB) L1(16KB) L2(2048KB) L1(16KB) *** 2 socket(s) depth is now: 4 number objects at this depth: 2 ob 0: 00000001 ob 0: 00000002 Last CPU Set: 00000002 Singleified CPU Set: 00000002 burning CPU ... and here's the code snippet from hwloc.d that generates the last part of that output: /* Get last level. */ obj = hwloc_get_obj_by_depth(topology, depth, hwloc_get_nbobjs_by_depth(topology, depth) - 1); if (obj) { /* Get a copy of its cpuset that we may modify. */ cpuset = hwloc_cpuset_dup(obj.cpuset); // convert it to a string nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset); Stdout("Last CPU Set: ")(string[0..nchars]).newline; /* Get only one logical processor (in case the core is SMT/hyperthreaded). */ hwloc_cpuset_singlify(cpuset); nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset); Stdout("Singleified CPU Set: ")(string[0..nchars]).newline; /* And try to bind ourself there. */ if (hwloc_set_cpubind(topology, cpuset, 0)) { char* str; nchars = hwloc_cpuset_asprintf(&str, obj.cpuset); Stdout.formatln("Couldn't bind to cpuset {}", str[0..nchars]); free(str); } Stdout("burning CPU").newline; /* do something we can see in htop */ for (j=0; j<100000000; j++) f+=f/2; /* Free our cpuset copy */ hwloc_cpuset_free(cpuset); } .... finally here's the output from the 'lstopo' utility in the hwloc distro that displays the topology visually (mine is a 4 socket, hexacore NUMA machine. Use a fixed font to view it). draco% Have fun and let me know if you need anything... eris
Jul 14 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from eris (jvburnes gmail.com)'s article
 This is a relatively difficult problem in general to do portably due to
hardware
differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner.
 I came to the conclusion that the people most motivated to keep up to date on a
portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc.
 The HWLoc C libraries are written by the open-mpi project here:
http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency.
Jul 14 2010
parent reply Georg Wrede <Georg.Wrede iki.fi> writes:
On 07/14/2010 08:55 PM, dsimcha wrote:
 == Quote from eris (jvburnes gmail.com)'s article
 This is a relatively difficult problem in general to do portably due to
hardware
differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner.
 I came to the conclusion that the people most motivated to keep up to date on a
portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc.
 The HWLoc C libraries are written by the open-mpi project here:
http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency.
I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources. The day that someone invents the 'unequal cores paradigm', where cores of dissimilar power are included in the same computer, should not expose us with our pants down. (A case in point, at bootup, the Linux core already enumerates and evaluates each found core individually.)
Jul 15 2010
parent reply Don <nospam nospam.com> writes:
Georg Wrede wrote:
 On 07/14/2010 08:55 PM, dsimcha wrote:
 == Quote from eris (jvburnes gmail.com)'s article
 This is a relatively difficult problem in general to do portably due 
 to hardware
differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner.
 I came to the conclusion that the people most motivated to keep up to 
 date on a
portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc.
 The HWLoc C libraries are written by the open-mpi project here:
http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency.
I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources.
 The day that someone invents the 'unequal cores paradigm', where cores 
 of dissimilar power are included in the same computer, should not expose 
 us with our pants down.
It really depends on what the purpose is. If you want to determine the precise core topology, the available information is heavily OS-dependent. Note that there's potentially a large difference between the number of cores in the machine, versus the number of cores which the OS makes available to your app. Generally the second number is the one which matters.
 (A case in point, at bootup, the Linux core already enumerates and 
 evaluates each found core individually.)
Of course it does. It's trivial when you're an OS and have unrestricted access to the machine. An app is severely limited to what it can get from the OS. Currently core.cpuid doesn't make any OS calls at all. I think std.cpuid should be replaced with a new module std.sysinfo, which determines more features (such as available RAM).
Jul 15 2010
parent Georg Wrede <Georg.Wrede iki.fi> writes:
On 07/16/2010 04:48 AM, Don wrote:
 Georg Wrede wrote:
 On 07/14/2010 08:55 PM, dsimcha wrote:
 It would be nice to not have to manually specify such a
 parameter on every run, but not nice enough to be
 worth introducing a dependency.
I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources.
 The day that someone invents the 'unequal cores paradigm', where cores
 of dissimilar power are included in the same computer, should not
 expose us with our pants down.
It really depends on what the purpose is. If you want to determine the precise core topology, the available information is heavily OS-dependent. Note that there's potentially a large difference between the number of cores in the machine, versus the number of cores which the OS makes available to your app. Generally the second number is the one which matters.
True. What my app needs D to tell it, is to enumerate and specify the processors (and of course any other variable resources) available to this particular instance of running program.
 (A case in point, at bootup, the Linux core already enumerates and
 evaluates each found core individually.)
Of course it does. It's trivial when you're an OS and have unrestricted access to the machine. An app is severely limited to what it can get from the OS. Currently core.cpuid doesn't make any OS calls at all. I think std.cpuid should be replaced with a new module std.sysinfo, which determines more features (such as available RAM).
Agreed. Runtime initialization code should find out the resources available to the current app instance. Anything else can (or should) be available from /proc/... (or, whatever equivalents Bill G or Steve J have decided), and the app is free to roam around in that directory tree.
Jul 17 2010