www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Of the use of unpredictableSeed

reply cym13 <cpicard openmailbox.org> writes:
Hi,

I found many times that people use unpredictableSeed in 
combination with normal PRNG for cryptographic purpose. Some even 
go as far as reseeding at each call to try making it more secure.

It is a dangerous practice, most PRNG are not designed with 
security (and unpredictability) in mind, and unpredictableSeed 
was definitely not designed with security in mind (or it failed 
heavily at it). It's a good tool when one needs randomness, not 
security.

I wrote a blog post to present exactly why this is a bad idea and 
how it could be exploited [1].

The best would be to add a standard CSPRNG interface to Phobos 
but we aren't there yet.

[1]: https://cym13.github.io/article/unpredictableSeed.html
Feb 26 2017
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
cym13 wrote:


 Hi,

 I found many times that people use unpredictableSeed in combination 
 with normal PRNG for cryptographic purpose. Some even go as far as 
 reseeding at each call to try making it more secure.

 It is a dangerous practice, most PRNG are not designed with security 
 (and unpredictability) in mind, and unpredictableSeed was definitely 
 not designed with security in mind (or it failed heavily at it). It's 
 a good tool when one needs randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea and how 
 it could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos but we 
 aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


"like /dev/random on Linux"
(sighs) it was so good until this...
Feb 26 2017
parent reply cym13 <cpicard openmailbox.org> writes:
On Sunday, 26 February 2017 at 18:33:08 UTC, ketmar wrote:

 cym13 wrote:


 Hi,

 I found many times that people use unpredictableSeed in 
 combination with normal PRNG for cryptographic purpose. Some 
 even go as far as reseeding at each call to try making it more 
 secure.

 It is a dangerous practice, most PRNG are not designed with 
 security (and unpredictability) in mind, and unpredictableSeed 
 was definitely not designed with security in mind (or it 
 failed heavily at it). It's a good tool when one needs 
 randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea 
 and how it could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos 
 but we aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


 "like /dev/random on Linux"
 (sighs) it was so good until this...


That's a typo actually, I meant urandom, I'll correct it.
Actually it would be better not to use urandom directly but use 
it as source to regularly reseed another PRNG in order to avoid 
some warts but meh. As a first step it's good enough as it is.
Feb 26 2017
parent ketmar <ketmar ketmar.no-ip.org> writes:
cym13 wrote:


 "like /dev/random on Linux"
 (sighs) it was so good until this...


 That's a typo actually, I meant urandom, I'll correct it.


thank you. sorry for me being rough: i was trying to make a joke, and 
i was pretty sure that it was a typo. but now i reread my post and 
found that the joke mysteriously turned into something i didn't meant 
to say.



 Actually it would be better not to use urandom directly but use it as 
 source to regularly reseed another PRNG in order to avoid some warts 
 but meh. As a first step it's good enough as it is.


yeah. yet, urandom is using cryptoprng (salsa now, afair), so it may be 
used as-is too. ah, even good old arc4 is not that bad -- as urandom is 
almost never have a "scratch start" case. and one can throw away 2k-3k 
of values just to be sure anyway. ;-)


ah, i *meant* to say: "good article". so: good article! ;-)
Feb 26 2017
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 02/26/2017 01:23 PM, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in combination with
 normal PRNG for cryptographic purpose. Some even go as far as reseeding
 at each call to try making it more secure.

 It is a dangerous practice, most PRNG are not designed with security
 (and unpredictability) in mind, and unpredictableSeed was definitely not
 designed with security in mind (or it failed heavily at it). It's a good
 tool when one needs randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea and how it
 could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos but we
 aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


FWIW, DAuth[1] uses, and offers, an implementation of Hash_DRBG, a 
well-known and established CSPRNG algorithm. It's entropy source (not 
exactly the same as a seed, but basically the CSPRNG equivalent) is 
customizable, but by default, it uses _RtlGenRandom on Windows (the same 
source used by the CryptGenRandom algorithms) and '/dev/urandom' on Posix:

https://github.com/Abscissa/DAuth/blob/master/src/dauth/hashdrbg.d

[1] Ugh, still haven't gotten around to finishing DAuth's new version, 
renamed "InstaUser".
Feb 26 2017
next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
Speaking of, another good source of more "how to not mess up security" 
info is here:

https://crackstation.net/hashing-security.htm

DAuth/InstaUser is based on the points in there.
Feb 26 2017
prev sibling parent reply cym13 <cpicard openmailbox.org> writes:
On Sunday, 26 February 2017 at 22:56:14 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 02/26/2017 01:23 PM, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in 
 combination with
 normal PRNG for cryptographic purpose. Some even go as far as 
 reseeding
 at each call to try making it more secure.

 It is a dangerous practice, most PRNG are not designed with 
 security
 (and unpredictability) in mind, and unpredictableSeed was 
 definitely not
 designed with security in mind (or it failed heavily at it). 
 It's a good
 tool when one needs randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea 
 and how it
 could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos 
 but we
 aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


 FWIW, DAuth[1] uses, and offers, an implementation of 
 Hash_DRBG, a well-known and established CSPRNG algorithm. It's 
 entropy source (not exactly the same as a seed, but basically 
 the CSPRNG equivalent) is customizable, but by default, it uses 
 _RtlGenRandom on Windows (the same source used by the 
 CryptGenRandom algorithms) and '/dev/urandom' on Posix:

 https://github.com/Abscissa/DAuth/blob/master/src/dauth/hashdrbg.d

 [1] Ugh, still haven't gotten around to finishing DAuth's new 
 version, renamed "InstaUser".


And it is very interesting to note that you fall into the very 
exact problem I am describing in my post if it weren't that your 
default PRNG isn't seedable. Use any other PRNG and you are 
vulnerable (if I read it correctly, I haven't spent much time on 
that).

 From random.d, the function randomToken calls randomBytes which 
inits a new PRNG which is seeded with unpredictableSeed which 
becomes the actual PRNG used:

string randomToken(Rand = DefaultCryptoRand)(size_t strength = 
defaultTokenStrength)
	if(isDAuthRandom!Rand)
{
	return TokenBase64.encode( randomBytes!Rand(strength) );
}

ubyte[] randomBytes(Rand = DefaultCryptoRand)(size_t numBytes)
	if(isDAuthRandom!Rand)
{
	Rand rand;
	rand.initRand();
	return randomBytes(numBytes, rand);
}

private void initRand(Rand)(ref Rand rand)
	if(isDAuthRandom!Rand)
{
	static if(isSeedable!Rand)
		rand.seed(unpredictableSeed);
}

The only thing that saves you here is that your DefaultCryptoRand 
isn't seedable. Note that I'm not saying that to bash you or 
anything, it just happens to show well why I think my article was 
necessary.
Feb 26 2017
parent Nick Sabalausky <a a.a> writes:
On Monday, 27 February 2017 at 00:10:31 UTC, cym13 wrote:

 The only thing that saves you here is that your 
 DefaultCryptoRand isn't seedable. Note that I'm not saying that 
 to bash you or anything, it just happens to show well why I 
 think my article was necessary.


Oh, don't get me wrong, I'm very much in favor of your article, 
ESPECIALLY if people are using the phobos rng in security 
contexts.

Keep in mind though, if my Hash_DRBG implementation were 
seedable, it wouldn't be a valid implementation of Hash_DRBG 
anyway. (A good example, I think, of why we coders shouldn't be 
inventing crypto algorithms, only implementing establised ones 
created and well-studied by the math/crypto experts.)
Feb 26 2017
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 02/26/2017 01:23 PM, cym13 wrote:

 [1]: https://cym13.github.io/article/unpredictableSeed.html


Good enlightening article. And Veles looks cool, never heard of it before.

One detail:

auto copyState(uint origin) {
     for (uint i=1; i>0; i++) {
         if (origin == Random(i).front)
             return origin;
     }
     return 0;
}

Shouldn't "return origin;" be "return i;"?
Feb 26 2017
parent cym13 <cpicard openmailbox.org> writes:
On Monday, 27 February 2017 at 04:51:37 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 02/26/2017 01:23 PM, cym13 wrote:

 [1]: https://cym13.github.io/article/unpredictableSeed.html


 Good enlightening article. And Veles looks cool, never heard of 
 it before.

 One detail:

 auto copyState(uint origin) {
     for (uint i=1; i>0; i++) {
         if (origin == Random(i).front)
             return origin;
     }
     return 0;
 }

 Shouldn't "return origin;" be "return i;"?


Yes, it should completely be "return i;", thanks for noticing 
I'll correct it.
Feb 26 2017
prev sibling next sibling parent reply Yuxuan Shui <yshuiv7 gmail.com> writes:
On Sunday, 26 February 2017 at 18:23:27 UTC, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in 
 combination with normal PRNG for cryptographic purpose. Some 
 even go as far as reseeding at each call to try making it more 
 secure.

 It is a dangerous practice, most PRNG are not designed with 
 security (and unpredictability) in mind, and unpredictableSeed 
 was definitely not designed with security in mind (or it failed 
 heavily at it). It's a good tool when one needs randomness, not 
 security.

 I wrote a blog post to present exactly why this is a bad idea 
 and how it could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos 
 but we aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


When I see the code for unpredictableSeed I went face palm really 
hard.

I did some digging, and it was way way worse:

https://github.com/dlang/phobos/commit/ff54d867e41abc8261075f0dce1261d68ee09180#diff-713ce153554afc99a07767cc8ba940aeL529

https://github.com/dlang/phobos/commit/c433c36658df45677bf90b00e93cba051883294e
Mar 02 2017
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/2/17 4:50 PM, Yuxuan Shui wrote:

 On Sunday, 26 February 2017 at 18:23:27 UTC, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in combination
 with normal PRNG for cryptographic purpose. Some even go as far as
 reseeding at each call to try making it more secure.

 It is a dangerous practice, most PRNG are not designed with security
 (and unpredictability) in mind, and unpredictableSeed was definitely
 not designed with security in mind (or it failed heavily at it). It's
 a good tool when one needs randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea and how
 it could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos but we
 aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


 When I see the code for unpredictableSeed I went face palm really hard.

 I did some digging, and it was way way worse:

 https://github.com/dlang/phobos/commit/ff54d867e41abc8261075f0dce1261d68ee09180#diff-713ce153554afc99a07767cc8ba940aeL529


 https://github.com/dlang/phobos/commit/c433c36658df45677bf90b00e93cba051883294e


Could you please submit a PR that makes is better? Thanks! -- Andrei
Mar 02 2017
prev sibling parent reply cym13 <cpicard openmailbox.org> writes:
On Thursday, 2 March 2017 at 21:50:36 UTC, Yuxuan Shui wrote:

 On Sunday, 26 February 2017 at 18:23:27 UTC, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in 
 combination with normal PRNG for cryptographic purpose. Some 
 even go as far as reseeding at each call to try making it more 
 secure.

 It is a dangerous practice, most PRNG are not designed with 
 security (and unpredictability) in mind, and unpredictableSeed 
 was definitely not designed with security in mind (or it 
 failed heavily at it). It's a good tool when one needs 
 randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea 
 and how it could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos 
 but we aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


 When I see the code for unpredictableSeed I went face palm 
 really hard.

 I did some digging, and it was way way worse:

 https://github.com/dlang/phobos/commit/ff54d867e41abc8261075f0dce1261d68ee09180#diff-713ce153554afc99a07767cc8ba940aeL529

 https://github.com/dlang/phobos/commit/c433c36658df45677bf90b00e93cba051883294e


This is a misunderstanding: unpredictableSeed is perfectly fine 
as it is. What's wrong is 1) using it for cryptographic purpose 
and 2) systematic reseeding.

1) There is no way to make a cryptographically secure 
pseudo-random number generator that is seedable. If a PRNG is 
seedable then his number of states is finite which makes it cycle 
one way or an other once you've expended all possible states. So 
no cryptographic application should use such PRNG, and therefore 
any seed. For non-cryptographic purpose unpredictableSeed is, 
honnestly, random enough. It isn't you're actual PRNG (or 
shouldn't be, see point 2 but is only used to reseed it from time 
to time.

2) The big mistake is systematic reseeding which is far more 
common than it should be. Using unpredictableSeed as a seed is 
fine, the actual PRNG that is seeded will add a lot of entropy to 
the output. However by systematically reseeding it makes 
unpredictableSeed the PRNG that is actually used (ie, it doesn't 
leave it any time to add entropy). And that is something that 
should never happen because the PRNG in unpredictableSeed is the 
weakest possible. It is not meant to be the actual PRNG.

So this article wasn't meant to be "Haha, Phobos is broken, 
*facepalm*". It was about using tools for what they're meant and 
nothing else (especially when dealing with cryptographic 
problems). The problem, if anything, is in the documentation that 
doesn't enforce that point.
Mar 05 2017
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/05/2017 05:48 AM, cym13 wrote:

 The problem, if anything, is in the documentation that doesn't enforce
 that point.


Thanks. Could we have you, Yishan, or other security expert submit a PR 
for the documentation? I'm not an expert and my coding of 
unpredictableSeed has been at the level of "I don't like those games 
that always do the same `random` things when you start playing."

Another good thing pointed by the article would be to use the Mersenne 
twister for unpredictableSeed, which would make it difficult to infer 
the sequence from a few samples. Please share if that would be a good 
thing to do.


Thanks! -- Andrei
Mar 05 2017
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Sunday, 5 March 2017 at 15:30:29 UTC, Andrei Alexandrescu 
wrote:

 Another good thing pointed by the article would be to use the 
 Mersenne twister for unpredictableSeed, which would make it 
 difficult to infer the sequence from a few samples. Please 
 share if that would be a good thing to do.


"From just a few samples" yes, but in general the internal state 
of Mersenne twister is easy to reverse – e.g. check out 
https://github.com/fx5/not_random for a nice little demonstration.

IMHO there is no problem with unpredictableSeed not being 
cryptographically secure, as long as the docs point out it is not 
to be used anywhere near where a CSPRNG should be. But I'm hardly 
an expert either.

  — David
Mar 05 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Mar 05, 2017 at 10:30:29AM -0500, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Another good thing pointed by the article would be to use the Mersenne
 twister for unpredictableSeed, which would make it difficult to infer
 the sequence from a few samples. Please share if that would be a good
 thing to do.


[...]

Wait, isn't that missing the point?

I thought the whole point of the article was that you shouldn't be using
unpredictableSeed as your PRNG.  It's only supposed to give a
random-enough value to get your chosen PRNG into a (hopefully)
unpredictable initial state. But you should be using values from the
PRNG, not from unpredictableSeed!  Otherwise that's totally missing the
point.

It's possible to use unpredictableSeed for occasionally reseeding your
PRNG, but that should be quite infrequent.  If you find yourself
reseeding your PRNG every other minute, or worse, every time you call
your PRNG, then you're doing something very, very wrong.

Using the Mersenne twister to generate unpredictableSeed seems to me to
be completely backwards.  It should be the other way round: the value of
unpredictableSeed should be random enough to be suitable for seeding a
Mersenne twister algorithm, so that it will start off the algorithm in a
random initial state (and you should be getting values from the
algorithm thereafter, not from unpredictableSeed).


T

-- 
Unix is my IDE. -- Justin Whear
Mar 05 2017
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/05/2017 07:25 PM, H. S. Teoh via Digitalmars-d wrote:

 On Sun, Mar 05, 2017 at 10:30:29AM -0500, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 Another good thing pointed by the article would be to use the Mersenne
 twister for unpredictableSeed, which would make it difficult to infer
 the sequence from a few samples. Please share if that would be a good
 thing to do.


 [...]

 Wait, isn't that missing the point?

 I thought the whole point of the article was that you shouldn't be using
 unpredictableSeed as your PRNG.  It's only supposed to give a
 random-enough value to get your chosen PRNG into a (hopefully)
 unpredictable initial state. But you should be using values from the
 PRNG, not from unpredictableSeed!  Otherwise that's totally missing the
 point.

 It's possible to use unpredictableSeed for occasionally reseeding your
 PRNG, but that should be quite infrequent.  If you find yourself
 reseeding your PRNG every other minute, or worse, every time you call
 your PRNG, then you're doing something very, very wrong.

 Using the Mersenne twister to generate unpredictableSeed seems to me to
 be completely backwards.  It should be the other way round: the value of
 unpredictableSeed should be random enough to be suitable for seeding a
 Mersenne twister algorithm, so that it will start off the algorithm in a
 random initial state (and you should be getting values from the
 algorithm thereafter, not from unpredictableSeed).


Well, the big point is that nothing involving seeds should come near 
anything security-releated. Then secondly, (like you say) don't use any 
initial-seed-getter *as* an RNG.

But that aside, *if*[1] we do want to increase the entropy in 
unpredictableSeed, we should use /dev/(u)random and _RtlGenRandom. Like 
you say, anything less is kinda missing the point (unless someone can 
argue the current one is insufficient for non-security randomization).

[1] And I'm not sure we necessarily do want to. Current 
unpredictableSeed seems good enough as-is for non-security purposes, and 
we don't want to give people more reason to erroneously think it's ok to 
use unpredictableSeed for salts, tokens and the like.
Mar 05 2017
prev sibling parent Kagamin <spam here.lot> writes:
On Sunday, 5 March 2017 at 15:30:29 UTC, Andrei Alexandrescu 
wrote:

 Another good thing pointed by the article would be to use the 
 Mersenne twister for unpredictableSeed, which would make it 
 difficult to infer the sequence from a few samples. Please 
 share if that would be a good thing to do.


Seeding a PRNG from the same PRNG often gives bad results.
Also maybe rename it to arbitrarySeed, unpredictability is not 
the right claim here.
Mar 06 2017
prev sibling parent reply Shachar Shemesh <shachar weka.io> writes:
On 26/02/17 20:23, cym13 wrote:

 Hi,

 I found many times that people use unpredictableSeed in combination with
 normal PRNG for cryptographic purpose. Some even go as far as reseeding
 at each call to try making it more secure.

 It is a dangerous practice, most PRNG are not designed with security
 (and unpredictability) in mind, and unpredictableSeed was definitely not
 designed with security in mind (or it failed heavily at it). It's a good
 tool when one needs randomness, not security.

 I wrote a blog post to present exactly why this is a bad idea and how it
 could be exploited [1].

 The best would be to add a standard CSPRNG interface to Phobos but we
 aren't there yet.

 [1]: https://cym13.github.io/article/unpredictableSeed.html


Excuse me if I'm asking a trivial question. Why not just seed it from 
/dev/urandom? (or equivalent on non-Linux platforms. I know at least 
Windows has an equivalent).

Shachar
Mar 06 2017
parent reply sarn <sarn theartofmachinery.com> writes:
On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh wrote:

 Excuse me if I'm asking a trivial question. Why not just seed 
 it from /dev/urandom? (or equivalent on non-Linux platforms. I 
 know at least Windows has an equivalent).

 Shachar


One reason is that /dev/urandom isn't always available, e.g., in 
a chroot.  Sure, these are corner cases, but it's annoying when 
stuff like this doesn't "just work".
Mar 06 2017
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/06/2017 05:19 PM, sarn wrote:

 On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh wrote:

 Excuse me if I'm asking a trivial question. Why not just seed it from
 /dev/urandom? (or equivalent on non-Linux platforms. I know at least
 Windows has an equivalent).

 Shachar


 One reason is that /dev/urandom isn't always available, e.g., in a
 chroot.  Sure, these are corner cases, but it's annoying when stuff like
 this doesn't "just work".


I don't claim to be any sort of linux expert or anything, but doesn't 
chroot have a reputation for being a bit of a finicky, leaky abstraction 
anyway? I haven't really used them, but that's been my understanding...?
Mar 06 2017
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, March 06, 2017 22:04:44 Nick Sabalausky  via Digitalmars-d wrote:

 On 03/06/2017 05:19 PM, sarn wrote:

 On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh wrote:

 Excuse me if I'm asking a trivial question. Why not just seed it from
 /dev/urandom? (or equivalent on non-Linux platforms. I know at least
 Windows has an equivalent).

 Shachar


 One reason is that /dev/urandom isn't always available, e.g., in a
 chroot.  Sure, these are corner cases, but it's annoying when stuff like
 this doesn't "just work".


 I don't claim to be any sort of linux expert or anything, but doesn't
 chroot have a reputation for being a bit of a finicky, leaky abstraction
 anyway? I haven't really used them, but that's been my understanding...?


If you want a fully secure chroot, then what you really want is BSD jails or
Solaris zones. chroots are indeed too leaky to be secure. But secure
container-ization doesn't really matter here, since a D program using D's
standard number generator should work regardless of where it's running. So,
it's a question of whether we're guaranteed to get at /dev/urandom or not,
and if not, how reasonable it is to require that it be accessible for the
program to run. There _are_ programs that require access to /dev, and /dev
is _usually_ available.

Regardless, if there is no guarantee that /dev/urandom (or whatever system
resource for getting randomness is) is going to be accessible, and we want
to use it, then we either have to require that it be accessible and error
out if it isn't, or we have to have a backup if accessing it fails. Ideally,
you'd be able to just use /dev/urandom and not worry about it, but I don't
know how common it is for /dev/urandom to be unavailable or how reasonable
it is to require that it be available.

In general though, using /dev/urandom to seed the pseudo-random number
generator seems like a good plan.

- Jonathan M Davis
Mar 06 2017
parent reply Seb <seb wilzba.ch> writes:
On Tuesday, 7 March 2017 at 03:43:42 UTC, Jonathan M Davis wrote:

 On Monday, March 06, 2017 22:04:44 Nick Sabalausky  via 
 Digitalmars-d wrote:

 On 03/06/2017 05:19 PM, sarn wrote:

 On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh 
 wrote:

 Excuse me if I'm asking a trivial question. Why not just 
 seed it from /dev/urandom? (or equivalent on non-Linux 
 platforms. I know at least Windows has an equivalent).

 Shachar


 One reason is that /dev/urandom isn't always available, 
 e.g., in a chroot.  Sure, these are corner cases, but it's 
 annoying when stuff like this doesn't "just work".


 I don't claim to be any sort of linux expert or anything, but 
 doesn't chroot have a reputation for being a bit of a finicky, 
 leaky abstraction anyway? I haven't really used them, but 
 that's been my understanding...?


 If you want a fully secure chroot, then what you really want is 
 BSD jails or Solaris zones. chroots are indeed too leaky to be 
 secure. But secure container-ization doesn't really matter 
 here, since a D program using D's standard number generator 
 should work regardless of where it's running. So, it's a 
 question of whether we're guaranteed to get at /dev/urandom or 
 not, and if not, how reasonable it is to require that it be 
 accessible for the program to run. There _are_ programs that 
 require access to /dev, and /dev is _usually_ available.

 Regardless, if there is no guarantee that /dev/urandom (or 
 whatever system resource for getting randomness is) is going to 
 be accessible, and we want to use it, then we either have to 
 require that it be accessible and error out if it isn't, or we 
 have to have a backup if accessing it fails. Ideally, you'd be 
 able to just use /dev/urandom and not worry about it, but I 
 don't know how common it is for /dev/urandom to be unavailable 
 or how reasonable it is to require that it be available.

 In general though, using /dev/urandom to seed the pseudo-random 
 number generator seems like a good plan.

 - Jonathan M Davis


As apparently no one here hasn't mentioned this, Linux >= 3.17 
has a dedicated syscall API. Please see:

http://man7.org/linux/man-pages/man2/getrandom.2.html

And this excellent introductory article:

https://lwn.net/Articles/605828

I did work on getting a nice getEntropy function into mir-random:

https://github.com/libmir/mir-random/pull/13

(For which it was planned to backport it to Phobos after some 
testing and real-world  feedback on the API.)
Mar 07 2017
next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 7 March 2017 at 10:18:52 UTC, Seb wrote:

 http://man7.org/linux/man-pages/man2/getrandom.2.html



Unnecessarily reading large quantities of data will have a 
negative impact on other users of the /dev/random and 
/dev/urandom devices.  Therefore, getrandom() should not be used 
for Monte Carlo simulations or other programs/algorithms which 
are doing probabilistic sampling.


In other words it shouldn't be used when not strictly necessary.
Mar 07 2017
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/07/2017 09:46 AM, Kagamin wrote:

 On Tuesday, 7 March 2017 at 10:18:52 UTC, Seb wrote:

 http://man7.org/linux/man-pages/man2/getrandom.2.html



 Unnecessarily reading large quantities of data will have a negative
 impact on other users of the /dev/random and /dev/urandom devices.
 Therefore, getrandom() should not be used for Monte Carlo simulations
 or other programs/algorithms which are doing probabilistic sampling.


 In other words it shouldn't be used when not strictly necessary.


Yes, although that's true of reading /dev/(u)random too. (Just to be clear.)
Mar 07 2017
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/07/2017 05:18 AM, Seb wrote:

 As apparently no one here hasn't mentioned this, Linux >= 3.17 has a
 dedicated syscall API. Please see:

 http://man7.org/linux/man-pages/man2/getrandom.2.html

 And this excellent introductory article:

 https://lwn.net/Articles/605828


Ooh, that's great to know! (Kinda sad that it seems necessary, given the 
"unix filesystem and unix design" ideals, but oh well, realities are 
realities.)

Is there a "proper" way to check for this function's existence on a 
local machine, other than test-compiling, or is parsing 'uname -a' as 
"right way" as it gets?

And anyone know about OSX? Would OSX use the getentropy the article you 
linked to mentions for OpenBSD? Or something else? Or just fallback to 
/dev/(u)random?

This really deserves a Phobos PR, IMO, FWIW. Maybe I'll try my hand at 
it if any OSX folk have tips for that OS.
Mar 07 2017
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-03-07 21:15, Nick Sabalausky (Abscissa) wrote:


 And anyone know about OSX? Would OSX use the getentropy the article you
 linked to mentions for OpenBSD?


As far as I can see, there's no "getentropy" on macOS. I see references 
to it online, but I cannot find it in any header files.


 Or something else?


I found something called "arc4random" and a couple of related functions [1].


 Or just fallback to /dev/(u)random?


/dev/(u)random does exist on macOS. This is what Apple seems to 
recommend [2].

[1] 
https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man3/arc4random.3.html

[2] 
https://developer.apple.com/library/content/documentation/Security/Conceptual/cryptoservices/RandomNumberGenerationAPIs/RandomNumberGenerationAPIs.html

-- 
/Jacob Carlborg
Mar 07 2017
prev sibling parent reply Yuxuan Shui <yshuiv7 gmail.com> writes:
On Tuesday, 7 March 2017 at 20:15:56 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 03/07/2017 05:18 AM, Seb wrote:

[...]


 Ooh, that's great to know! (Kinda sad that it seems necessary, 
 given the "unix filesystem and unix design" ideals, but oh 
 well, realities are realities.)

 Is there a "proper" way to check for this function's existence 
 on a local machine, other than test-compiling, or is parsing 
 'uname -a' as "right way" as it gets?

 And anyone know about OSX? Would OSX use the getentropy the 
 article you linked to mentions for OpenBSD? Or something else? 
 Or just fallback to /dev/(u)random?

 This really deserves a Phobos PR, IMO, FWIW.


https://github.com/dlang/phobos/pull/5230
Mar 08 2017
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/8/17 11:24 PM, Yuxuan Shui wrote:

 On Tuesday, 7 March 2017 at 20:15:56 UTC, Nick Sabalausky (Abscissa) wrote:

 On 03/07/2017 05:18 AM, Seb wrote:

 [...]


 Ooh, that's great to know! (Kinda sad that it seems necessary, given
 the "unix filesystem and unix design" ideals, but oh well, realities
 are realities.)

 Is there a "proper" way to check for this function's existence on a
 local machine, other than test-compiling, or is parsing 'uname -a' as
 "right way" as it gets?

 And anyone know about OSX? Would OSX use the getentropy the article
 you linked to mentions for OpenBSD? Or something else? Or just
 fallback to /dev/(u)random?

 This really deserves a Phobos PR, IMO, FWIW.


 https://github.com/dlang/phobos/pull/5230


Thanks Yuxuan, sorry for missing this. Can we have this peer reviewed by 
1-2 crypto experts? Thanks! -- Andrei
Mar 21 2017
next sibling parent reply sarn <sarn theartofmachinery.com> writes:
On Tuesday, 21 March 2017 at 10:27:27 UTC, Andrei Alexandrescu 
wrote:

 Thanks Yuxuan, sorry for missing this. Can we have this peer 
 reviewed by 1-2 crypto experts? Thanks! -- Andrei


By API, unpredictableSeed() only returns a 32b uint and will 
never meet crypto standards.  Beware of anyone who offers to 
review it based on their "crypto expertise".

unpredictableSeed() is just for things like making single-player 
games more interesting.  It simply isn't for security, and that's 
pretty much what cym13's post was about.
Mar 21 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Mar 21, 2017 at 10:11:44PM +0000, sarn via Digitalmars-d wrote:

 On Tuesday, 21 March 2017 at 10:27:27 UTC, Andrei Alexandrescu wrote:

 Thanks Yuxuan, sorry for missing this. Can we have this peer
 reviewed by 1-2 crypto experts? Thanks! -- Andrei


 
 By API, unpredictableSeed() only returns a 32b uint and will never
 meet crypto standards.  Beware of anyone who offers to review it based
 on their "crypto expertise".
 
 unpredictableSeed() is just for things like making single-player games
 more interesting.  It simply isn't for security, and that's pretty
 much what cym13's post was about.


Yeah, why is it that people still keep thinking unpredictableSeed(), or
indeed, the whole of the current std.random, is useful for *anything*
related to crypto??

If you want to do crypto, you should be using a crypto library that is
*designed* to be cryptographically secure and *verified* by
cryptoanalysts to be secure.  std.random is a far cry from that, and
crypto isn't even its charter anyway.

What std.random is useful for is to make games more interesting, or for
certain kinds of Monte Carlo simulations. (Note that some Monte Carlo
simulations may be sensitive to hidden patterns in std.random PRNGs, so
you should choose your PRNG carefully, and/or take the simulation
results with a grain of salt.) Or for things like probabilistic
algorithms (e.g., probabilistic approximate solution finders for
NP-complete problems and the like).

Using it for anything crypto- or security-related is just begging to be
hacked, esp. in this day and age.


T

-- 
Why waste time learning, when ignorance is instantaneous? -- Hobbes, from
Calvin & Hobbes
Mar 24 2017
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/24/2017 02:56 PM, H. S. Teoh via Digitalmars-d wrote:

 Yeah, why is it that people still keep thinking unpredictableSeed(), or
 indeed, the whole of the current std.random, is useful for *anything*
 related to crypto??


Seems there's two issues there:

1. The name "unpredictableSeed" is highly misleading, if not outright false.

2. Maybe the std.random docs need to be more clear, right up top, that 
it's not for anything security-related.
Mar 26 2017
parent reply Yuxuan Shui <yshuiv7 gmail.com> writes:
On Sunday, 26 March 2017 at 17:55:20 UTC, Nick Sabalausky 
(Abscissa) wrote:

 2. Maybe the std.random docs need to be more clear, right up 
 top, that it's not for anything security-related.


Agreed. Like what Python did here: 
https://docs.python.org/3/library/random.html
Mar 26 2017
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Mar 26, 2017 at 11:46:44PM +0000, Yuxuan Shui via Digitalmars-d wrote:

 On Sunday, 26 March 2017 at 17:55:20 UTC, Nick Sabalausky (Abscissa) wrote:

 
 2. Maybe the std.random docs need to be more clear, right up top,
 that it's not for anything security-related.


 
 Agreed. Like what Python did here:
 https://docs.python.org/3/library/random.html


https://github.com/dlang/phobos/pull/5306


T

-- 
I see that you JS got Bach.
Mar 26 2017
prev sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 21 March 2017 at 10:27:27 UTC, Andrei Alexandrescu 
wrote:

 Thanks Yuxuan, sorry for missing this. Can we have this peer 
 reviewed by 1-2 crypto experts? Thanks! -- Andrei


It's not recommended to use system CSPRNG for non-cryptographic 
purposes: 
https://forum.dlang.org/post/xwlzzeyvatwsohqcynka forum.dlang.org
Mar 22 2017
parent reply Shachar Shemesh <shachar weka.io> writes:
On 22/03/17 11:08, Kagamin wrote:

 On Tuesday, 21 March 2017 at 10:27:27 UTC, Andrei Alexandrescu wrote:

 Thanks Yuxuan, sorry for missing this. Can we have this peer reviewed
 by 1-2 crypto experts? Thanks! -- Andrei


 It's not recommended to use system CSPRNG for non-cryptographic
 purposes: https://forum.dlang.org/post/xwlzzeyvatwsohqcynka forum.dlang.org


That does not apply here. We're not talking about using the random pool 
for generating random numbers. Only for generating the random 
algorithm's seed.

Shachar
Mar 22 2017
parent Kagamin <spam here.lot> writes:
On Wednesday, 22 March 2017 at 09:12:14 UTC, Shachar Shemesh 
wrote:

 That does not apply here. We're not talking about using the 
 random pool for generating random numbers. Only for generating 
 the random algorithm's seed.


See OP:
On Sunday, 26 February 2017 at 18:23:27 UTC, cym13 wrote:

 Some even go as far as reseeding at each call to try making it 
 more secure.


Also MonoTime resets on every reboot.
Mar 22 2017
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 06 Mar 2017 22:04:44 -0500
schrieb "Nick Sabalausky (Abscissa)"
<SeeWebsiteToContactMe semitwist.com>:


 On 03/06/2017 05:19 PM, sarn wrote:

 On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh wrote:  

 Excuse me if I'm asking a trivial question. Why not just seed it
 from /dev/urandom? (or equivalent on non-Linux platforms. I know
 at least Windows has an equivalent).

 Shachar  


 One reason is that /dev/urandom isn't always available, e.g., in a
 chroot.  Sure, these are corner cases, but it's annoying when stuff
 like this doesn't "just work".  


 
 I don't claim to be any sort of linux expert or anything, but doesn't 
 chroot have a reputation for being a bit of a finicky, leaky
 abstraction anyway? I haven't really used them, but that's been my
 understanding...?


chroots were used for security stuff in the past (chrooting a ftp
server and similar stuff) and they're indeed a leaky abstraction in
that case.

However, chroots can also be used to 'chroot into another OS'. E.g.
people sometimes  chroot into the OS on a harddisk from a livecd. This
is sometimes useful to repair a system, install packages, ...

-- Johannes
Mar 06 2017
parent Shachar Shemesh <shachar weka.io> writes:
On 07/03/17 09:13, Johannes Pfau wrote:

 chroots were used for security stuff in the past (chrooting a ftp
 server and similar stuff) and they're indeed a leaky abstraction in
 that case.


I'm not aware of an attack against a process chrooting and dropping 
privileges (in other words, I'm not aware of a way to escape a chroot 
jail unless you have root access).

Care to provide counter examples?

Thanks,
Shachar
Mar 06 2017
prev sibling parent Shachar Shemesh <shachar weka.io> writes:
On 07/03/17 00:19, sarn wrote:

 On Monday, 6 March 2017 at 10:12:09 UTC, Shachar Shemesh wrote:

 Excuse me if I'm asking a trivial question. Why not just seed it from
 /dev/urandom? (or equivalent on non-Linux platforms. I know at least
 Windows has an equivalent).

 Shachar


 One reason is that /dev/urandom isn't always available, e.g., in a
 chroot.  Sure, these are corner cases, but it's annoying when stuff like
 this doesn't "just work".


If I write a program that relies on there being /dev/null, /dev/zero or 
/dev/tty, I don't go ahead and publish this on the program requirements. 
It is okay to assume those are available to me, and if your chroot/LFS 
install doesn't have them and my program breaks, well, tough to be you.

I don't think /dev/urandom is any different in that regard. Worst case, 
you can try to use it, and fall back to whatever it is we're doing today 
if it doesn't work. I don't think the chroot argument is a good one, though.

Shachar
Mar 06 2017