www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Documentation style, SHA1, stream filters and malloc w/ GC

reply Daniel Keep <daniel.keep dummy.gmail.com> writes:
Hi all.  Just in case the subject didn't make it quite obvious, I have 
several questions :P

1) Documentation Style


there was an official/sanctioned documentation style to use ala 
javadoc/Python docstrings/etc.

2) SHA1

I noticed that the standard library has an implementation of MD5, but no 
SHA1.  Since MD5 was recently broken, and because I didn't know any 
better (:P), I decided to write an SHA1 implementation in D.  I was 
wondering if there was any interest in this for Phobos?

I've got a port of the RFC3174 reference code done, and I plan on 
refactoring it so that it's similar to the MD5 module.  I was also 
thinking about writing a Stream class that progressively computes the 
hash as you write to it.

Which was what gave me an idea for...

3) Stream filters

I don't know if there's any interest in something like this, but here 
goes.  We already have Streams in the standard library, so how about 
Stream Filters.  They would basically just be subclasses of the standard 
Stream class (so you can pass them to existing routines).  The 
difference is that they would perform some operation on data that gets 
read/written through them.

I had a quick Google around, and it looks like Java and PHP already have 
something like this, so it could be useful.  The first application that 
came to mind would be hashing the contents of a file as you write it 
out, instead of having to either use two streams, or an SHA1 context and 
a stream.

4) Malloc and the GC
(Sorry if this appears in another post... I doesn't seem to be posting 
my messages properly...)

Last question (for now), so I'll make it quick: can I allocate memory 
with malloc, add a root to the GC, and trust that the GC will be able to 
clean it up, or is mixing memory allocation systems a no-no?

Anyway, thanks for your time.  Let's hope it posts this time :P

	-- Daniel
Nov 12 2004
next sibling parent "Asaf Karagila" <kas1 netvision.net.il> writes:
<snip>

 2) SHA1

 I noticed that the standard library has an implementation of MD5, but no 
 SHA1.  Since MD5 was recently broken, and because I didn't know any better 
 (:P), I decided to write an SHA1 implementation in D.  I was wondering if 
 there was any interest in this for Phobos?

 I've got a port of the RFC3174 reference code done, and I plan on 
 refactoring it so that it's similar to the MD5 module.  I was also 
 thinking about writing a Stream class that progressively computes the hash 
 as you write to it.

 Which was what gave me an idea for...


<snip>

i think there should be a part of phobos that would be like a built in 
crypto module.
one way hashes SHA1, MD5, RipeMD160, Tiger, etc..
symmetric ciphers like AES and NESSIE candidates (and chosen)
and asymmetric ciphers like RSA, Rabin, DH, etc..

- Asaf.
Nov 12 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"Daniel Keep" <daniel.keep dummy.gmail.com> wrote in message
news:cn2e73$be2$1 digitaldaemon.com...

 Hi all.  Just in case the subject didn't make it quite obvious, I have
 several questions :P

 1) Documentation Style


 there was an official/sanctioned documentation style to use ala
 javadoc/Python docstrings/etc.


It ranges from javadoc to doxygen to simple // comments

[snip]


 3) Stream filters

 I don't know if there's any interest in something like this, but here
 goes.  We already have Streams in the standard library, so how about
 Stream Filters.  They would basically just be subclasses of the standard
 Stream class (so you can pass them to existing routines).  The
 difference is that they would perform some operation on data that gets
 read/written through them.

 I had a quick Google around, and it looks like Java and PHP already have
 something like this, so it could be useful.  The first application that
 came to mind would be hashing the contents of a file as you write it
 out, instead of having to either use two streams, or an SHA1 context and
 a stream.


See the std.stream.BufferedStream for an example of a D filter stream. There
currently isn't a class for generic filter streams but such a class would
probably just declare the source stream and maybe it would also includes
some logic about opening and closing the source stream. I remember people
tossing around ideas for std.stream and this was one of them - though I
don't remember seeing any implementations.


 4) Malloc and the GC
 (Sorry if this appears in another post... I doesn't seem to be posting
 my messages properly...)

 Last question (for now), so I'll make it quick: can I allocate memory
 with malloc, add a root to the GC, and trust that the GC will be able to
 clean it up, or is mixing memory allocation systems a no-no?


The GC can't free memory it didn't allocate. The "addRoot" function just
takes a pointer to a GC resource and adds it to a list that is scanned by
the GC. This is useful when a pointer is passed to an "external resource"
that the GC doesn't scan and so without calling addRoot the memory would be
collected. The addRange function adds an entire chunk of memory to the scan
list.


 Anyway, thanks for your time.  Let's hope it posts this time :P

 -- Daniel
Nov 12 2004
parent reply Daniel Keep <daniel.keep dummy.gmail.com> writes:
Ben Hinkle wrote:
[snip] :P

 
 The GC can't free memory it didn't allocate. The "addRoot" function just
 takes a pointer to a GC resource and adds it to a list that is scanned by
 the GC. This is useful when a pointer is passed to an "external resource"
 that the GC doesn't scan and so without calling addRoot the memory would be
 collected. The addRange function adds an entire chunk of memory to the scan
 list.
 


Oh... sh^H^Hbother.  I was afraid of that.  I have a really fast piece 
of code... that spends more time zeroing out an array then it does in 
actual processing -_-

I would just code it in straight assembler, and then link it in, but 
then I'd have the same problem.  I suppose that barring some magical way 
to get the GC to give me uninitialized memory, I'm stuffed, eh?

It's a pity I agree with Walter's reasons for automatically initializing 
memory... because otherwise I could have a go at him for it :P
Nov 12 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cn2rui$v8j$1 digitaldaemon.com>, Daniel Keep says...

I would just code it in straight assembler, and then link it in, but 
then I'd have the same problem.  I suppose that barring some magical way 
to get the GC to give me uninitialized memory, I'm stuffed, eh?

It's a pity I agree with Walter's reasons for automatically initializing 
memory... because otherwise I could have a go at him for it :P


You could always create some kind of allocator class.  Use alloca, malloc, or
just hold on to GC-allocated memory for reuse.


Sean
Nov 12 2004
parent Daniel Keep <daniel.keep dummy.gmail.com> writes:
Hmm... hadn't thought of that.  I suppose I could write a class that 
mimics regular arrays, but it seems like it wouldn't be as useful as a 
normal, run of the mill array (and not extendable, if I understand 
things correctly).

What I may end up doing is making the function an in-place operation (so 
the programmer has to manually duplicate arrays if need be), and make a 
version available for normal arrays, pointers to data, and for funky 
"kinda array" objects :)

Anyway, thanks for the tip.

Sean Kelly wrote:

 In article <cn2rui$v8j$1 digitaldaemon.com>, Daniel Keep says...
 

I would just code it in straight assembler, and then link it in, but 
then I'd have the same problem.  I suppose that barring some magical way 
to get the GC to give me uninitialized memory, I'm stuffed, eh?

It's a pity I agree with Walter's reasons for automatically initializing 
memory... because otherwise I could have a go at him for it :P


 
 
 You could always create some kind of allocator class.  Use alloca, malloc, or
 just hold on to GC-allocated memory for reuse.
 
 
 Sean
Nov 13 2004
prev sibling next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Daniel Keep" <daniel.keep dummy.gmail.com> wrote in message
news:cn2e73$be2$1 digitaldaemon.com...

 2) SHA1

 I noticed that the standard library has an implementation of MD5, but no
 SHA1.  Since MD5 was recently broken, and because I didn't know any
 better (:P), I decided to write an SHA1 implementation in D.  I was
 wondering if there was any interest in this for Phobos?


The reason SHA1 was not added to Phobos is the government spec for it says
it would need an export license to ship it. I've been informed that this no
longer applies, but if the person that informed me was mistaken, it's my
head on the chopping block :-(
Nov 13 2004
parent reply Daniel Keep <daniel.keep dummy.gmail.com> writes:
Export license?  Hmm...  Then I wonder how everyone else gets away with 
it :P

None the less, I'd love to see SHA-1, and all the other crypto stuff in 
Phobos, so I'll try and do some digging... see if I can't find anything.

Walter wrote:

 "Daniel Keep" <daniel.keep dummy.gmail.com> wrote in message
 news:cn2e73$be2$1 digitaldaemon.com...
 

2) SHA1

I noticed that the standard library has an implementation of MD5, but no
SHA1.  Since MD5 was recently broken, and because I didn't know any
better (:P), I decided to write an SHA1 implementation in D.  I was
wondering if there was any interest in this for Phobos?


 
 
 The reason SHA1 was not added to Phobos is the government spec for it says
 it would need an export license to ship it. I've been informed that this no
 longer applies, but if the person that informed me was mistaken, it's my
 head on the chopping block :-(
Nov 13 2004
parent reply "Thomas Kuehne" <thomas-dloop kuehne.cn> writes:
Daniel Keep schrieb:

 Export license?  Hmm...  Then I wonder how everyone else gets away with
 it :P

 None the less, I'd love to see SHA-1, and all the other crypto stuff in
 Phobos, so I'll try and do some digging... see if I can't find anything.


We could use the Debain way. NON-US developer(s) implement SHA-1 (and all the
other crypto
stuff) and it's hosted on s NON-US server.

If a server is needed, I'd be willing to host it on a German server.

Thomas
Nov 14 2004
parent reply Daniel Keep <daniel.keep dummy.gmail.com> writes:
I don't think that will be neccecary.  I've been digging all day, and my 
eyes hurt from all the monospaced documents I've been reading, but I 
think I've nutted out the core of the matter.

Basically, you *can* freely export cryptography software from the US, 
provided:

1) It's source is publicly available
2) You notify the US government that you are exporting crypto. software 
(I'm still not 100% sure on the specifics, but it looks as though you 
just have to email them w/ your URL)
3) You don't *consciously* export to any blacklisted countries (North 
Korea, Iran, etc.)

The Debian mailing list has been very helpful in this little quest :P

Also, I checked my Debian install, and sha1sum is included in the main 
package--ie: it's being exported from the US.  So we shouldn't have any 
problems with SHA-1 (specifically, anyway).

The following is quoted from the revised EAR section 740.13(e)(5), taken 
from http://lists.debian.org/debian-legal/2002/06/msg00247.html:

     (5) Notification requirement. You must provide BIS written
     notification of the Internet location (e.g., URL or Internet
     address) of the source code or a copy of the source code by the time
     of export.
     Submit the notification by email to BIS at crypt bis.doc.gov, and
     provide a copy of the notification to the ENC Encryption Request
     Coordinator at enc ncsc.mil.

There is also an excellent discussion on the Debian mailing list about 
the inclusion of crypto code into the "main" distribution (which gets 
hosted on US sites) here: 
http://lists.debian.org/debian-policy/2001/01/msg00036.html

Information on US crypto. policies can be found here: 
http://www.bxa.doc.gov/Encryption/Default.htm

So, I think we just need to take a careful look at what information we 
need to email to the addresses mentioned above, and we should be golden.

I was thinking about possibly posting to the Debian mailing list and/or 
the Mono mailing list (another project with lots of open source crypto 
code) with regards to what they had to do.  But that might seem a little 
rude :P

Anyway, I'll keep looking into this.  Hope this is of help.

	-- Daniel

Thomas Kuehne wrote:

 Daniel Keep schrieb:
 

Export license?  Hmm...  Then I wonder how everyone else gets away with
it :P

None the less, I'd love to see SHA-1, and all the other crypto stuff in
Phobos, so I'll try and do some digging... see if I can't find anything.


 
 
 We could use the Debain way. NON-US developer(s) implement SHA-1 (and all the
other crypto
 stuff) and it's hosted on s NON-US server.
 
 If a server is needed, I'd be willing to host it on a German server.
 
 Thomas
Nov 14 2004
parent "Asaf Karagila" <kas1 netvision.net.il> writes:
I would seriously consider adding many other ciphers,
there are many good ciphers around there, it's true we can't have them all,
but to add a built in crypto unit, sorta like the CryptoAPIs in windows..

- Asaf.
Nov 14 2004
prev sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Sat, 13 Nov 2004 00:38:28 +1100, Daniel Keep 
<daniel.keep dummy.gmail.com> wrote:

 2) SHA1

 I noticed that the standard library has an implementation of MD5, but no 
 SHA1.  Since MD5 was recently broken, and because I didn't know any 
 better (:P), I decided to write an SHA1 implementation in D.  I was 
 wondering if there was any interest in this for Phobos?

 I've got a port of the RFC3174 reference code done, and I plan on 
 refactoring it so that it's similar to the MD5 module.  I was also 
 thinking about writing a Stream class that progressively computes the 
 hash as you write to it.

 Which was what gave me an idea for...


Take a look at my implementations of:
  MD4
  MD5
  SHA0
  SHA1
  SHA256
  SHA512
  Tiger

Here:
   http://www.dsource.org/projects/deimos/

You can get/view the source here:
   http://svn.dsource.org/svn/projects/deimos/trunk/etc/crypto/hash/

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Nov 14 2004
parent reply Daniel Keep <daniel.keep dummy.gmail.com> writes:
I meant to get Deimos, but I got put off by the "No Downloads" text.

<checks out with subversion>

<looks at Deimos, then at own SHA-1 code>

Well, there goes that project...  But on the bright side, *mine* has a 
version switch to compile to an actual sha1sum program. :P

Ah well.  I suppose I can go back to making my super-cool cached linked 
list implementation... :)

Thanks for the heads up,

	-- Daniel "So much for that idea"

Regan Heath wrote:

 On Sat, 13 Nov 2004 00:38:28 +1100, Daniel Keep 
 <daniel.keep dummy.gmail.com> wrote:
 

 2) SHA1

 I noticed that the standard library has an implementation of MD5, but 
 no SHA1.  Since MD5 was recently broken, and because I didn't know any 
 better (:P), I decided to write an SHA1 implementation in D.  I was 
 wondering if there was any interest in this for Phobos?

 I've got a port of the RFC3174 reference code done, and I plan on 
 refactoring it so that it's similar to the MD5 module.  I was also 
 thinking about writing a Stream class that progressively computes the 
 hash as you write to it.

 Which was what gave me an idea for...


 
 
 Take a look at my implementations of:
  MD4
  MD5
  SHA0
  SHA1
  SHA256
  SHA512
  Tiger
 
 Here:
   http://www.dsource.org/projects/deimos/
 
 You can get/view the source here:
   http://svn.dsource.org/svn/projects/deimos/trunk/etc/crypto/hash/
 
 Regan
Nov 15 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 16 Nov 2004 02:42:28 +1100, Daniel Keep 
<daniel.keep dummy.gmail.com> wrote:

 I meant to get Deimos, but I got put off by the "No Downloads" text.

 <checks out with subversion>

 <looks at Deimos, then at own SHA-1 code>

 Well, there goes that project...  But on the bright side, *mine* has a 
 version switch to compile to an actual sha1sum program. :P


Then please feel free to add it to the Deimos version, if you want, no 
pressure :)


 Ah well.  I suppose I can go back to making my super-cool cached linked 
 list implementation... :)


Sounds cool.


 Thanks for the heads up,


NP.

Regan


 Regan Heath wrote:

 On Sat, 13 Nov 2004 00:38:28 +1100, Daniel Keep 
 <daniel.keep dummy.gmail.com> wrote:


 2) SHA1

 I noticed that the standard library has an implementation of MD5, but 
 no SHA1.  Since MD5 was recently broken, and because I didn't know any 
 better (:P), I decided to write an SHA1 implementation in D.  I was 
 wondering if there was any interest in this for Phobos?

 I've got a port of the RFC3174 reference code done, and I plan on 
 refactoring it so that it's similar to the MD5 module.  I was also 
 thinking about writing a Stream class that progressively computes the 
 hash as you write to it.

 Which was what gave me an idea for...



 Take a look at my implementations of:
  MD4
  MD5
  SHA0
  SHA1
  SHA256
  SHA512
  Tiger

 Here:
   http://www.dsource.org/projects/deimos/

 You can get/view the source here:
   http://svn.dsource.org/svn/projects/deimos/trunk/etc/crypto/hash/

 Regan






-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Nov 15 2004