digitalmars.D.learn - Working on a library: request for code review
- Mike (15/15) Jun 11 2014 Hello.
- cal (8/10) Jun 11 2014 Hi, sorry didn't read through your code yet, but while ago I
- Mike (7/9) Jun 12 2014 Well I started working on TGA because I was disappointed that no
- Rene Zwanenburg (14/24) Jun 12 2014 I'm looking over your code ATM but I'd like to reply to this
- Xavier Bigand (14/38) Jun 12 2014 I think it can be a great advantage to have some things like image
- Xavier Bigand (4/46) Jun 12 2014 I am thinking an other benefit is what you see as a default, the
- Mike (20/20) Jun 12 2014 I can shed some light on my, beginners, point of view and the
- Rene Zwanenburg (91/93) Jun 12 2014 I usually don't trust shortened URL's. Can you please post full
- Mike (3/7) Jun 13 2014 Thank you for feedback, hoping for more! I will apply the
- Mike Wey (4/8) Jun 13 2014 http://dlang.org/library/std/algorithm/among.html
- Mike (2/12) Jun 15 2014 Will apply this as well, thanks :-)
- Mike (7/7) Jun 16 2014 I have refactored the code as recommended.
- Rene Zwanenburg (3/10) Jun 16 2014 I've been pretty busy this weekend, so sorry for the late reply.
- Rene Zwanenburg (69/76) Jun 16 2014 https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L80
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (6/11) Jun 17 2014 If you define nested structs/unions, it's better to make them
- Rene Zwanenburg (5/17) Jun 18 2014 Good call. In this case it doesn't add the hidden field, but for
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (5/19) Jun 19 2014 Interesting, I thought it's there for all nested structs. But I
- Rene Zwanenburg (4/26) Jun 19 2014 The absence of the hidden field without methods is probably to
- Mike (2/2) Jun 19 2014 Once again thanks for the feedback. The RLE rewrite using std is
- Mike (5/5) Jun 20 2014 Do you think it's ready for a v0.1 release? It be willing to add
- Rene Zwanenburg (4/6) Jun 21 2014 Sure, go for it :). Dub package versioning should be done using
- Mike (7/13) Jun 17 2014 A non-mapped image may contain a color map ;-0 it's simply
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (2/15) Jun 18 2014 Perhaps `return !!(...);` ?
- Rene Zwanenburg (2/25) Jun 18 2014 test
- Rene Zwanenburg (6/32) Jun 18 2014 The web interface didn't work for me for about a day. It seems to
- Rene Zwanenburg (4/7) Jun 18 2014 Here's a version using std.algorithm and std.range. I think it's
Hello. I am new to D and I must admit I really like the language. In my opinion it takes the best from C++ and, say, Python and combines it really elegantly. Great work! I am currently working on my first library in D - related to TARGA image format. Here's the link to the repo: http://bit.ly/1mIuGhv It's a work-in-progress case: at the moment the library does what I need for my other projects, but there is a couple of things that I want to add/fix soon. Perhaps someone could have a look at the code and point out some obvious traps that I fell into etc. Any feedback would be great! Best regards, Mike
Jun 11 2014
On Wednesday, 11 June 2014 at 18:29:27 UTC, Mike wrote:Hello. Here's the link to the repo: http://bit.ly/1mIuGhvHi, sorry didn't read through your code yet, but while ago I wrote some encoders/decoders for jpeg and png (https://github.com/callumenator/imaged, haven't compiled it in a while). Might it be worth stitching things together into a proper image processing package? Cheers, cal
Jun 11 2014
On Thursday, 12 June 2014 at 00:20:28 UTC, cal wrote:Might it be worth stitching things together into a proper image processing package?Well I started working on TGA because I was disappointed that no image abstraction is present in Phobos. Go has some imaging APIs and I think D would benefit from having one too (out of the box). Would I work on std.image? Sure. Best, Mike
Jun 12 2014
On Thursday, 12 June 2014 at 15:46:12 UTC, Mike wrote:On Thursday, 12 June 2014 at 00:20:28 UTC, cal wrote:I'm looking over your code ATM but I'd like to reply to this first. IMO it's not a good idea to create something like std.image. The std lib should ideally never have breaking changes, so it's easy to get stuck with a sub optimal API or become increasingly hard to maintain. We have Dub. Better keep the std lib lean and maintainable. If you're looking to create an awesome idiomatic D image library you're probably better of building it on top of derelict-devil or derelict-freeimage, then publish it on code.dlang.org The discoverability of good code.dlang.org projects is still limited. Some kind of rating or like system would be useful, but that's another discussion.Might it be worth stitching things together into a proper image processing package?Well I started working on TGA because I was disappointed that no image abstraction is present in Phobos. Go has some imaging APIs and I think D would benefit from having one too (out of the box). Would I work on std.image? Sure. Best, Mike
Jun 12 2014
Le 12/06/2014 20:09, Rene Zwanenburg a écrit :On Thursday, 12 June 2014 at 15:46:12 UTC, Mike wrote:I think it can be a great advantage to have some things like image management in phobos, cause often dub projects are big and users don't want necessary a complete multimedia library but just small pieces that are standard. For example a GUI library, will allow image manipulations, but not only, and extracting only the image modules can be hard. That the case of our DQuick project. For a project like DQuick, I would be happy to find things like images, geometric algebra,environment analysis,... in phobos. This will allow use to be focused on other things making an UI framework (Windows, events, rendering, resource management,...). Having such minimalistic APIs would be a great benefit IMO, maybe in this case some extending libraries would appear in dub.On Thursday, 12 June 2014 at 00:20:28 UTC, cal wrote:I'm looking over your code ATM but I'd like to reply to this first. IMO it's not a good idea to create something like std.image. The std lib should ideally never have breaking changes, so it's easy to get stuck with a sub optimal API or become increasingly hard to maintain. We have Dub. Better keep the std lib lean and maintainable. If you're looking to create an awesome idiomatic D image library you're probably better of building it on top of derelict-devil or derelict-freeimage, then publish it on code.dlang.org The discoverability of good code.dlang.org projects is still limited. Some kind of rating or like system would be useful, but that's another discussion.Might it be worth stitching things together into a proper image processing package?Well I started working on TGA because I was disappointed that no image abstraction is present in Phobos. Go has some imaging APIs and I think D would benefit from having one too (out of the box). Would I work on std.image? Sure. Best, Mike
Jun 12 2014
Le 12/06/2014 20:35, Xavier Bigand a écrit :Le 12/06/2014 20:09, Rene Zwanenburg a écrit :I am thinking an other benefit is what you see as a default, the assurance of D standard conformances (portability, safety, quality, support,...).On Thursday, 12 June 2014 at 15:46:12 UTC, Mike wrote:I think it can be a great advantage to have some things like image management in phobos, cause often dub projects are big and users don't want necessary a complete multimedia library but just small pieces that are standard. For example a GUI library, will allow image manipulations, but not only, and extracting only the image modules can be hard. That the case of our DQuick project. For a project like DQuick, I would be happy to find things like images, geometric algebra,environment analysis,... in phobos. This will allow use to be focused on other things making an UI framework (Windows, events, rendering, resource management,...). Having such minimalistic APIs would be a great benefit IMO, maybe in this case some extending libraries would appear in dub.On Thursday, 12 June 2014 at 00:20:28 UTC, cal wrote:I'm looking over your code ATM but I'd like to reply to this first. IMO it's not a good idea to create something like std.image. The std lib should ideally never have breaking changes, so it's easy to get stuck with a sub optimal API or become increasingly hard to maintain. We have Dub. Better keep the std lib lean and maintainable. If you're looking to create an awesome idiomatic D image library you're probably better of building it on top of derelict-devil or derelict-freeimage, then publish it on code.dlang.org The discoverability of good code.dlang.org projects is still limited. Some kind of rating or like system would be useful, but that's another discussion.Might it be worth stitching things together into a proper image processing package?Well I started working on TGA because I was disappointed that no image abstraction is present in Phobos. Go has some imaging APIs and I think D would benefit from having one too (out of the box). Would I work on std.image? Sure. Best, Mike
Jun 12 2014
I can shed some light on my, beginners, point of view and the rationale behind this tiny library. I want to create a bar/matrix-code generator (QR, DataMatrix etc.). I really like graphics-related subjects and I thought this would be a good start in D. Coming from Java island and having experience in Go I expected to find some basic imaging functionalities in the standard library: not necessary support for all image formats, but at least some bitmap i/o and data model (Pixel, Image, Filter ...). I found none of that :-( IMHO (one of the) pain(s) of C++ is that the stdlib is far behind what a modern developer would consider "elementary". But I expected something from D, because Phobos is already more than C++'s stdlib is: Phobos has e.g. digests. IMO digests are not (as) necessary for a standard library as, say, a qsort is. So if there are digests, why not imaging? Either way, D is really nice and if I can help, I will :-) So far - the TGA lib. Best, Mike
Jun 12 2014
On Wednesday, 11 June 2014 at 18:29:27 UTC, Mike wrote:Here's the link to the repo: http://bit.ly/1mIuGhvI usually don't trust shortened URL's. Can you please post full URL's when not constrained by a character limit?Any feedback would be great!First of all, I like your coding style. It's nice and readable :) https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d All those functions can be marked pure. Pure is D is a bit different than in other languages. I can recommend the following article as a good explanation: http://klickverbot.at/blog/2012/05/purity-in-d/ Also, depending on your preference, you may want to put those functions in a pure nothrow { } block, or put pure nothrow: at the top of the source file. I'm not sure if all those functions can be nothrow though. https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L8 These array literals will be allocated every time the function is called. 'only' can be used instead: only(16, 32).canFind // etc.. http://dlang.org/library/std/range/only.html I remember a function which does something like only only + canFind on one go. It would look something like header.colorMapDepth.among(16, 32); but I can't find it right now.. Maybe it was only proposed but never added. https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L101 This has a pretty bad time complexity. In the worst case this is O(n^2), I think. You could use a hashmap to build the table, then convert to an array. Or even only use hashmaps as color tables internally to improve performance across the board. https://github.com/emesx/TGA/blob/develop/source/tga/model/types.d#L11 Depending on your taste, a union + anonymous struct could be used: struct Pixel { union { ubyte[4] bytes; struct { ubyte b, g, r, a; } } } https://github.com/emesx/TGA/blob/develop/source/tga/model/validation.d#L47 to!string is so common it has a special alias: text. It's placed in std.conv so you still need to import that. "Invalid color map pixel depth: " ~ header.colorMapDepth.text https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L24 Yeah, we need something to read an entire struct from a file and convert it to the correct endianness.. https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L24 https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L91 Unnecessary GC allocation in a hot loop == :( Make a buffer outside the loop and call the normal rawRead: auto buffer = new ubyte[colorMapByteDepth]; foreach(i; 0 .. (header.colorMapLength - header.colorMapOffset)) { file.rawRead(buffer); colorMap[i] = pixelReader(buffer); } Even better, use a static array with the max byte depth, then slice to the correct length: ubyte[MaxByteDepth] buffer; foreach(i; 0 .. (header.colorMapLength - header.colorMapOffset)) { colorMap[i] = file.rawRead(buffer[0 .. colorMapByteDepth]).pixelReader; } Do the same thing here: https://github.com/emesx/TGA/blob/develop/source/tga/io/utils.d#L5 T read(T)(File file) if(isNumeric!T){ ubyte[T.sizeof] bytes; file.rawRead(s[]); return littleEndianToNative!T(bytes); } https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L144 Depending on your taste. The first is built in, the other needs std.algorithm. pixel.bytes[] = chunk[]; chunk.copy(pixel.bytes); https://github.com/emesx/TGA/blob/develop/source/tga/io/utils.d#L40 There's a version in std.algorithm: http://dlang.org/phobos/std_algorithm.html#min I need to go. Please don't mind any typo's and untested code ;). Didn't take a look at writers yet, readers and util need some more scrutiny, but the main theme is: eliminate unnecessary temporary GC allocations.
Jun 12 2014
On Thursday, 12 June 2014 at 19:30:06 UTC, Rene Zwanenburg wrote:I need to go. Please don't mind any typo's and untested code ;). Didn't take a look at writers yet, readers and util need some more scrutiny, but the main theme is: eliminate unnecessary temporary GC allocations.Thank you for feedback, hoping for more! I will apply the suggestions tonight.
Jun 13 2014
On 06/12/2014 09:30 PM, Rene Zwanenburg wrote:I remember a function which does something like only only + canFind on one go. It would look something like header.colorMapDepth.among(16, 32); but I can't find it right now.. Maybe it was only proposed but never added.http://dlang.org/library/std/algorithm/among.html -- Mike Wey
Jun 13 2014
On Friday, 13 June 2014 at 21:09:23 UTC, Mike Wey wrote:On 06/12/2014 09:30 PM, Rene Zwanenburg wrote:Will apply this as well, thanks :-)I remember a function which does something like only only + canFind on one go. It would look something like header.colorMapDepth.among(16, 32); but I can't find it right now.. Maybe it was only proposed but never added.http://dlang.org/library/std/algorithm/among.html
Jun 15 2014
I have refactored the code as recommended. I have also modified the not-yet-reviewed writers part to take advantage of the same approach (preallocated static-sized buffer) rather than allocate slices in loops. Hoping to hear something from you guys! Best, Mike
Jun 16 2014
On Monday, 16 June 2014 at 19:42:14 UTC, Mike wrote:I have refactored the code as recommended. I have also modified the not-yet-reviewed writers part to take advantage of the same approach (preallocated static-sized buffer) rather than allocate slices in loops. Hoping to hear something from you guys! Best, MikeI've been pretty busy this weekend, so sorry for the late reply. I'm taking another look right now.
Jun 16 2014
On Monday, 16 June 2014 at 19:42:14 UTC, Mike wrote:I have refactored the code as recommended. I have also modified the not-yet-reviewed writers part to take advantage of the same approach (preallocated static-sized buffer) rather than allocate slices in loops. Hoping to hear something from you guys! Best, Mikehttps://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L80 Fully qualified names are only necessary when there's a name conflict. The std.algorithm can be removed. https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L8 It's a shame the result of among can't be returned directly but casts are terribly heavy handed. They're best reserved for when you're poking holes in the type system ;) For safe conversions use to!something: return to!bool(isColorMapped(header) ? header.colorMapDepth.among(16, 32) : header.pixelDepth.among(16, 32)); https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L29 This one depends on taste, but these helpers can be eliminated by changing the Header definition a little. It's the same union / anonymous struct trick from the previous post, only this time with bitfields: http://dpaste.dzfl.pl/13258b0ce0c4 https://github.com/emesx/TGA/blob/develop/source/tga/model/utils.d#L92 Hashtable lookup would indeed be nice here, but if you ever need a linear lookup it's included in std.algorithm: http://dlang.org/library/std/algorithm/countUntil.html const index = colorMap.countUntil(pixel); enforce(index >= 0, "Pixel color not found in color map: " ~ pixel.text); return index.to!ushort; Note the use of to!ushort to avoid the cast. to!ushort also performs overflow checking. https://github.com/emesx/TGA/blob/develop/source/tga/model/validation.d#L52 This is one of the many places where the allowed bitdepths are enumerated in an array / among(). Putting the allowed bitdepths in an enum has some advantages: use of '.max' for buffer sizes, EnumMembers in conjunction with among and only, to! checking for correctness... Some examples here: http://dpaste.dzfl.pl/30d2a593468f But it depends on your taste. Personally I prefer to use strong typing like this whenever possible. Helps to find bugs earlier, and makes maintenance easier when your software and data format are still in flux. https://github.com/emesx/TGA/blob/develop/source/tga/model/validation.d#L39 This should be checked both ways. The current method will not detect an error when the image type is not mapped but a color map is present. At least I assume that is not a valid TGA file? enforce(header.colorMapDepth.among(EnumMembers!ColorMapDepth)); enforce(header.isColorMapped == (header.colorMapType == ColorMapType.PRESENT)); enforce(header.isColorMapped == (header.colorMapDepth != ColorMapDepth.BPP0)); https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L95 You could change this to foreach(ref pixel; pixels) { pixel = ... } https://github.com/emesx/TGA/blob/develop/source/tga/io/readers.d#L165 Readability can be improved using the union + bitfields thingy. It's a bit longer but much easier on the eyes IMO: http://dpaste.dzfl.pl/77ecbafa1e0e https://github.com/emesx/TGA/blob/develop/source/tga/io/writers.d#L23 The with statement can be used here: with(image.header) { write(file, idLength); ... } I think this is everything I can come up with. The writeCompressed function can probably be simplified using std.algorithm but it's not straightforward. I'll need to ponder it for a while.
Jun 16 2014
On Monday, 16 June 2014 at 23:04:33 UTC, Rene Zwanenburg wrote:This one depends on taste, but these helpers can be eliminated by changing the Header definition a little. It's the same union / anonymous struct trick from the previous post, only this time with bitfields: http://dpaste.dzfl.pl/13258b0ce0c4If you define nested structs/unions, it's better to make them static whenever possible, because non-static nested structs have an additional hidden field for the context. Also, when you work with bit-(un)packing, you need to make sure it works on big- and little-endian CPUs.
Jun 17 2014
On Tuesday, 17 June 2014 at 13:07:33 UTC, Marc Schütz wrote:On Monday, 16 June 2014 at 23:04:33 UTC, Rene Zwanenburg wrote:Good call. In this case it doesn't add the hidden field, but for the sake of consistency I agree it's better to add static.This one depends on taste, but these helpers can be eliminated by changing the Header definition a little. It's the same union / anonymous struct trick from the previous post, only this time with bitfields: http://dpaste.dzfl.pl/13258b0ce0c4If you define nested structs/unions, it's better to make them static whenever possible, because non-static nested structs have an additional hidden field for the context.Also, when you work with bit-(un)packing, you need to make sure it works on big- and little-endian CPUs.Yeah. He's using a custom read function that takes endianness into account, so I didn't mention it.
Jun 18 2014
On Wednesday, 18 June 2014 at 14:05:12 UTC, Rene Zwanenburg wrote:On Tuesday, 17 June 2014 at 13:07:33 UTC, Marc Schütz wrote:Interesting, I thought it's there for all nested structs. But I see it's only added if the struct has a method, but then it's always there, no matter whether the methods actually accesses the outer context. Probably for consistency reasons...On Monday, 16 June 2014 at 23:04:33 UTC, Rene Zwanenburg wrote:Good call. In this case it doesn't add the hidden field, but for the sake of consistency I agree it's better to add static.This one depends on taste, but these helpers can be eliminated by changing the Header definition a little. It's the same union / anonymous struct trick from the previous post, only this time with bitfields: http://dpaste.dzfl.pl/13258b0ce0c4If you define nested structs/unions, it's better to make them static whenever possible, because non-static nested structs have an additional hidden field for the context.
Jun 19 2014
On Thursday, 19 June 2014 at 12:06:58 UTC, Marc Schütz wrote:On Wednesday, 18 June 2014 at 14:05:12 UTC, Rene Zwanenburg wrote:The absence of the hidden field without methods is probably to maintain C compitability; code that is both valid C and D must have the same semantics. I think that includes struct layout?On Tuesday, 17 June 2014 at 13:07:33 UTC, Marc Schütz wrote:Interesting, I thought it's there for all nested structs. But I see it's only added if the struct has a method, but then it's always there, no matter whether the methods actually accesses the outer context. Probably for consistency reasons...On Monday, 16 June 2014 at 23:04:33 UTC, Rene Zwanenburg wrote:Good call. In this case it doesn't add the hidden field, but for the sake of consistency I agree it's better to add static.This one depends on taste, but these helpers can be eliminated by changing the Header definition a little. It's the same union / anonymous struct trick from the previous post, only this time with bitfields: http://dpaste.dzfl.pl/13258b0ce0c4If you define nested structs/unions, it's better to make them static whenever possible, because non-static nested structs have an additional hidden field for the context.
Jun 19 2014
Once again thanks for the feedback. The RLE rewrite using std is impressive! I will use it today or tomorrow, great stuff.
Jun 19 2014
Do you think it's ready for a v0.1 release? It be willing to add it to dub if it passes general D coding stardards. Once again, thanks for feedback and vast improvements of the code. Best, Mike
Jun 20 2014
On Friday, 20 June 2014 at 11:15:20 UTC, Mike wrote:Do you think it's ready for a v0.1 release? It be willing to add it to dub if it passes general D coding stardards.Sure, go for it :). Dub package versioning should be done using SemVer so there are some restrictions on how to number a release. http://semver.org/
Jun 21 2014
Thanks, will work on fixes tonight.The current method will not detect an error when the image type is not mapped but a color map is present. At least I assume that is not a valid TGA file?A non-mapped image may contain a color map ;-0 it's simply discarded. As per your ideasreturn to!bool(isColorMapped(header) ? header.colorMapDepth.among(16, 32) : header.pixelDepth.among(16, 32));this is not nothrow, but as you can see there is nothing that can ever throw.. so perhaps I'll stick to the cast(bool) thing.. Any idea?
Jun 17 2014
On Tuesday, 17 June 2014 at 18:35:34 UTC, Mike wrote:Thanks, will work on fixes tonight.Perhaps `return !!(...);` ?The current method will not detect an error when the image type is not mapped but a color map is present. At least I assume that is not a valid TGA file?A non-mapped image may contain a color map ;-0 it's simply discarded. As per your ideasreturn to!bool(isColorMapped(header) ? header.colorMapDepth.among(16, 32) : header.pixelDepth.among(16, 32));this is not nothrow, but as you can see there is nothing that can ever throw.. so perhaps I'll stick to the cast(bool) thing.. Any idea?
Jun 18 2014
On Wednesday, 18 June 2014 at 09:41:01 UTC, Marc Schütz wrote:On Tuesday, 17 June 2014 at 18:35:34 UTC, Mike wrote:testThanks, will work on fixes tonight.Perhaps `return !!(...);` ?The current method will not detect an error when the image type is not mapped but a color map is present. At least I assume that is not a valid TGA file?A non-mapped image may contain a color map ;-0 it's simply discarded. As per your ideasreturn to!bool(isColorMapped(header) ? header.colorMapDepth.among(16, 32) : header.pixelDepth.among(16, 32));this is not nothrow, but as you can see there is nothing that can ever throw.. so perhaps I'll stick to the cast(bool) thing.. Any idea?
Jun 18 2014
On Wednesday, 18 June 2014 at 13:42:32 UTC, Rene Zwanenburg wrote:On Wednesday, 18 June 2014 at 09:41:01 UTC, Marc Schütz wrote:The web interface didn't work for me for about a day. It seems to be over now. Yeah I didn't take nothrow into account. AFAIK using !!() to convert to bool in a context where it doesn't happen implicitly is indeed idiomatic. At least it's used in Phobos.On Tuesday, 17 June 2014 at 18:35:34 UTC, Mike wrote:testThanks, will work on fixes tonight.Perhaps `return !!(...);` ?The current method will not detect an error when the image type is not mapped but a color map is present. At least I assume that is not a valid TGA file?A non-mapped image may contain a color map ;-0 it's simply discarded. As per your ideasreturn to!bool(isColorMapped(header) ? header.colorMapDepth.among(16, 32) : header.pixelDepth.among(16, 32));this is not nothrow, but as you can see there is nothing that can ever throw.. so perhaps I'll stick to the cast(bool) thing.. Any idea?
Jun 18 2014
On Monday, 16 June 2014 at 23:04:33 UTC, Rene Zwanenburg wrote:The writeCompressed function can probably be simplified using std.algorithm but it's not straightforward. I'll need to ponder it for a while.Here's a version using std.algorithm and std.range. I think it's easier to understand than the original: http://dpaste.dzfl.pl/88ba2592ac8d
Jun 18 2014