www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - color lib

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
I've done another pass incorporating prior feedback, mostly focusing
on documentation.

http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

Can interested parties please give it another once-over and add
further comments?
How can I get this to a point where people would like to see it in phobos?

Repo: https://github.com/TurkeyMan/color
PR: https://github.com/dlang/phobos/pull/2845
Oct 06 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly 
 focusing on documentation.

 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and add
 further comments?
 How can I get this to a point where people would like to see it 
 in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Could you please make `colorFromString` nothrow nogc? Or make `nothrow nogc` analog. -- Ilya
Oct 06 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly focusing on
 documentation.


 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and add
 further comments?
 How can I get this to a point where people would like to see it in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Could you please make `colorFromString` nothrow nogc? Or make `nothrow nogc` analog. -- Ilya
I think throwing is the precedented action in that failure case... what would you suggest? I could have an overload that returns an error or something...? I wonder how the work towards throwing RC things is going?
Oct 06 2016
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
 I think throwing is the precedented action in that failure case...
 what would you suggest?
revised their preferred mechanism and started offering methods like: static bool TryParse(string str, out DateTime dt) {...}
Oct 06 2016
next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
 I think throwing is the precedented action in that failure case...
 what would you suggest?
revised their preferred mechanism and started offering methods like: static bool TryParse(string str, out DateTime dt) {...}
Indeed, but what should WE do?
Oct 06 2016
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 12:38, Manu <turkeyman gmail.com> wrote:
 On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
 I think throwing is the precedented action in that failure case...
 what would you suggest?
revised their preferred mechanism and started offering methods like: static bool TryParse(string str, out DateTime dt) {...}
Indeed, but what should WE do?
I've rolled with: bool colorFromString(Color = RGB8)(scope const(char)[] str, out Color color) pure nothrow safe nogc
Oct 06 2016
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 12:55, Manu <turkeyman gmail.com> wrote:
 On 7 October 2016 at 12:38, Manu <turkeyman gmail.com> wrote:
 On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
 I think throwing is the precedented action in that failure case...
 what would you suggest?
revised their preferred mechanism and started offering methods like: static bool TryParse(string str, out DateTime dt) {...}
Indeed, but what should WE do?
I've rolled with: bool colorFromString(Color = RGB8)(scope const(char)[] str, out Color color) pure nothrow safe nogc
More updates and doco tweaks pushed.
Oct 06 2016
prev sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Friday, 7 October 2016 at 01:42:08 UTC, Manu wrote:
 On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 [...]
Could you please make `colorFromString` nothrow nogc? Or make `nothrow nogc` analog. -- Ilya
I think throwing is the precedented action in that failure case... what would you suggest? I could have an overload that returns an error or something...? I wonder how the work towards throwing RC things is going?
No idea. The reason to do not throw Exceptions is to be able to use Phobos in betterC mode without DRuntime at all.
Oct 06 2016
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 11:42, Manu <turkeyman gmail.com> wrote:
 On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly focusing on
 documentation.


 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and add
 further comments?
 How can I get this to a point where people would like to see it in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Could you please make `colorFromString` nothrow nogc? Or make `nothrow nogc` analog. -- Ilya
I think throwing is the precedented action in that failure case... what would you suggest? I could have an overload that returns an error or something...?
Problem with overloading is that the function type only differs by return type... is there a phobos convention for function naming where a parallel nothrow nogc version is also supplied?
Oct 06 2016
prev sibling next sibling parent reply Random D user <no email.com> writes:
On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly 
 focusing on documentation.
Just a quick minor comment on: A8 RGB!("a",ubyte,false,0) 8 bit alpha-only color type. --> Reads like, "False what ???". Also "What is 0 ???". --> How about RGB!("a", ubyte, Linear.No, Colorspace.sRGB) or something like that, since there's going to be a list of these in the docs. What does colorspace 0 mean actually? (AdobeRGB??? i.e. first from colorspace enum according the docs).
Oct 06 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 04:48, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly focusing on
 documentation.
Just a quick minor comment on: A8 RGB!("a",ubyte,false,0) 8 bit alpha-only color type. --> Reads like, "False what ???". Also "What is 0 ???". --> How about RGB!("a", ubyte, Linear.No, Colorspace.sRGB) or something like that, since there's going to be a list of these in the docs. What does colorspace 0 mean actually? (AdobeRGB??? i.e. first from colorspace enum according the docs).
It's kind of upsetting that the docs don't show the default arg as the enum it is... The struct is defined: struct RGB(string components_, ComponentType_, bool linear_ = false, RGBColorSpace colorSpace_ = RGBColorSpace.sRGB) I don't know why the docs translate that to be '0'? Why not just omit the default args from the docs as they are in the code: alias A8 = RGB!("a", ubyte); ?? It also seems the docs have rearranged the order of the enum... seriously? Why wouldn't that retain the order the programmer specified? Regarding 'Linear.No'... yeah... I dunno. I've had this argument before. I really hate that pattern. If it's required, I'll do it, otherwise I'm really not enthusiastic about it personally. It just seems like pointless bloat in symbol names to me, is if they weren't already bloated enough.
Oct 06 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
 Regarding 'Linear.No'... yeah... I dunno. I've had this 
 argument before.
 I really hate that pattern. If it's required, I'll do it
At least as far as readability goes, explicit parameterisation lets you understand the invocation at a glance rather than already knowing the actual name of the parameter or having to go elsewhere in code to see the prototype. For a library, I'd favor readability.
Oct 07 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
 Regarding 'Linear.No'... yeah... I dunno. I've had this argument before.
 I really hate that pattern. If it's required, I'll do it
At least as far as readability goes, explicit parameterisation lets you understand the invocation at a glance rather than already knowing the actual name of the parameter or having to go elsewhere in code to see the prototype. For a library, I'd favor readability.
Oh no, you too? >_< Incidentally, have you had a geez over the core API? An efficient API will emerge when we work out how to work batched operations into ranges...
Oct 08 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
 On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
 Regarding 'Linear.No'... yeah... I dunno. I've had this 
 argument before. I really hate that pattern. If it's 
 required, I'll do it
At least as far as readability goes, explicit parameterisation lets you understand the invocation at a glance rather than already knowing the actual name of the parameter or having to go elsewhere in code to see the prototype. For a library, I'd favor readability.
Oh no, you too? >_< Incidentally, have you had a geez over the core API? An efficient API will emerge when we work out how to work batched operations into ranges...
I like true/false here :-)
Oct 08 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 October 2016 at 23:28, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
 On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
 Regarding 'Linear.No'... yeah... I dunno. I've had this argument before.
 I really hate that pattern. If it's required, I'll do it
At least as far as readability goes, explicit parameterisation lets you understand the invocation at a glance rather than already knowing the actual name of the parameter or having to go elsewhere in code to see the prototype. For a library, I'd favor readability.
Oh no, you too? >_< Incidentally, have you had a geez over the core API? An efficient API will emerge when we work out how to work batched operations into ranges...
I like true/false here :-)
I like this guy ;)
Oct 08 2016
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
 Oh no, you too? >_<
 Incidentally, have you had a geez over the core API? An 
 efficient API
 will emerge when we work out how to work batched operations into
 ranges...
How far would `r.inBatchesOf!(N)` go in terms of compiler optimisations (e.g. vectorisation) if N is a power of 2? auto inBatchesOf(size_t N,R)(R r) if(N!=0 &&isInputRange!R && hasLength!R) { struct InBatchesOfN { R r; ElementType!(R)[N] batch; this(R _r) { assert(_r.length % N ==0);// could have overloads where undefined elements == ElementType!(R).init r = _r; foreach( i; 0..N) { batch[i] = r.front; r.popFront; } } bool empty() { return r.empty; } auto front { return batch; } void popFront() { foreach( i; 0..N) { batch[i] = r.front; r.popFront; } } } return InBatchesOfN(r); }
Oct 08 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
 Oh no, you too? >_<
 Incidentally, have you had a geez over the core API? An efficient API
 will emerge when we work out how to work batched operations into
 ranges...
How far would `r.inBatchesOf!(N)` go in terms of compiler optimisations (e.g. vectorisation) if N is a power of 2? auto inBatchesOf(size_t N,R)(R r) if(N!=0 &&isInputRange!R && hasLength!R) { struct InBatchesOfN { R r; ElementType!(R)[N] batch; this(R _r) { assert(_r.length % N ==0);// could have overloads where undefined elements == ElementType!(R).init r = _r; foreach( i; 0..N) { batch[i] = r.front; r.popFront; } } bool empty() { return r.empty; } auto front { return batch; } void popFront() { foreach( i; 0..N) { batch[i] = r.front; r.popFront; } } } return InBatchesOfN(r); }
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Oct 08 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Oct 08 2016
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
anything that is able to be. Given that ElementType!InBatchesOfN are a static array of ElementType!(R), the compiler can* (assuming no branching and anything else that impedes vectorisation) combine most expressions into equivalent vector instruction. This approach might not work so well for colours as is but should work if we "transpose" the colour i.e. rgbargbargbargba -> rrrrggggbbbbaaaa and then transpose it back. *I know this is the sufficiently intelligent compiler argument
Oct 09 2016
next sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 9 October 2016 at 08:39:57 UTC, Nicholas Wilson wrote:
 On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko 
 wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
anything that is able to be. Given that ElementType!InBatchesOfN are a static array of ElementType!(R), the compiler can* (assuming no branching and anything else that impedes vectorisation) combine most expressions into equivalent vector instruction. This approach might not work so well for colours as is but should work if we "transpose" the colour i.e. rgbargbargbargba -> rrrrggggbbbbaaaa and then transpose it back. *I know this is the sufficiently intelligent compiler argument
static foreach can help for static arrays. For example, ndslice uses static foreach a lot. mir.ndslice.algorithm allows to perform vectorized operations. Some conversion algorithms for ndslices will be added to Mir or DCV. Please open a pool request or fill an issue.
Oct 09 2016
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 18:39, Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
anything that is able to be. Given that ElementType!InBatchesOfN are a static array of ElementType!(R), the compiler can* (assuming no branching and anything else that impedes vectorisation) combine most expressions into equivalent vector instruction. This approach might not work so well for colours as is but should work if we "transpose" the colour i.e. rgbargbargbargba -> rrrrggggbbbbaaaa and then transpose it back. *I know this is the sufficiently intelligent compiler argument
I've been waiting for that compiler for almost 2 decades. I've shipped 17 commercial games in that time while waiting, and I had to resort to manual code that didn't yet have an opportunity to leverage such a sufficiently intelligent compilers awesome optimiser. I've never seen an auto-vectoriser go ANYWHERE NEAR the sort of intelligence required here. I've seen it start to do some good work with arrays of floats... that's about where it ends. Arrays of RGBA_10_10_10_2 require unpacking, and then shuffling. When you stuff the unpack in the way, that will almost always throw an auto-vectoriser off the scent. I'm also waiting to see saturation expressions written in C code promote to saturating SIMD arithmetic. Basically every operation is saturating when working with colours.
Oct 09 2016
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Let's consider a super simple blend: dest = src.rgb * src.a + dest.rgb * (1-src.alpha); This is perhaps the most common blend that exists. If this is a ubyte[4] color, which is the most common format, then to do it efficiently, runs of 16 colors (4x ubyte[16] vectors), needs to be rearranged into: ubyte[16][3] rgb = [ [RGBRGBRGBRGBRGBR], [GBRGBRGBRGBRGBRG], [BRGBRGBRGBRGBRGB] ]; ubyte[16][3] a = [ [AAAaaaAAAaaaAAAa], [aaAAAaaaAAAaaaAA], [AaaaAAAaaaAAAaaa] ]; You can do this with gather loads, or with a couple of shuffle's after loading. Then obviously do the work in this configuration. Or you might expand it to [ [RRRRRRRRRRRRRRRR], [GGGGGGGGGGGGGGGG], [BBBBBBBBBBBBBBBB] ], etc, depends on the work, and which expansion is cheaper for the platform (ie, shuffling limitations). Now, this might not look like much of a win for this blend, but as you extend the sequence of ops, the win gets much much bigger. Particularly so if you want to do gamma-correct stuff, which would usually involve expanding those ubytes into floats, then doing vector pow's and stuff like that. Either way, you need to iterate the image 4 vectors at a time. That's the sort of batching I'm talking about. Trouble is, this work needs to be wrapped into a function that receives inputs in batches, like: RGBA8[16] doBulkBlend(RGBA8[16] buffer) { ... bulk blend code ... } This sort of thing: buffer.map!(e => src.rgb * src.a).copy(output); Super readable! Would be really nice to express, but I have no idea how we can make that sort of thing efficient. You could start writing this sort of thing: buffer.chunksOf!16.map!(e => doBulkBlend(e[0..16])).deChunk.copy(output); Yeah, it's code... it would compile, but I consider that to be completely obfuscated. You can't look at that and understand anything much about what it does... so I don't think that's a good goal-post at all. If I showed that to a colleague, I don't think they'd be impressed. We can't reach that point and say D is awesome for data-stream processing... we need to go a lot further than that. Anyway, I think this sort of thing is a minimum target. I'd like to see how this sort of batching would integrate into ndslice nicely, because it introduces the 'nd' iteration element... there's a heap of challenges; element alignment, unaligned line-strides, mid-vector slices, etc. I can imagine certain filter algorithms that work on 2d slices ('blocks') rather than 1d slices like the example above. What if the image is rotated or transposed? How can applying a per-pixel operation on a buffer iterate the memory-linear fashion even though it's working in batched of elements? I haven't sat and tried plugging this into ndslice much yet. Haven't had enough time, and I really wanted to get colour to a point I'm happy with.
Oct 09 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 9 October 2016 at 13:18:22 UTC, Manu wrote:
 On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Let's consider a super simple blend: dest = src.rgb * src.a + dest.rgb * (1-src.alpha);
This code do not need transposition. And can be partially vectorised using mir.ndslice.algorithm. To perform full vectorization image should be regrouped in memory channels. For example many computer vision algorithms work better with each channel regrouped in memory channels. Relevant issue for this type of optmizations: https://github.com/libmir/mir/issues/343 Please comment on this issue and provide a set of functions you would like to be vectorised. --Ilya
Oct 09 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 23:36, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 13:18:22 UTC, Manu wrote:
 On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Let's consider a super simple blend: dest = src.rgb * src.a + dest.rgb * (1-src.alpha);
This code do not need transposition. And can be partially vectorised using mir.ndslice.algorithm.
How so? Sorry, let me write the actual work in full: RGBA8[] src, dest; // allocate somehow zip(src, dest).map!((e) { ubyte r = cast(ubyte)clamp(((cast(int)e[0].r * e[0].a * 0x1011) >> 20) + ((cast(int)e[1].r * (255 - e[0].a) * 0x1011) >> 20), 0, 255); ubyte g = cast(ubyte)clamp(((cast(int)e[0].g * e[0].a * 0x1011) >> 20) + ((cast(int)e[1].g * (255 - e[0].a) * 0x1011) >> 20), 0, 255); ubyte b = cast(ubyte)clamp(((cast(int)e[0].b * e[0].a * 0x1011) >> 20) + ((cast(int)e[1].b * (255 - e[0].a) * 0x1011) >> 20), 0, 255); return RGBA8(r, g, b, e[0].a); }).copy(dest); If you can coerce the auto-vectoriser to do the right thing with that, I'll be shocked.
 To perform full vectorization image should be regrouped in memory channels.
 For example many computer vision algorithms work better with each channel
 regrouped in memory channels.
Yes, but many (most) applications just want to call a function, and don't want to rearrange their data. Most people aren't researchers writing high-tech computer vision software ;) Also, most images in realtime software are textures, which are usually compressed in some way, and don't generally work well split into channels; that would be multiple textures (for each plane), and multiple samples per texel (sample for each plane). Image processing definitely needs to work without having the working dataset pre-arranged into planes.
 Relevant issue for this type of optmizations:
 https://github.com/libmir/mir/issues/343

 Please comment on this issue and provide a set of functions you would like
 to be vectorised. --Ilya
I'll take a look.
Oct 09 2016
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Even operations that don't require shuffling, eg: RGBA8[] a, b; zip(a, b).map!(e => e[0] + e[1]).copy(output); Which I've suggested before (and Walter liked the idea), could be sugared up by making use of the languages largely under-used array operation syntax: output[] = a[] + b[]; // ie, types have overloaded addition operators, so this array expression would be lowered to the pipeline expression above. This would be super-useful for HEAPS of things! Even these still need to be done in batches since colour adds are saturating operations, and there are SIMD instructions for saturating arithmetic, so we basically always have to do colour work in SIMD, which means batching, and that basically ruins any chance for natural, readable, expressions in your code. I want to find a way that we can express these operations naturally, without having to always manually handle the batching. If we can get there, then I will say D is a good language for stream-data processing.
Oct 09 2016
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 9 October 2016 at 13:28:05 UTC, Manu wrote:
 On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
Could you please give an example what type of operation should be vectorized?
Even operations that don't require shuffling, eg: RGBA8[] a, b; zip(a, b).map!(e => e[0] + e[1]).copy(output); Which I've suggested before (and Walter liked the idea), could be sugared up by making use of the languages largely under-used array operation syntax: output[] = a[] + b[]; // ie, types have overloaded addition operators, so this array expression would be lowered to the pipeline expression above. This would be super-useful for HEAPS of things! Even these still need to be done in batches since colour adds are saturating operations, and there are SIMD instructions for saturating arithmetic, so we basically always have to do colour work in SIMD, which means batching, and that basically ruins any chance for natural, readable, expressions in your code. I want to find a way that we can express these operations naturally, without having to always manually handle the batching. If we can get there, then I will say D is a good language for stream-data processing.
What about forwarding the array ops to a foreach of the static array? Like as above but instead of: ElementType!(R)[N] batch; have: static struct Batch { ElementType!(R)[N] elements; auto get() { return elements[];} Batch opBinary(string op)(Batch rhs) if(hasOperator!(ElementType!(R),op)) { Batch b; foreach(i; iota(N)) mixin("b.elements[i] = elememts[i]" ~op~"rhs.elements[i]"); return b; } //repeat for opUnary,opOpAssign... } Batch batch; I'll make another forum thread for this.
Oct 10 2016
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 11 October 2016 at 00:10:04 UTC, Nicholas Wilson 
wrote:
 What about forwarding the array ops to a foreach of the static 
 array?
 Like as above but instead of:

 ElementType!(R)[N] batch;

 have:

 static struct Batch
 {
     ElementType!(R)[N] elements;
     auto get() { return elements[];}

     Batch opBinary(string op)(Batch rhs) 
 if(hasOperator!(ElementType!(R),op))
     {
         Batch b;
         foreach(i; iota(N)) mixin("b.elements[i] = elememts[i]" 
 ~op~"rhs.elements[i]");
         return b;
     }
     //repeat for opUnary,opOpAssign...
 }
 Batch batch;

 I'll make another forum thread for this.
whoops missed an alias get this;
Oct 10 2016
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 How far would `r.inBatchesOf!(N)` go in terms of compiler 
 optimisations
 (e.g. vectorisation) if N is a power of 2?

 auto inBatchesOf(size_t N,R)(R r) if(N!=0 &&isInputRange!R &&  
 hasLength!R)
 {
     struct InBatchesOfN
     {
         R r;
         ElementType!(R)[N] batch;
         this(R _r)
         {
              assert(_r.length % N ==0);// could have overloads 
 where
 undefined elements == ElementType!(R).init
              r = _r;
              foreach( i; 0..N)
              {
                   batch[i] = r.front;
                   r.popFront;
              }
         }

         bool empty() { return r.empty; }
         auto front { return batch; }
         void popFront()
         {
              foreach( i; 0..N)
              {
                   batch[i] = r.front;
                   r.popFront;
              }
         }
     }

     return InBatchesOfN(r);
 }
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
How? All you need is an extra `each` e.g. r.inBatchesOf!(8).each!(a =>a[].map!(convertColor!RGBA8)) perhaps define a helper function for it that does each + the explicit slice + map, but it certainly doesn't scream completely different API to me.
Oct 09 2016
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 9 October 2016 at 08:25:40 UTC, Nicholas Wilson wrote:
 How?  All you need is an extra `each` e.g. 
 r.inBatchesOf!(8).each!(a =>a[].map!(convertColor!RGBA8))

 perhaps define a helper function for it that does each + the 
 explicit slice + map, but it certainly doesn't scream 
 completely different API to me.
Ha, realised I went full circle. Still might be useful if the compiler is able to use the fact that the range is a multiple of N (particularly is N is a power of 2). Are you able to apply arbitrary attributes to the delegate passed (i.e. things like fastmath)?
Oct 09 2016
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 October 2016 at 18:25, Nicholas Wilson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 How far would `r.inBatchesOf!(N)` go in terms of compiler optimisations
 (e.g. vectorisation) if N is a power of 2?

 auto inBatchesOf(size_t N,R)(R r) if(N!=0 &&isInputRange!R &&
 hasLength!R)
 {
     struct InBatchesOfN
     {
         R r;
         ElementType!(R)[N] batch;
         this(R _r)
         {
              assert(_r.length % N ==0);// could have overloads where
 undefined elements == ElementType!(R).init
              r = _r;
              foreach( i; 0..N)
              {
                   batch[i] = r.front;
                   r.popFront;
              }
         }

         bool empty() { return r.empty; }
         auto front { return batch; }
         void popFront()
         {
              foreach( i; 0..N)
              {
                   batch[i] = r.front;
                   r.popFront;
              }
         }
     }

     return InBatchesOfN(r);
 }
Well the trouble is the lambda that you might give to 'map' won't work anymore. Operators don't work on batches, you need to use a completely different API, and I think that's unfortunate.
How? All you need is an extra `each` e.g. r.inBatchesOf!(8).each!(a =>a[].map!(convertColor!RGBA8)) perhaps define a helper function for it that does each + the explicit slice + map, but it certainly doesn't scream completely different API to me.
As you demonstrate; convertColor doesn't accept RGBA8[16], it accepts a single RGBA8... there's no way the optimiser will be able to magic-up an efficient inline of convertColor which works with 16 elements at a time, but I could easily write a super-fast version by hand. My point about the separate API is, any function that works on a single element would need a compliment of functions that work on 'n' elements, where 'n' is some context-specific number of elements that suits that particular workload. Now, that's conceivable, and it's even possible to make the magic meta that calls these functions work out there is a batch overload and call it if it can, but we're miles away from std.algorithm and common ranges now. The other issue is that every such efficient batch version would need to be hand-written, and that sucks because there are too many permutations.
Oct 09 2016
prev sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
 Oh no, you too? >_<
Yeah, I've been going on a readability bender lately, especially in public facing code. My thinking there is that statements in code that don't immediately give context are essentially a cipher. Because that's exactly what you need to do to understand the code - look something up to see what it means. Named parameters and variable names that provide the context avoid that to a large degree. I'm especially trying to make Binderoo readable as there's so many programmers that are scared by metaprogramming. My GDCE talk spent a lot of time attempting to make it all understandable. Making the code descriptive seals the deal. If I can make my code more descriptive, and it compiles out just the same but makes the compiler do a bit more work... Make the compiler do more work and optimise the compiler. I'm far more lax on not-publicly-facing code (so basically API implementations and supporting code that isn't part of a public interface). Anything I expect someone other than myself to interact with gets the readability treatment. Which, as you know, is important because readable code generally isn't efficient code - as is evidenced by the vectorisation/buffer processing thread going on in here. It's also interesting how many programmers get vehemently defensive when you call out non-descriptive programming practices as programming for your own convenience and no one else. I have this argument with using i/j/k/foo/bar/etc in loops as well.
 Incidentally, have you had a geez over the core API? An 
 efficient API
 will emerge when we work out how to work batched operations into
 ranges...
Been slowly making my way through it. Seems solid enough, but I haven't looked through it all thoroughly yet.
Oct 10 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-10-10 12:39, Ethan Watson wrote:

 I'm especially trying to make Binderoo readable as there's so many
 programmers that are scared by metaprogramming.
Isn't std.typecons.Flag metaprogramming ;) -- /Jacob Carlborg
Oct 10 2016
parent Ethan Watson <gooberman gmail.com> writes:
On Monday, 10 October 2016 at 12:10:56 UTC, Jacob Carlborg wrote:
 Isn't std.typecons.Flag metaprogramming ;)
Hahaha, oh wow. If ever there was a case for mixins.
Oct 10 2016
prev sibling parent reply Andrea Fontana <nospam example.com> writes:
On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly 
 focusing on documentation.

 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and add
 further comments?
 How can I get this to a point where people would like to see it 
 in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Nice work! colorFromString should be colorFromRGBString :)
Oct 10 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 October 2016 at 17:29, Andrea Fontana via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly focusing on
 documentation.


 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and add
 further comments?
 How can I get this to a point where people would like to see it in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Nice work! colorFromString should be colorFromRGBString :)
Nar. It parses any form of colour-in-a-string.
Oct 10 2016
parent reply Andrea Fontana <nospam example.com> writes:
On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:
 On 10 October 2016 at 17:29, Andrea Fontana via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
 I've done another pass incorporating prior feedback, mostly 
 focusing on documentation.


 http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

 Can interested parties please give it another once-over and 
 add
 further comments?
 How can I get this to a point where people would like to see 
 it in phobos?

 Repo: https://github.com/TurkeyMan/color
 PR: https://github.com/dlang/phobos/pull/2845
Nice work! colorFromString should be colorFromRGBString :)
Nar. It parses any form of colour-in-a-string.
From doc: colorFromString Create a color from a string. May be a hex color the name of any color from the Colors enum. It seems it reads just rgb. (+ enum) I think that: colorFromString("red"); colorFromString!"bgra"("#ff1232dd"); makes more sense. Andrea
Oct 10 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:

 From doc:
 colorFromString Create a color from a string. May be a hex color in the

 color from the Colors enum.

 It seems it reads just rgb. (+ enum)

 I think that:
 colorFromString("red");

 colorFromString!"bgra"("#ff1232dd");

 makes more sense.

 Andrea
Why? I see no value in that function being a template... It's not like you can confuse "#FF0080" and "LightGoldenrodYellow". As far as I know, there's no possible ambiguity in colour strings, so why make them separate functions?
Oct 10 2016
parent reply Andrea Fontana <nospam example.com> writes:
On Monday, 10 October 2016 at 13:25:07 UTC, Manu wrote:
 On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:

 From doc:
 colorFromString Create a color from a string. May be a hex 
 color in the

 name of any
 color from the Colors enum.

 It seems it reads just rgb. (+ enum)

 I think that:
 colorFromString("red");

 colorFromString!"bgra"("#ff1232dd");

 makes more sense.

 Andrea
Why? I see no value in that function being a template... It's not like you can confuse "#FF0080" and "LightGoldenrodYellow". As far as I know, there's no possible ambiguity in colour strings, so why make them separate functions?
But it would be useful to create rgb, bgr, argb, bgra, or other color space using a string. If a third party library or source gives me code in rgba, I have to preprocess it to convert as argb and then pass it to your library. Anyway, I don't know if a code with letters a-f can be composed. In that case an ambiguity exists.
Oct 10 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 October 2016 at 23:41, Andrea Fontana via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 October 2016 at 13:25:07 UTC, Manu wrote:
 On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:

 From doc:
 colorFromString Create a color from a string. May be a hex color in the

 color from the Colors enum.

 It seems it reads just rgb. (+ enum)

 I think that:
 colorFromString("red");

 colorFromString!"bgra"("#ff1232dd");

 makes more sense.

 Andrea
Why? I see no value in that function being a template... It's not like you can confuse "#FF0080" and "LightGoldenrodYellow". As far as I know, there's no possible ambiguity in colour strings, so why make them separate functions?
But it would be useful to create rgb, bgr, argb, bgra, or other color space using a string.
Give the preferred format as template arg?
 If a third party library or source gives me code in rgba, I have to
 preprocess it to convert as argb and then pass it to your library.
Sorry, what are we talking about? My lib supports basically every format or arrangement I've ever encountered... you can work with practically any data format you can think of.
 Anyway, I don't know if a code with letters a-f can be composed. In that
 case an ambiguity exists.
It must also be exactly 3,4,6,8 letters long, and begin with the I'm not sure why it matters what format the colour you have is... Strings are in the form #RRGGBB, or #AARRGGBB. That is all. It's the standard I've seen used everywhere ever, including the web, which is a pretty good precedent :P If you support swizzled forms of strings, then ambiguity exists. Better not to allow it. If you want a BGR from a string, use: colorFromString!BGR8("#FF0000"); If you want Lab: colorFromString!(Lab!float)("#FF0000");
Oct 10 2016
parent reply Andrea Fontana <nospam example.com> writes:
On Monday, 10 October 2016 at 23:26:53 UTC, Manu wrote:
 I'm not sure why it matters what format the colour you have 
 is...
 Strings are in the form #RRGGBB, or #AARRGGBB. That is all.
 It's the standard I've seen used everywhere ever, including the 
 web,
 which is a pretty good precedent :P
If the web is a good precedent (CSS4 specs): "The first 6 digits are interpreted identically to the 6-digit notation. The last pair of digits, interpreted as a hexadecimal number, specifies the alpha channel of the color, where 00 represents a fully transparent color and ff represent a fully opaque color." https://drafts.csswg.org/css-color/#hex-notation CSS3 doesn't support hex string with alpha but they suggest you to use rgba() function. I think argb() doesn't exists instead. https://www.w3.org/TR/2011/REC-css3-color-20110607/#rgba-color Chrome 52 supports it: https://googlechrome.github.io/samples/css-alpha-channel/ Android instead: https://developer.android.com/reference/android/graphics/Color.html#parseColor(java.lang.String) "Parse the color string, and return the corresponding color-int. If the string cannot be parsed, throws an IllegalArgumentException exception. Supported formats are: #RRGGBB #AARRGGBB [...]" Please notice that on PNG file format rgba is quite common (also on bmp with semi-official apha support) PNG: http://www.libpng.org/pub/png/book/chapter08.html
Oct 11 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 October 2016 at 18:10, Andrea Fontana via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 10 October 2016 at 23:26:53 UTC, Manu wrote:
 I'm not sure why it matters what format the colour you have is...
 Strings are in the form #RRGGBB, or #AARRGGBB. That is all.
 It's the standard I've seen used everywhere ever, including the web,
 which is a pretty good precedent :P
If the web is a good precedent (CSS4 specs): "The first 6 digits are interpreted identically to the 6-digit notation. The last pair of digits, interpreted as a hexadecimal number, specifies the alpha channel of the color, where 00 represents a fully transparent color and ff represent a fully opaque color." https://drafts.csswg.org/css-color/#hex-notation CSS3 doesn't support hex string with alpha but they suggest you to use rgba() function. I think argb() doesn't exists instead. https://www.w3.org/TR/2011/REC-css3-color-20110607/#rgba-color Chrome 52 supports it: https://googlechrome.github.io/samples/css-alpha-channel/ Android instead: https://developer.android.com/reference/android/graphics/Color.html#parseColor(java.lang.String) "Parse the color string, and return the corresponding color-int. If the string cannot be parsed, throws an IllegalArgumentException exception. Supported formats are: #RRGGBB #AARRGGBB [...]" Please notice that on PNG file format rgba is quite common (also on bmp with semi-official apha support) PNG: http://www.libpng.org/pub/png/book/chapter08.html
Oh dear... thanks for digging that up. I didn't know the web had a standard for alpha. Certainly 0xAARRGGBB has been used in windows code for as long as I've been programming... but now there's a competing #RRGGBBAA version... How to resolve this? I guess, go with the web? I should probably change it to the CSS4 way.
Oct 11 2016
parent Andrea Fontana <nospam example.com> writes:
On Tuesday, 11 October 2016 at 12:14:37 UTC, Manu wrote:
 Oh dear... thanks for digging that up.
 I didn't know the web had a standard for alpha. Certainly 
 0xAARRGGBB
 has been used in windows code for as long as I've been 
 programming...
 but now there's a competing #RRGGBBAA version...
 How to resolve this? I guess, go with the web? I should probably
 change it to the CSS4 way.
My idea is still to use a template: colorFromString!"rgba" or colorFromString!"argb" (please notice that AFAIK, the second most used way is actually "abgr" rather than "argb" - because of byte-order) And I think it's a good idea to set template argument to some default. MS is not sure about this, anyway. Read carefully this: https://msdn.microsoft.com/en-us/library/windows/desktop/ms534427(v=vs.85).aspx VS: https://msdn.microsoft.com/en-us/library/ee505694(v=winembedded.60).aspx VS: https://msdn.microsoft.com/it-it/library/windows/desktop/dd183449(v=vs.85).aspx OpenGL use rgba (and abgr?) directX argb: https://www.opengl.org/wiki/Image_Format https://www.opengl.org/wiki/Direct3D_Compatibility Andrea
Oct 11 2016