## digitalmars.D.learn - splitter trouble

• =?UTF-8?Q?Ali_=c3=87ehreli?= (26/26) Oct 30 2016 While working on a solution for Alfred Newman's thread, I came up with
• John Colvin (25/53) Nov 01 2016 As usual, auto-decoding has plumbed the sewage line straight in
=?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
```While working on a solution for Alfred Newman's thread, I came up with
the following interim solution, which compiled but failed:

auto parse(R, S)(R range, S separators) {
import std.algorithm : splitter, filter, canFind;
import std.range : empty;

static bool pred(E, S)(E e, S s) {
return s.canFind(e);
}

return range.splitter!pred(separators).filter!(token => !token.empty);
}

unittest {
import std.algorithm : equal;
import std.string : format;
auto parsed = parse("_My   input.string", " _,.");
assert(parsed.equal([ "My", "input", "string" ]), format("%s",
parsed));
}

void main() {
}

The unit test fails and prints

["put", "ing"]

not the expected

["My", "input", "string"].

How is that happening? Am I unintentionally hitting a weird overload of
splitter?

Ali
```
Oct 30 2016
John Colvin <john.loughran.colvin gmail.com> writes:
```On Sunday, 30 October 2016 at 23:57:11 UTC, Ali Ã‡ehreli wrote:
While working on a solution for Alfred Newman's thread, I came
up with the following interim solution, which compiled but
failed:

auto parse(R, S)(R range, S separators) {
import std.algorithm : splitter, filter, canFind;
import std.range : empty;

static bool pred(E, S)(E e, S s) {
return s.canFind(e);
}

return range.splitter!pred(separators).filter!(token =>
!token.empty);
}

unittest {
import std.algorithm : equal;
import std.string : format;
auto parsed = parse("_My   input.string", " _,.");
assert(parsed.equal([ "My", "input", "string" ]),
format("%s", parsed));
}

void main() {
}

The unit test fails and prints

["put", "ing"]

not the expected

["My", "input", "string"].

How is that happening? Am I unintentionally hitting a weird

Ali

As usual, auto-decoding has plumbed the sewage line straight in
to the drinking water...

Splitter needs to know how far to skip when it hits a match.
Normally speaking - for the pred(r.front, s) overload that you're
using here - the answer to that question is always 1. Except in
the case of narrow strings, where it's whatever the encoded
length of the separator is in the encoding of the source range
(in this case utf-8), in order to skip e.g. a big dchar.* But in
only want to skip forward one, because your separator isn't
really a separator.

* see
https://github.com/dlang/phobos/blob/d6572c2a44d69f449bfe2b07461b2f0a1d6503f9/std/algorithm/iteration.d#L3710

Basically, what you're doing isn't going to work. A separator is
considered to be a separator, i.e. something to be skipped over
and twisting the definition causes problems.

This will work, but I can't see any way to make it  nogc:

auto parse(R, S)(R range, S separators) {
import std.algorithm : splitter, filter, canFind;
import std.range : save, empty;

return range
.splitter!(e => separators.save.canFind(e))
.filter!(token => !token.empty);
}
```
Nov 01 2016