www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Regex and utf8

reply Roman Balitskiy <realis_toleroATtoleroDOTorg_fake fake.com> writes:
When I try to parse cyrillic text I get "Error: 4invalid UTF-8 sequence". I use
dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have tryed upcomming gdc 0.25 with
the same results.


cyrillic letter 'je'
		writefln("%s[%s]%s", m.pre, m.match(0), m.post);
Jul 20 2008
next sibling parent "Koroskin Denis" <2korden gmail.com> writes:
On Sun, 20 Jul 2008 22:50:06 +0400, Roman Balitskiy  
<realis_toleroATtoleroDOTorg_fake fake.com> wrote:

 When I try to parse cyrillic text I get "Error: 4invalid UTF-8  
 sequence". I use dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have tryed  
 upcomming gdc 0.25 with the same results.


 is cyrillic letter 'je'
 		writefln("%s[%s]%s", m.pre, m.match(0), m.post);
Try removing braces, the following code sample works for me: import std.stdio; import std.regexp; void main() { if (auto m = std.regexp.search("abÖdef", "Ö")) { writefln("%s[%s]%s", m.pre, m.match(0), m.post); } } I don't know if it's a bug or not, most probably it is. But since Phobos console is not Unicode aware, you won't see "ab[Ö]def" as expected but rather something like "ab[º´]def" (my output, might be different on other locale settings). By constrast, the Tango console I/O is more Unicode-friendly: import tango.text.Regex; import tango.io.Stdout; void main() { foreach(m; Regex("Ö").search("abÖdef")) { Stdout.formatln("{}[{}]{}", m.pre, m.match(0), m.post); } } Hope that helps.
Jul 20 2008
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Roman Balitskiy wrote:
 When I try to parse cyrillic text I get "Error: 4invalid UTF-8 sequence". I
use dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have tryed upcomming gdc 0.25
with the same results.
 

cyrillic letter 'je'
 		writefln("%s[%s]%s", m.pre, m.match(0), m.post);
 
The back quotes are for wysiwyg strings, and the UTF translation doesn't happen. Try using "" strings instead.
Jul 20 2008
parent reply "Koroskin Denis" <2korden gmail.com> writes:
On Sun, 20 Jul 2008 23:45:34 +0400, Walter Bright  =

<newshound1 digitalmars.com> wrote:

 Roman Balitskiy wrote:
 When I try to parse cyrillic text I get "Error: 4invalid UTF-8  =
 sequence". I use dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have  =
 tryed upcomming gdc 0.25 with the same results.

=
 Here is cyrillic letter 'je'
 		writefln("%s[%s]%s", m.pre, m.match(0), m.post);
The back quotes are for wysiwyg strings, and the UTF translation doesn=
't =
 happen. Try using "" strings instead.
Nope, it doesn't help. However, removing square brackets does.
Jul 20 2008
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Koroskin Denis wrote:
 On Sun, 20 Jul 2008 23:45:34 +0400, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Roman Balitskiy wrote:
 When I try to parse cyrillic text I get "Error: 4invalid UTF-8 
 sequence". I use dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have 
 tryed upcomming gdc 0.25 with the same results.

 // Here is cyrillic letter 'je'
         writefln("%s[%s]%s", m.pre, m.match(0), m.post);
The back quotes are for wysiwyg strings, and the UTF translation doesn't happen. Try using "" strings instead.
Nope, it doesn't help. However, removing square brackets does.
That's a bug with the regex engine, then. Who wants to put it in bugzilla? <g>
Jul 20 2008
parent Roman Balitskiy <realis_toleroATtoleroDOTorg_fake fake.com> writes:
Walter Bright Wrote:

 When I try to parse cyrillic text I get "Error: 4invalid UTF-8 
 sequence". I use dmd 1.030 on Ubuntu 8.04 with utf8 locale. I have 
 tryed upcomming gdc 0.25 with the same results.

 // Here is cyrillic letter 'je'
         writefln("%s[%s]%s", m.pre, m.match(0), m.post);
That's a bug with the regex engine, then. Who wants to put it in bugzilla? <g>
Is there any progress towards fix of that bug?
Aug 13 2008