www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Implicit encoding conversion on string ~= int ?

reply Marco Leise <Marco.Leise gmx.de> writes:
I've seen some C code, that does something like string[i] =
int, which seems to implicitly cast the int to a char.
Now in D to get it running I just did string ~= int and
wondered why the resulting code page 850 string looked correct
on the UTF-8 terminal. Then I asserted that 'string' only ever
grows by one byte for each append and the assertion failed. So
there is a hidden conversion from some charset (probably
Windows or Latin-1?) to a UTF-8 multi-byte string going on.

While it is convenient, this code uses some form of LZ77 and
will from time to time append copies of previous parts of
'string' to it. In that case the byte offsets wouldn't match
any more and the result be garbage.

Eventually I'd have looked over the code and created the CP850
string in a temporary ubyte[], but in the mean time I wonder
what the rationale behind this automatic conversion is and if
we want to keep it like that. Is this documented behavior ?

-- 
Marco
Jun 23 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
I think what's happening is the compiler considers chars to be 
integral types (like they were in C), which means some implicit 
conversions between char, int, dchar, and others happen.

So

char[] a;
int b = 1000;
a ~= b;

the "a ~= b" is more like "a ~= cast(dchar) b", and then dchar -> 
char means it may be multibyte encoded, going from utf-32 to 
utf-8.
Jun 23 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Adam D. Ruppe:


 char[] a;
 int b = 1000;
 a ~= b;

 the "a ~= b" is more like "a ~= cast(dchar) b", and then dchar 
 -> char means it may be multibyte encoded, going from utf-32 to 
 utf-8.


I didn't know that, is this already in Bugzilla?

Bye,
bearophile
Jun 23 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 23 Jun 2013 18:37:16 +0200
schrieb "bearophile" <bearophileHUGS lycos.com>:


 Adam D. Ruppe:
=20

 char[] a;
 int b =3D 1000;
 a ~=3D b;

 the "a ~=3D b" is more like "a ~=3D cast(dchar) b", and then dchar=20
 -> char means it may be multibyte encoded, going from utf-32 to=20
 utf-8.




No no no, this is not what happens. In my case it was:
string a;
int b =3D 228;  // CP850 value for '=C3=A4'. Note: fits in a single byte!
a ~=3D b;

Maybe it goes as follows:
o compiler sees ~=3D to a string and becomes "aware" of wchar and dchar
  conversions to char
o appended value is only checked for size (type and signedness are lost)
  and maps int to dchar
o this dchar value is now checked for Unicode conformity and fails the test
o the dchar value is now assumed to be Latin-1, Windows-1252 or similar
  and a conversion routine invoked
o the dchar value is converted to utf-8 and...
o appended as a multi-byte string to variable "a".

That still doesn't sound right to me thought. What if the dchar value is
not valid Unicode AND >=3D 256 ?


 I didn't know that, is this already in Bugzilla?
=20
 Bye,
 bearophile


I don't know what exactly is supposed to happen here.

--=20
Marco
Jun 23 2013
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 23 June 2013 at 17:12:41 UTC, Marco Leise wrote:

 int b = 228;  // CP850 value for 'ä'. Note: fits in a single 
 byte!


228 (e4 in hex) is also the Unicode code point for ä, which is 
[195, 164] when encoded as UTF-8. see: 
http://www.utf8-chartable.de/unicode-utf8-table.pl?number=512&utf8=dec

While the number 228 would fit in a byte normally, utf-8 uses the 
high bits as markers that this is part of a multibyte sequence 
(this helps with ascii compatibility), so any code point > 127 
will always be a multibyte sequence in utf-8. see: 
http://en.wikipedia.org/wiki/UTF-8#Description
Jun 23 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 23 Jun 2013 19:12:21 +0200
schrieb Marco Leise <Marco.Leise gmx.de>:


 Am Sun, 23 Jun 2013 18:37:16 +0200
 schrieb "bearophile" <bearophileHUGS lycos.com>:
=20

 Adam D. Ruppe:
=20

 char[] a;
 int b =3D 1000;
 a ~=3D b;

 the "a ~=3D b" is more like "a ~=3D cast(dchar) b", and then dchar=20
 -> char means it may be multibyte encoded, going from utf-32 to=20
 utf-8.




=20
 No no no, this is not what happens. In my case it was:
 string a;
 int b =3D 228;  // CP850 value for '=C3=A4'. Note: fits in a single byte!
 a ~=3D b;
=20
 Maybe it goes as follows:
 o compiler sees ~=3D to a string and becomes "aware" of wchar and dchar
   conversions to char
 o appended value is only checked for size (type and signedness are lost)
   and maps int to dchar
 o this dchar value is now checked for Unicode conformity and fails the te=


st

 o the dchar value is now assumed to be Latin-1, Windows-1252 or similar
   and a conversion routine invoked
 o the dchar value is converted to utf-8 and...
 o appended as a multi-byte string to variable "a".
=20
 That still doesn't sound right to me thought. What if the dchar value is
 not valid Unicode AND >=3D 256 ?


Actually you were 100% right, Adam. I was distracted by the
fact that the source was CP850.
UTF-32 maps all of Latin-1 in a 1:1 correspondence and most of
CP850 has the same code in Latin-1. So yes, all the compiler
was doing is to append a dchar value.
And with char/ubyte I do find it convenient to mix them
without casting. E.g. "if (someChar < 0x80)" and similar code.

As confusing as it was for me, I agree with "WONT FIX".

--=20
Marco
Jun 23 2013
prev sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 23 June 2013 at 16:37:18 UTC, bearophile wrote:

 I didn't know that, is this already in Bugzilla?


I don't know, but if it is, it is probably marked as won't fix 
because I'm pretty sure this has come up before, but it is 
actually by design because a char in C is considered an integral 
type too.
Jun 23 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, June 23, 2013 19:25:41 Adam D. Ruppe wrote:

 On Sunday, 23 June 2013 at 16:37:18 UTC, bearophile wrote:

 I didn't know that, is this already in Bugzilla?


 
 I don't know, but if it is, it is probably marked as won't fix
 because I'm pretty sure this has come up before, but it is
 actually by design because a char in C is considered an integral
 type too.


This is definitely by design. Walter is definitely in the camp that thinks that 
chars are integral types, so they follow all of the various integral 
conversion rules. In some cases this is nice. In others, it's bug-prone, but I 
think that we're stuck with it regardless of whether it's ultimately a good 
idea or not. I don't think that we even succeeded at coming close to 
convincing Walter that _bool_ isn't an integral type and shouldn't be treated 
as such (when it was discussed right before deconf), and that should be a far 
more clearcut case.

- Jonathan M Davis
Jun 23 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 23 Jun 2013 17:50:01 -0700
schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 I don't think that we even succeeded at coming close to 
 convincing Walter that _bool_ isn't an integral type and shouldn't be treated 
 as such (when it was discussed right before deconf), and that should be a far 
 more clearcut case.
 
 - Jonathan M Davis


You can take bool to int promotion out of my...

// best way to toggle forth and back between 0 and 1. "!" returns a bool.
value = !value  

// don't ask, I've seen this :)
arr[someBool]

// sometimes the bool has just the value you need
length -= boolRemoveTerminator

-- 
Marco
Jun 23 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 24, 2013 07:20:10 Marco Leise wrote:

 Am Sun, 23 Jun 2013 17:50:01 -0700
 
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 I don't think that we even succeeded at coming close to
 convincing Walter that _bool_ isn't an integral type and shouldn't be
 treated as such (when it was discussed right before deconf), and that
 should be a far more clearcut case.
 
 - Jonathan M Davis


 
 You can take bool to int promotion out of my...
 
 // best way to toggle forth and back between 0 and 1. "!" returns a bool.
 value = !value
 
 // don't ask, I've seen this :)
 arr[someBool]
 
 // sometimes the bool has just the value you need
 length -= boolRemoveTerminator


And in all those cases, you can cast to int to get the value you want. The 
case that brought up the big discussion on it a couple of months ago was when 
you had

auto foo(bool b) {...}
auto foo(long l) {...}

Which one does foo(1) call? It calls the bool version, because of how the 
integer conversion rules work. IMHO, this is _very_ broken, but Walter's 
response is that the solution is to add the overload

auto foo(int i) {...}

And that does fix the code in question, but it means that bool is _not_ 
strongly typed in D, and you get a variety of weird cases that cause bugs 
because of such implicit conversions. I would strongly argue that the case 
where you want bool to act like an integer is by far the rarer case and that 
casting fixes that problem nicely. Plenty of others agree with me. But no one 
has been able to convince Walter.

You can read the thread here:

http://forum.dlang.org/post/klc5r7$3c4$1 digitalmars.com

- Jonathan M Davis
Jun 23 2013
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/24/2013 07:20 AM, Marco Leise wrote:

 Am Sun, 23 Jun 2013 17:50:01 -0700
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 I don't think that we even succeeded at coming close to
 convincing Walter that _bool_ isn't an integral type and shouldn't be treated
 as such (when it was discussed right before deconf), and that should be a far
 more clearcut case.

 - Jonathan M Davis


 You can take bool to int promotion out of my...

 // best way to toggle forth and back between 0 and 1. "!" returns a bool.
 value = !value


value^=1


 // don't ask, I've seen this :)
 arr[someBool]

 // sometimes the bool has just the value you need
 length -= boolRemoveTerminator
Jun 24 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, June 24, 2013 07:20:10 Marco Leise wrote:

 Am Sun, 23 Jun 2013 17:50:01 -0700
 
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 I don't think that we even succeeded at coming close to
 convincing Walter that _bool_ isn't an integral type and shouldn't be
 treated as such (when it was discussed right before deconf), and that
 should be a far more clearcut case.
 
 - Jonathan M Davis


 
 You can take bool to int promotion out of my...
 
 // best way to toggle forth and back between 0 and 1. "!" returns a bool.
 value = !value
 
 // don't ask, I've seen this :)
 arr[someBool]
 
 // sometimes the bool has just the value you need
 length -= boolRemoveTerminator


And in all those cases, you can cast to int to get the value you want. The 
case that brought up the big discussion on it a couple of months ago was when 
you had

auto foo(bool b) {...}
auto foo(long l) {...}

Which one does foo(1) call? It calls the bool version, because of how the 
integer conversion rules work. IMHO, this is _very_ broken, but Walter's 
response is that the solution is to add the overload

auto foo(int i) {...}

And that does fix the code in question, but it means that bool is _not_ 
strongly typed in D, and you get a variety of weird cases that cause bugs 
because of such implicit conversions. I would strongly argue that the case 
where you want bool to act like an integer is by far the rarer case and that 
casting fixes that problem nicely. Plenty of others agree with me. But no one 
has been able to convince Walter.

You can read the thread here:

http://forum.dlang.org/post/klc5r7$3c4$1 digitalmars.com

- Jonathan M Davis
Jun 24 2013
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 24 June 2013 at 05:20:31 UTC, Marco Leise wrote:

 // best way to toggle forth and back between 0 and 1. "!" 
 returns a bool.
 value = !value


If you're switching between 0 and 1, chances are you should be 
using a bool in the first place.


 // don't ask, I've seen this :)
 arr[someBool]


Ew.


 // sometimes the bool has just the value you need
 length -= boolRemoveTerminator


Ew.

I think it's a big plus that it stops this needless obfuscation.
Jun 25 2013
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 24 June 2013 at 00:50:17 UTC, Jonathan M Davis wrote:

 On Sunday, June 23, 2013 19:25:41 Adam D. Ruppe wrote:

 On Sunday, 23 June 2013 at 16:37:18 UTC, bearophile wrote:

 I didn't know that, is this already in Bugzilla?


 
 I don't know, but if it is, it is probably marked as won't fix
 because I'm pretty sure this has come up before, but it is
 actually by design because a char in C is considered an 
 integral
 type too.


 This is definitely by design. Walter is definitely in the camp 
 that thinks that
 chars are integral types, so they follow all of the various 
 integral
 conversion rules.


Well, chars are integral values... but are integral values char?

I mean, I only see the promotion making sense one way: Converting 
a char to an uint can make sense for range analysis, but what 
about the other way around?

Same with bool: I can see bool to int making sense, but int to 
bool not so much, which is why a cast is required (except in an 
if).

In C, int to char was important, since char is the "byte" type. 
But D has byte, so I don't see why we'd allow int to byte...
Jun 23 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 24 Jun 2013 08:03:27 +0200
schrieb "monarch_dodra" <monarchdodra gmail.com>:


 Well, chars are integral values... but are integral values char?
 
 I mean, I only see the promotion making sense one way: Converting 
 a char to an uint can make sense for range analysis, but what 
 about the other way around?
 
 Same with bool: I can see bool to int making sense, but int to 
 bool not so much, which is why a cast is required (except in an 
 if).
 
 In C, int to char was important, since char is the "byte" type. 
 But D has byte, so I don't see why we'd allow int to byte...


+1. This is probably the best thinking. It would allow:

value = !value

and its ilk, but prevent

foo(1)

from going to a bool overload. But should that work?:

char D = 'A' + 3;
char E = 4 + 'A';

-- 
Marco
Jun 23 2013