www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Weird bit-shift behavior on void*.

reply C. Dunn <cdunn2001 gmail.com> writes:
I wish I could provide a testcase, but the problem is too weird:

    void* q = cast(void*)0x402A8000;
    writefln("%x %x %d %d %x %x", q, v, q.sizeof, v.sizeof, (cast(size_t)q >>
48),(cast(size_t)v >> 48));

'v' is a void* too. It is set during the program, and it is a real pointer
address.  According to the output, q and v have the same value and the same
data-type size:
  402A8000 402A8000 4 4 0 402a
(This output comes directly from the writefln as shown.)
So why do they have different bit-shifted results?  The weird thing is, the
bit-shift operand on 'v' is taken mod 32.  It gives '0' for 'v >> 32' and
repeats the previous sequence up to 'v >> 64', etc.

But I cannot produce a testcase with this behavior because when I create a
void* from the value of 'v' (e.g. 'q') I do not see the weird behavior.  The
issue is not '>>' v. '>>>'; I've tried both.

The solution is to use (cast(ulong)v >> 48).  (Again, '>>' v. '>>>' makes no
difference.)  Then I get:
  402A8000 402A8000 4 4 0 0
as expected.  But the cast(size_t) on 'q' works fine.

Does anyone have any idea what is going on here?  It is very magical, and it
cost me a TON of debug time.
Aug 09 2007
next sibling parent Jascha Wetzel <firstname mainia.de> writes:
C. Dunn wrote:

 I wish I could provide a testcase, but the problem is too weird:
 
     void* q = cast(void*)0x402A8000;
     writefln("%x %x %d %d %x %x", q, v, q.sizeof, v.sizeof, (cast(size_t)q >>
48),(cast(size_t)v >> 48));
 
 'v' is a void* too. It is set during the program, and it is a real pointer
address.  According to the output, q and v have the same value and the same
data-type size:
   402A8000 402A8000 4 4 0 402a
 (This output comes directly from the writefln as shown.)
 So why do they have different bit-shifted results?  The weird thing is, the
bit-shift operand on 'v' is taken mod 32.  It gives '0' for 'v >> 32' and
repeats the previous sequence up to 'v >> 64', etc.
 
 But I cannot produce a testcase with this behavior because when I create a
void* from the value of 'v' (e.g. 'q') I do not see the weird behavior.  The
issue is not '>>' v. '>>>'; I've tried both.
 
 The solution is to use (cast(ulong)v >> 48).  (Again, '>>' v. '>>>' makes no
difference.)  Then I get:
   402A8000 402A8000 4 4 0 0
 as expected.  But the cast(size_t) on 'q' works fine.
 
 Does anyone have any idea what is going on here?  It is very magical, and it
cost me a TON of debug time.


did you try looking at the code DMD generates (using obj2asm or a debugger)?
Aug 09 2007
prev sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
C. Dunn wrote:

 I wish I could provide a testcase, but the problem is too weird:
 
     void* q = cast(void*)0x402A8000;
     writefln("%x %x %d %d %x %x", q, v, q.sizeof, v.sizeof, (cast(size_t)q >>
48),(cast(size_t)v >> 48));
 
 'v' is a void* too. It is set during the program, and it is a real pointer
address.  According to the output, q and v have the same value and the same
data-type size:
   402A8000 402A8000 4 4 0 402a
 (This output comes directly from the writefln as shown.)
 So why do they have different bit-shifted results?  The weird thing is, the
bit-shift operand on 'v' is taken mod 32.  It gives '0' for 'v >> 32' and
repeats the previous sequence up to 'v >> 64', etc.
 
 But I cannot produce a testcase with this behavior because when I create a
void* from the value of 'v' (e.g. 'q') I do not see the weird behavior.  The
issue is not '>>' v. '>>>'; I've tried both.
 
 The solution is to use (cast(ulong)v >> 48).  (Again, '>>' v. '>>>' makes no
difference.)  Then I get:
   402A8000 402A8000 4 4 0 0
 as expected.  But the cast(size_t) on 'q' works fine.
 
 Does anyone have any idea what is going on here?  It is very magical, and it
cost me a TON of debug time.


The last sentence of 
http://www.digitalmars.com/d/1.0/expression.html#ShiftExpression : "It's 
illegal to shift by more bits than the size of the quantity being shifted"
That means that if you shift by more bits than the type can hold you 
can't rely on the result. (And the compiler may in fact even refuse to 
compile it)
cast(long) works around this issue by upping the number of bits, but the 
problem may recur if more than 64 bits are used (and it most likely 
*will* on x86-64 systems, but I'm not sure how DMDs 'software longs' 
handle that case)

It looks like the const-folding code in the compiler handles oversized 
shifts differently than the machine code generated. (I can only 
reproduce your problem by adding '-O' to the command line, without 
optimizations the result is the same for q and v, and both shifts are 
performed modulo 32)
I can't say it's much of a surprise that the run-time shifted (v >> 48) 
is done modulo 32, since that's exactly how the x86 'shr' opcode works. 
Apparently the const-folding code performs the shift in a more intuitive 
way, perhaps by storing integral and pointer constants in longs or just 
by a runtime check on whether the amount shifted is too large...
Aug 10 2007
parent C. Dunn <cdunn2001 gmail.com> writes:
Frits van Bommel Wrote:


 The last sentence of 
 http://www.digitalmars.com/d/1.0/expression.html#ShiftExpression : "It's 
 illegal to shift by more bits than the size of the quantity being shifted"
 That means that if you shift by more bits than the type can hold you 
 can't rely on the result. (And the compiler may in fact even refuse to 
 compile it)
 cast(long) works around this issue by upping the number of bits, but the 
 problem may recur if more than 64 bits are used (and it most likely 
 *will* on x86-64 systems, but I'm not sure how DMDs 'software longs' 
 handle that case)
 
 It looks like the const-folding code in the compiler handles oversized 
 shifts differently than the machine code generated. (I can only 
 reproduce your problem by adding '-O' to the command line, without 
 optimizations the result is the same for q and v, and both shifts are 
 performed modulo 32)


Interesting.



 I can't say it's much of a surprise that the run-time shifted (v >> 48) 
 is done modulo 32, since that's exactly how the x86 'shr' opcode works. 
 Apparently the const-folding code performs the shift in a more intuitive 
 way, perhaps by storing integral and pointer constants in longs or just 
 by a runtime check on whether the amount shifted is too large...


I didn't know that.  I don't think the other processors I've used (years ago,
when I first learned bit-shiftig) worked that way.  Interesting.

This came up because I was porting someone else's code.  Now I understand why
it looked like this in C:
  (long)((long long)(long)v >> 48)

Very, very helpful reply.  Thank you!
Aug 10 2007