• Etienne Cimon (14/14) Oct 23 2015 Hello,
• Etienne Cimon (4/18) Oct 23 2015 I found out that there was an integer conversion going on behind

Etienne Cimon <etcimon gmail.com> writes:

Hello,

I've been trying to understand this for a while now:

https://github.com/etcimon/botan/blob/master/source/botan/math/mp/mp_core.d#L765

This comparison (looking at it with windbg during cmp operation) 
has these invalid values in the respective registers:

rdx: 9366584610601550696
r15: 8407293697099479287

When moving them into a ulong variable with a mov [R11], RDX 
before the CMP command, I get:
RDX: 7549031027420429441
R15: 17850297365717953652

Which are the valid values.

Any idea how these values could have gotten corrupted this way? 
Is there a signed integer conversion going on behind the scenes?

Etienne Cimon <etcimon gmail.com> writes:

On Friday, 23 October 2015 at 15:17:43 UTC, Etienne Cimon wrote:
 Hello,

 I've been trying to understand this for a while now:

 https://github.com/etcimon/botan/blob/master/source/botan/math/mp/mp_core.d#L765

 This comparison (looking at it with windbg during cmp 
 operation) has these invalid values in the respective registers:

 rdx: 9366584610601550696
 r15: 8407293697099479287

 When moving them into a ulong variable with a mov [R11], RDX 
 before the CMP command, I get:
 RDX: 7549031027420429441
 R15: 17850297365717953652

 Which are the valid values.

 Any idea how these values could have gotten corrupted this way? 
 Is there a signed integer conversion going on behind the scenes?

I found out that there was an integer conversion going on behind 
the scenes when using jnl. I had to use jnb

http://stackoverflow.com/questions/27284895/how-to-compare-a-signed-value-and-an-unsigned-value-in-x86-assembly