Digital Mars - c++ - Huge pointers fail in 16-bit large model?

↑ ↓ ← → Jan Knepper <jan smartsoft.cc> writes:

Mark,

How large is this circular buffer you are talking about?

Have you considered building a 'class' that could handle multiple buffers
of a certain size (16K, 32K, 64K) with two members 'Put' and 'Get' that
automagically will 'put' and 'get' from the right position in the right
buffer?

Don't worry, be Kneppie!
Jan



Mark Evans wrote:

 One of the bugaboos with 16-bit code is pointer comparison.
 I don't claim to have discovered a bug, but maybe you
 could offer advice?

 I thought that comparing huge pointers was kosher because the compiler
 keeps them normalized.  I have a scanning loop on a (huge) circular
 buffer that wraps at end-of-buffer.  The wrapping code breaks on a
 pointer comparison.  To wit, the compiler claims that

         0E5F:0001 >= 0EBF:0000

 where both are represented as huge pointers to char.  It appears to me
 that the compiler is just comparing the offsets.  Is this how it's
 supposed to work, do we have a bug, or what?  Thanks man,

 Kind regards,

 __________________________________________________________________
 Mark Evans              Control Systems Engineer
 Zyvex Corporation       http://www.zyvex.com/
 mevans zyvex.com   tel: 972.235.7881 ext 253    fax: 972.235.7882
 __________________________________________________________________

May 08 2001

↑ ↓ ← → Mark Evans <mevans zyvex.com> writes:

As large as I can reasonably make it.

I have a pseudo-class (pure C, but written in C++ style if you please) which
encapsulates the memory management issues.  It can be any size you want, but
will always use a huge buffer.  It 
already has methods similar to put and get.

The circularity of the buffer is not really the issue.  The issues in my mind
are the fundamental issues of huge pointers in Digital Mars.  I will see if I
can reduplicate my problem with the 
inequality operator on huge pointers.  Possibly my compile switches have
something to do with it...

Mark


On Tue, 08 May 2001 07:48:07 -0400, Jan Knepper <jan smartsoft.cc> wrote:
 Mark,
 
 How large is this circular buffer you are talking about?
 
 Have you considered building a 'class' that could handle multiple buffers
 of a certain size (16K, 32K, 64K) with two members 'Put' and 'Get' that
 automagically will 'put' and 'get' from the right position in the right
 buffer?
 
 Don't worry, be Kneppie!
 Jan
 
 
 
 Mark Evans wrote:
 
 One of the bugaboos with 16-bit code is pointer comparison.
 I don't claim to have discovered a bug, but maybe you
 could offer advice?

 I thought that comparing huge pointers was kosher because the compiler
 keeps them normalized.  I have a scanning loop on a (huge) circular
 buffer that wraps at end-of-buffer.  The wrapping code breaks on a
 pointer comparison.  To wit, the compiler claims that

         0E5F:0001 >= 0EBF:0000

 where both are represented as huge pointers to char.  It appears to me
 that the compiler is just comparing the offsets.  Is this how it's
 supposed to work, do we have a bug, or what?  Thanks man,

 Kind regards,

 __________________________________________________________________
 Mark Evans              Control Systems Engineer
 Zyvex Corporation       http://www.zyvex.com/
 mevans zyvex.com   tel: 972.235.7881 ext 253    fax: 972.235.7882
 __________________________________________________________________

May 08 2001

↑ ↓ ← → Jan Knepper <jan smartsoft.cc> writes:

Mark Evans wrote:

 I have a pseudo-class (pure C, but written in C++ style if you please) which
encapsulates the memory management issues.  It can be any size you want, but
will always use a huge buffer.  It
 already has methods similar to put and get.


So... Why should it have to use a huge buffer?
Once you 'wrap' put and get into a class or pseudo-class mechanism it's
basically a black box inside which magic things can happen...???!!!

Than my next honest question would be... Since you seem to use a 3rd partly
.DLL that keeps you on the Win16 target. Does this .DLL use HUGE pointers as
parameters to the functions? If not, what
would be the real reason for using HUGE pointers? Because it's easier to work
with large buffers in 16 bit code? That can be a very valid reason. Don't get
me wrong.

Also, does your .DLL work in a Win32 environment?
Considered thunking at all?

Don't worry, be Kneppie!
Jan

May 08 2001

↑ ↓ ← → Mark Evans <mevans zyvex.com> writes:

Jan,

The short answers:

- I considered and rejected thunking
- The DLL does not use huge pointers but I need them
- Black boxes are fine, do you have one written already?
- I'm already "that close" to success

The long answers:

My project is a nice way to bypass thunking.  My 16-bit Digital Mars app is a
server daemon which is linked directly to the 16-bit DLL.  The server app opens
a socket and receives ASCII 
requests to make DLL calls, returning ASCII versions of return values from the
DLL.  It's effectively a thin interface layer around the 16-bit DLL.

The server can talk to any code that supports sockets, such as Python scripts,
LabVIEW diagrams, or compiled C on Win32.  I am not limited to C (as is the
case with thunking).  So for the 
price of a little socket work, I avoid thunking and broaden the range of
application languages.

Thunking is a very complicated mess anyway, and hard to debug.  We have 150+
API calls in the DLL, which is a 2 MB giant.  We have no source code for the
DLL and no vendor support.  
It's a true legacy code problem.  The vendor is even out of business and the
programmers who wrote the DLL are long gone.  

Why huge data?  The DLL controls vendor equipment that makes images.  These are
typically 256x256 pixels.  At 8 bits per pixel, that is already 64K, the limit
of far data.  A single 512x512 
image is 128K.  And if I want to turn the data into ASCII for transport then
we're talking 5-10 bytes per pixel on average (the bytes are characters).  And
as you say, huge pointers are 
conceptually closer to true "flat" or 32-bit pointers so they're easier to work
with.

In any communications scheme, you generally want a buffer as large as the
largest message possible and/or some typical multiple of an average message
size.  With ASCII protocols and 
images that figure gets pretty fat.  So a huge circular buffer seems the best
thing.  I have one for input, another for output.

Things are actually going pretty well.  The problems I'm having revolve around
a socket call -- send() -- not liking my artificial far pointers.  I am trying
to feed it far pointers pointing into the huge 
buffer.  These are computed quantities, not memory manager quantities.  My
buffer pseudo-object has a call, GetDataPtrFar(), to return a legitimate (char
__far *) into the middle of the huge 
allocation block, wherever its current marker happens to be.  However send()
keeps spitting back WSAEINVAL on the pointer.

I am linking with the snippets.org library.  It has two calls farnormal() and
addptr() which I could not improve upon.  No combination of these seems to
satisfy send() at this point.  That will be my 
debugging exercise for tomorrow.

Of course Win16 limits me to Winsock 1.1 and can't access 2.0, but that seems
to be OK.  The Win16 app is running on a Windows 98 machine.

Mark



On Tue, 08 May 2001 12:00:48 -0400, Jan Knepper <jan smartsoft.cc> wrote:
 Mark Evans wrote:
 
 I have a pseudo-class (pure C, but written in C++ style if you please) which
encapsulates the memory management issues.  It can be any size you want, but
will always use a huge buffer.  




 already has methods similar to put and get.


 So... Why should it have to use a huge buffer?
 Once you 'wrap' put and get into a class or pseudo-class mechanism it's
basically a black box inside which magic things can happen...???!!!
 
 Than my next honest question would be... Since you seem to use a 3rd partly
.DLL that keeps you on the Win16 target. Does this .DLL use HUGE pointers as
parameters to the functions? 


 would be the real reason for using HUGE pointers? Because it's easier to work
with large buffers in 16 bit code? That can be a very valid reason. Don't get
me wrong.
 
 Also, does your .DLL work in a Win32 environment?
 Considered thunking at all?
 
 Don't worry, be Kneppie!
 Jan

May 09 2001

↑ ↓ ← → Jan Knepper <jan smartsoft.cc> writes:

Hi Mark,

I read this and it all makes sense to me.

I have many black-boxes... Besides that I write end user
applications I also get to write libraries to make designing and
developing those applications simpler. Those libraries are being
reused by other
programmer(s) and I am sure many of them just use them as
black-boxes that work and they hopefully never have to look
into.

Anyways, no I do not have a black box written for you problem
yet. I could not as I do not know what the design would have to
look like. If is't just an encapsulated buffer with Get ( char )
and Put ( char ) I could build one in a very short time. If I
only had the time...

I personally would not rely on farnormal () and addptr () from
an other library unless it has been proven to work with the
compiler.

As for the __huge to __far translation of the pointers I guess
the best thing to do is to take the segment and offset from the
__huge pointer and normalize it to a new segment and offset and
make a __far pointer with MK_FP in such a way that the segment
forms most of the pointer and the offset is close to 0. This so
you can access almost 64K from the position of the __far
pointer.

Don't worry, be Kneppie!
Jan



Mark Evans wrote:

 Jan,

 The short answers:

 - I considered and rejected thunking
 - The DLL does not use huge pointers but I need them
 - Black boxes are fine, do you have one written already?
 - I'm already "that close" to success

 The long answers:

 My project is a nice way to bypass thunking.  My 16-bit Digital Mars app is a
server daemon which is linked directly to the 16-bit DLL.  The server app opens
a socket and receives ASCII
 requests to make DLL calls, returning ASCII versions of return values from the
DLL.  It's effectively a thin interface layer around the 16-bit DLL.

 The server can talk to any code that supports sockets, such as Python scripts,
LabVIEW diagrams, or compiled C on Win32.  I am not limited to C (as is the
case with thunking).  So for the
 price of a little socket work, I avoid thunking and broaden the range of
application languages.

 Thunking is a very complicated mess anyway, and hard to debug.  We have 150+
API calls in the DLL, which is a 2 MB giant.  We have no source code for the
DLL and no vendor support.
 It's a true legacy code problem.  The vendor is even out of business and the
programmers who wrote the DLL are long gone.

 Why huge data?  The DLL controls vendor equipment that makes images.  These
are typically 256x256 pixels.  At 8 bits per pixel, that is already 64K, the
limit of far data.  A single 512x512
 image is 128K.  And if I want to turn the data into ASCII for transport then
we're talking 5-10 bytes per pixel on average (the bytes are characters).  And
as you say, huge pointers are
 conceptually closer to true "flat" or 32-bit pointers so they're easier to
work with.

 In any communications scheme, you generally want a buffer as large as the
largest message possible and/or some typical multiple of an average message
size.  With ASCII protocols and
 images that figure gets pretty fat.  So a huge circular buffer seems the best
thing.  I have one for input, another for output.

 Things are actually going pretty well.  The problems I'm having revolve around
a socket call -- send() -- not liking my artificial far pointers.  I am trying
to feed it far pointers pointing into the huge
 buffer.  These are computed quantities, not memory manager quantities.  My
buffer pseudo-object has a call, GetDataPtrFar(), to return a legitimate (char
__far *) into the middle of the huge
 allocation block, wherever its current marker happens to be.  However send()
keeps spitting back WSAEINVAL on the pointer.

 I am linking with the snippets.org library.  It has two calls farnormal() and
addptr() which I could not improve upon.  No combination of these seems to
satisfy send() at this point.  That will be my
 debugging exercise for tomorrow.

 Of course Win16 limits me to Winsock 1.1 and can't access 2.0, but that seems
to be OK.  The Win16 app is running on a Windows 98 machine.

 Mark

 On Tue, 08 May 2001 12:00:48 -0400, Jan Knepper <jan smartsoft.cc> wrote:
 Mark Evans wrote:

 I have a pseudo-class (pure C, but written in C++ style if you please) which
encapsulates the memory management issues.  It can be any size you want, but
will always use a huge buffer.




 already has methods similar to put and get.


 So... Why should it have to use a huge buffer?
 Once you 'wrap' put and get into a class or pseudo-class mechanism it's
basically a black box inside which magic things can happen...???!!!

 Than my next honest question would be... Since you seem to use a 3rd partly
.DLL that keeps you on the Win16 target. Does this .DLL use HUGE pointers as
parameters to the functions?


 would be the real reason for using HUGE pointers? Because it's easier to work
with large buffers in 16 bit code? That can be a very valid reason. Don't get
me wrong.

 Also, does your .DLL work in a Win32 environment?
 Considered thunking at all?

 Don't worry, be Kneppie!
 Jan

May 10 2001

↑ ↓ ← → Mark Evans <mevans zyvex.com> writes:

Jan,

Walter Bright actually has some contributions in the snippets.org library.  I'm
not using those particular ones (yet) but the source code for the pointer
arithmetic is so short, maybe you could 
look at it.  I have no reason to doubt its accuracy.  I would suppose that any
flaw in its execution could be chalked up to the compiler.

Mark


/* +++Date last modified: 05-Jul-1997 */

/*
**  FPTR_ADD.C
**
**  Add any add any value to a far pointer and returns the result as a
**  normalized far pointer.
**
**  Public Domain by Soleil Lapierre
*/

#include "snpdosys.h"
#include "mk_fp.h"


void FAR *addptr (char FAR *p, unsigned long num)
{
      unsigned seg,off;
 
      seg = FP_SEG(p); off = FP_OFF(p);
      seg += off>>4;   off &= 0x000F;

      off += (unsigned)(num&0x0000000fL);

      seg += off>>4;   off &= 0x000F;
      seg += (unsigned)num>>4;

      return(MK_FP(seg,off));
}

/*
**  Normalize a far pointer
*/

void FAR *farnormal(void FAR *ptr)
{
      unsigned long base, para;

      base = ((unsigned long)(ptr) & 0xffff000fL);
      para = ((unsigned long)(ptr) & 0x0000fff0L);
      para <<= 12;
      return (void FAR *)(base + para);
}

May 10 2001

↑ ↓ ← → NancyEtRoland <nancyetroland free.fr> writes:

me i would have writen those function like this:

(!! not tested, but it should be ok !!)

inline unsigned long dosptrtolinaddr(void far* dosptr) {
//convert 16 bit far pointer to linear addresse
    return ((((unsigned long) dosptr) & ~0xffff)>>12) + (((unsigned long)
dosptr) & 0xfffful )
}

inline void far* linaddrtodosptr(unsigned long linaddr) {
//convert linear addresse to 16 bit far pointer
   return void far* ((linaddr<<12) & ~0xffff)    +  (linaddr & 0xful);
}

inline void FAR *addptr (char FAR *p, unsigned long num) {
    return linaddrtodosptr(dosptrtolinaddr(p)+num);
}

for me its ok:

void FAR *farnormal(void FAR *ptr)
{
      unsigned long base, para;

      base = ((unsigned long)(ptr) & 0xffff000fL);
      para = ((unsigned long)(ptr) & 0x0000fff0L);
      para <<= 12;
      return (void FAR *)(base + para);
}


roland


Mark Evans a �crit :

 Jan,

 Walter Bright actually has some contributions in the snippets.org library. 
I'm not using those particular ones (yet) but the source code for the pointer
arithmetic is so short, maybe you could
 look at it.  I have no reason to doubt its accuracy.  I would suppose that any
flaw in its execution could be chalked up to the compiler.

 Mark

 /* +++Date last modified: 05-Jul-1997 */

 /*
 **  FPTR_ADD.C
 **
 **  Add any add any value to a far pointer and returns the result as a
 **  normalized far pointer.
 **
 **  Public Domain by Soleil Lapierre
 */

 #include "snpdosys.h"
 #include "mk_fp.h"

 void FAR *addptr (char FAR *p, unsigned long num)
 {
       unsigned seg,off;

       seg = FP_SEG(p); off = FP_OFF(p);
       seg += off>>4;   off &= 0x000F;

       off += (unsigned)(num&0x0000000fL);

       seg += off>>4;   off &= 0x000F;
       seg += (unsigned)num>>4;

       return(MK_FP(seg,off));
 }

 /*
 **  Normalize a far pointer
 */

 void FAR *farnormal(void FAR *ptr)
 {
       unsigned long base, para;

       base = ((unsigned long)(ptr) & 0xffff000fL);
       para = ((unsigned long)(ptr) & 0x0000fff0L);
       para <<= 12;
       return (void FAR *)(base + para);
 }

May 12 2001

↑ ↓ ← → Roland <rv ronetech.com> writes:

Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

oops

i forgot some ul: it is 0xfffful


NancyEtRoland a �crit :

 me i would have writen those function like this:

 (!! not tested, but it should be ok !!)

 inline unsigned long dosptrtolinaddr(void far* dosptr) {
 //convert 16 bit far pointer to linear addresse
     return ((((unsigned long) dosptr) & ~0xffff)>>12) + (((unsigned long)
dosptr) & 0xfffful )
 }

 inline void far* linaddrtodosptr(unsigned long linaddr) {
 //convert linear addresse to 16 bit far pointer
    return void far* ((linaddr<<12) & ~0xffff)    +  (linaddr & 0xful);
 }

 inline void FAR *addptr (char FAR *p, unsigned long num) {
     return linaddrtodosptr(dosptrtolinaddr(p)+num);
 }

 for me its ok:

 void FAR *farnormal(void FAR *ptr)
 {
       unsigned long base, para;

       base = ((unsigned long)(ptr) & 0xffff000fL);
       para = ((unsigned long)(ptr) & 0x0000fff0L);
       para <<= 12;
       return (void FAR *)(base + para);
 }

 roland

 Mark Evans a �crit :

 Jan,

 Walter Bright actually has some contributions in the snippets.org library. 
I'm not using those particular ones (yet) but the source code for the pointer
arithmetic is so short, maybe you could
 look at it.  I have no reason to doubt its accuracy.  I would suppose that any
flaw in its execution could be chalked up to the compiler.

 Mark

 /* +++Date last modified: 05-Jul-1997 */

 /*
 **  FPTR_ADD.C
 **
 **  Add any add any value to a far pointer and returns the result as a
 **  normalized far pointer.
 **
 **  Public Domain by Soleil Lapierre
 */

 #include "snpdosys.h"
 #include "mk_fp.h"

 void FAR *addptr (char FAR *p, unsigned long num)
 {
       unsigned seg,off;

       seg = FP_SEG(p); off = FP_OFF(p);
       seg += off>>4;   off &= 0x000F;

       off += (unsigned)(num&0x0000000fL);

       seg += off>>4;   off &= 0x000F;
       seg += (unsigned)num>>4;

       return(MK_FP(seg,off));
 }

 /*
 **  Normalize a far pointer
 */

 void FAR *farnormal(void FAR *ptr)
 {
       unsigned long base, para;

       base = ((unsigned long)(ptr) & 0xffff000fL);
       para = ((unsigned long)(ptr) & 0x0000fff0L);
       para <<= 12;
       return (void FAR *)(base + para);
 }

May 14 2001

↑ ↓ ← → Jan Knepper <jan smartsoft.cc> writes:

Mark,

I tried the following little program:

#include <stdio.h>



int  main ( int, char **, char ** )
{
   char __huge   *ptr0;
   char __huge   *ptr1;

   ptr0  = ( char __huge * ) 0x0E5F0001L;
   ptr1  = ( char __huge * ) 0x0EBF0000L;

   printf ( "%p >= %p = %d\n", ptr0, ptr1, ptr0 >= ptr1 );
   printf ( "%p <= %p = %d\n", ptr0, ptr1, ptr0 <= ptr1 );

   return (  0 );
}

Compiled it:
sc -ml Test0140.cpp

Run it:
And the output is:

P:\TMP>test0140
0E5F:0001 >= 0EBF:0000 = 0
0E5F:0001 <= 0EBF:0000 = 1

I could me misreading things here, but what exactly is your problem?

Don't worry, be Kneppie!
Jan



Mark Evans wrote:

 One of the bugaboos with 16-bit code is pointer comparison.
 I don't claim to have discovered a bug, but maybe you
 could offer advice?

 I thought that comparing huge pointers was kosher because the compiler
 keeps them normalized.  I have a scanning loop on a (huge) circular
 buffer that wraps at end-of-buffer.  The wrapping code breaks on a
 pointer comparison.  To wit, the compiler claims that

         0E5F:0001 >= 0EBF:0000

 where both are represented as huge pointers to char.  It appears to me
 that the compiler is just comparing the offsets.  Is this how it's
 supposed to work, do we have a bug, or what?  Thanks man,

 Kind regards,

 __________________________________________________________________
 Mark Evans              Control Systems Engineer
 Zyvex Corporation       http://www.zyvex.com/
 mevans zyvex.com   tel: 972.235.7881 ext 253    fax: 972.235.7882
 __________________________________________________________________

May 08 2001