www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - start partial binary

reply Moritz Warning <moritzwarning _nospam_web.de> writes:
I like to put some data into a binary
that should reside on disk even when the program
is started.

The data should be loaded in memory on request of the running part.

The aim is to get rid of zip file distributions:
- need to be extracted
- files need to be keep together

Is that possible with D?
Mar 12 2008
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:

 Is that possible with D?


I'm pretty sure this is an operating system thing. I don't think it can be
done on Linux, but on Windows, you might be able to do it with resources
embedded in the executable, just like you would with C programs.


-- 
Adam D. Ruppe
http://arsdnet.net
Mar 12 2008
parent reply Moritz Warning <moritzwarning _nospam_web.de> writes:
On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:


 On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:

 Is that possible with D?


 
 I'm pretty sure this is an operating system thing. I don't think it can
 be done on Linux, but on Windows, you might be able to do it with
 resources embedded in the executable, just like you would with C
 programs.


Would be a neat feature if it would be possible (on Linux).
The problem is to load only a part of the binary without the data payload.
Loading data from the binary into RAM would be less of a problem.
Mar 12 2008
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Moritz Warning (moritzwarning _nospam_web.de)'s article

 On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:

 On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:

 Is that possible with D?


 I'm pretty sure this is an operating system thing. I don't think it can
 be done on Linux, but on Windows, you might be able to do it with
 resources embedded in the executable, just like you would with C
 programs.


 Would be a neat feature if it would be possible (on Linux).
 The problem is to load only a part of the binary without the data payload.
 Loading data from the binary into RAM would be less of a problem.


Does the full binary of an application get loaded when the app executes?  I was
under the assumption
that code pages were loaded as needed.


Sean
Mar 12 2008
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thu, Mar 13, 2008 at 12:44:46AM +0000, Sean Kelly wrote:

 Does the full binary of an application get loaded when the app executes?
 I was under the assumption that code pages were loaded as needed.

 Sean


Yes, it looks like you're right according to a web search. So it should
be possible on Linux - using import()'s seems to me that it would be
the easiest way.

How it might be done in D:

-------------

import std.stdio;
string hugeFile = import("hugefile.txt");
	// If a reader is new to D, the above loads "hugefile.txt" at compile
	// time, as if I wrote:
	// string hugeFile = "the literal contents of hugefile.txt here";

void main(){
	writefln("...");
	readln();
	writefln("%s", hugeFile);
}

-------------

And to see how it works, watch for disk activity before and after the readln.
But, I'm foiled in testing the efficiency of this on the disk by the fact that
when making the executable, Linux caches the whole thing in RAM, so it doesn't
need to hit the disk anyway.

Note that the import used there doesn't work for absolutely huge files. It
works for anything less than at least 5 MB though, and probably more, which
might be enough to do the job wanted.

It's worth a try.

-- 
Adam D. Ruppe
http://arsdnet.net
Mar 12 2008
prev sibling parent "David Wilson" <dw botanicus.net> writes:
On Wed, Mar 12, 2008 at 11:59 PM, Moritz Warning
<moritzwarning _nospam_web.de> wrote:

 On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:

  > On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:


 Is that possible with D?




  >


 I'm pretty sure this is an operating system thing. I don't think it can


  > be done on Linux, but on Windows, you might be able to do it with
  > resources embedded in the executable, just like you would with C
  > programs.

  Would be a neat feature if it would be possible (on Linux).
  The problem is to load only a part of the binary without the data payload.
  Loading data from the binary into RAM would be less of a problem.


One way is to simply concatenate the data onto the end of the binary,
and write a function to open a standard stream on the file and seek to
the start of the data. For example:

my.exe:
  - <PE data>
  - <ZIP file>
  - <uint32 offset to first byte of zip file>

All a function needs to do is open and read the last 4 bytes, seek to
that position, and test for a zip signature. I'm pretty sure this
works for both PE and ELF, it's the way the Winzip self extractor
works, and some equivalent tools on Linux (I think info-zip provides
something similar).
Mar 12 2008
prev sibling next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Moritz,


 I like to put some data into a binary
 that should reside on disk even when the program
 is started.
 The data should be loaded in memory on request of the running part.
 
 The aim is to get rid of zip file distributions:
 - need to be extracted
 - files need to be keep together
 Is that possible with D?
 


I think most OS's map in the whole program but only load stuff on demand 
using the page swapping and virtual memeory systems.
Mar 12 2008
next sibling parent Moritz Warning <moritzwarning _nospam_web.de> writes:
Thanks for all the helpful replies!
Mar 12 2008
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
BCS wrote:

 Reply to Moritz,
 

 I like to put some data into a binary
 that should reside on disk even when the program
 is started.
 The data should be loaded in memory on request of the running part.

 The aim is to get rid of zip file distributions:
 - need to be extracted
 - files need to be keep together
 Is that possible with D?


 
 I think most OS's map in the whole program but only load stuff on demand 
 using the page swapping and virtual memeory systems.


Typically not even the whole binary file isn't even mapped, only certain 
segments are.  For example, the debug symbol segment is not mapped into 
memory.  For the parts that are, every unix-like and windows flavor in 
use today does demand paging of both the executables as well as any 
loaded libraries (be they .so or .dll as appropriate).

Later,
Brad
Mar 12 2008
prev sibling parent "Lionello Lunesu" <lionello lunesu.remove.com> writes:
On DOS, Windows9x I did this very often:

* build the exe, note its final size;
* keep this exe size in some global constant;
* rebuild the exe, make sure the size is the same;
* append other data to the exe (DOS: copy /b foo.exe+my.dat final.exe)

In code you can then simply open the exe as any other file and skip to the 
actual data.

I'm not 100% sure whether this trick still works on XP+ or linux, but I 
think it should work, with the right combination of file-open/sharding 
rights.

L.

----- Original Message ----- 
From: "Moritz Warning" <moritzwarning _nospam_web.de>
Newsgroups: digitalmars.D
Sent: Thursday, March 13, 2008 7:21 AM
Subject: start partial binary



I like to put some data into a binary
 that should reside on disk even when the program
 is started.

 The data should be loaded in memory on request of the running part.

 The aim is to get rid of zip file distributions:
 - need to be extracted
 - files need to be keep together

 Is that possible with D?
Mar 12 2008