www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A different kind of Walter? :-)

reply John Reimer <terminal.node gmail.com> writes:
See this:

http://www.beosmax.org/wiki/index.php/WalterOS

Seems Haiku, the long developed BeOS clone, is going to be a
Walter.  But where is D? :D

-JJR
Apr 12 2007
next sibling parent reply Davidl <Davidl 126.com> writes:
i would rather call it BrightOS


 See this:

 http://www.beosmax.org/wiki/index.php/WalterOS

 Seems Haiku, the long developed BeOS clone, is going to be a
 Walter.  But where is D? :D

 -JJR
Apr 12 2007
next sibling parent reply John Reimer <terminal.node gmail.com> writes:
Yeah, true.  Regardless, I'd love to see a D-based OS someday. :-)

-JJR
Apr 12 2007
parent reply Alexander Panek <alexander.panek brainsware.org> writes:
On Fri, 13 Apr 2007 05:31:49 +0000 (UTC)
John Reimer <terminal.node gmail.com> wrote:


 Yeah, true.  Regardless, I'd love to see a D-based OS someday. :-)


Just.. give me some time (and pay me, then it'll take less time,
haha! :P). Eventually hakware (from #d) made interrupts working, and
added quite some features upon my codebase - even a working garbage
collector is present at kernel-level, IIRC. :D

I'm sure going to spend some time getting a proper codebase for further
development done, and write some documentation about how to actually
get to the point of being able to write an operating system with D. I
think that's a major weak point of D, as it claims to be a systems
language, but there's no actual system written in it from scratch,
neither documentation on how to achieve that.

Kind regards,
Alex
Apr 17 2007
next sibling parent reply Dan <murpsoft hotmail.com> writes:
Alexander Panek Wrote:

 On Fri, 13 Apr 2007 05:31:49 +0000 (UTC)
 John Reimer <terminal.node gmail.com> wrote:
 

 Yeah, true.  Regardless, I'd love to see a D-based OS someday. :-)


 
 Just.. give me some time (and pay me, then it'll take less time,
 haha! :P). Eventually hakware (from #d) made interrupts working, and
 added quite some features upon my codebase - even a working garbage
 collector is present at kernel-level, IIRC. :D
 
 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.


Alex, please try to contact me.  I'm interested in assisting with this project,
and am well able to contribute.
Apr 17 2007
parent reply Alexander Panek <a.panek brainsware.org> writes:
Dan wrote:

 Alex, please try to contact me.  I'm interested in assisting with this
project, and am well able to contribute.


Do you use IRC? If so, hakware would be glad to have assistance, I bet. 
(#xana   Freenode)

I'm not actively developing the OS at the moment, as I have my finals at 
school now.. sucks up all my time, nerves and such. But I will continue 
developing when I'm finished with school, which is about the 5th of June.

Best regards,
Alex
Apr 17 2007
parent reply Dan <murpsoft hotmail.com> writes:
Alexander Panek Wrote:


 Dan wrote:

 Alex, please try to contact me.  I'm interested in assisting with this
project, and am well able to contribute.


 
 Do you use IRC? If so, hakware would be glad to have assistance, I bet. 
 (#xana   Freenode)
 
 I'm not actively developing the OS at the moment, as I have my finals at 
 school now.. sucks up all my time, nerves and such. But I will continue 
 developing when I'm finished with school, which is about the 5th of June.
 
 Best regards,
 Alex


Oh my... IRC?  I haven't used that for almost 12 years.  Like Windows 3.11.  : )

That said, what url is "Freenode", and/or how else can I contact this "hakware"?

: p  I feel like an old man, and I'm only 23.  : p
Apr 17 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Dan wrote:

 
 That said, what url is "Freenode", and/or how else can I contact this
"hakware"?


If you're using Firefox, this link should work:

irc://chat.freenode.net

Then /join #D.


Sean
Apr 17 2007
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Sean Kelly wrote:

 Dan wrote:

 That said, what url is "Freenode", and/or how else can I contact this 
 "hakware"?


 
 If you're using Firefox, this link should work:
 
 irc://chat.freenode.net
 
 Then /join #D.
 
 
 Sean


Alert
------------------
Firefox doesn't know how to open this address, because the protocol 
(irc) isn't associated with any program.

Darn.  I was all set to be impressed by Firefox's versatility. 
Apparently it doesn't handle irc internally, but just opens up an 
external client for you.  And it seems that's not set up by default -- 
at least on Windows.

--bb
Apr 17 2007
next sibling parent James Dennett <jdennett acm.org> writes:
Bill Baxter wrote:

 Sean Kelly wrote:

 Dan wrote:

 That said, what url is "Freenode", and/or how else can I contact this
 "hakware"?


 If you're using Firefox, this link should work:

 irc://chat.freenode.net

 Then /join #D.


 Sean


 
 Alert
 ------------------
 Firefox doesn't know how to open this address, because the protocol
 (irc) isn't associated with any program.
 
 Darn.  I was all set to be impressed by Firefox's versatility.
 Apparently it doesn't handle irc internally, but just opens up an
 external client for you.  And it seems that's not set up by default --
 at least on Windows.


It's an option for Firefox to handle it with an integrated
IRC client.  It's not a great client though; I think it's
still viewed as experimental, but I may be out of date.

-- James
Apr 17 2007
prev sibling next sibling parent reply Mike Parker <aldacron71 yahoo.com> writes:
Bill Baxter wrote:


 ------------------
 Firefox doesn't know how to open this address, because the protocol 
 (irc) isn't associated with any program.
 
 Darn.  I was all set to be impressed by Firefox's versatility. 
 Apparently it doesn't handle irc internally, but just opens up an 
 external client for you.  And it seems that's not set up by default -- 
 at least on Windows.


You can install ChatZilla via the Firefox/Mozilla extension system:

https://addons.mozilla.org/en-US/firefox/addon/16
Apr 17 2007
parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Mike Parker wrote:

 Bill Baxter wrote:
 

 ------------------
 Firefox doesn't know how to open this address, because the protocol 
 (irc) isn't associated with any program.

 Darn.  I was all set to be impressed by Firefox's versatility. 
 Apparently it doesn't handle irc internally, but just opens up an 
 external client for you.  And it seems that's not set up by default -- 
 at least on Windows.


 
 You can install ChatZilla via the Firefox/Mozilla extension system:
 
 https://addons.mozilla.org/en-US/firefox/addon/16


Neat.  Thanks.

--bb
Apr 17 2007
prev sibling parent Sean Kelly <sean f4.ca> writes:
Bill Baxter wrote:

 Sean Kelly wrote:

 Dan wrote:

 That said, what url is "Freenode", and/or how else can I contact this 
 "hakware"?


 If you're using Firefox, this link should work:

 irc://chat.freenode.net

 Then /join #D.


 Sean


 
 Alert
 ------------------
 Firefox doesn't know how to open this address, because the protocol 
 (irc) isn't associated with any program.
 
 Darn.  I was all set to be impressed by Firefox's versatility. 
 Apparently it doesn't handle irc internally, but just opens up an 
 external client for you.  And it seems that's not set up by default -- 
 at least on Windows.


Huh.  I'm pretty sure that used to work.  Or maybe that's from when I 
used Mozilla.  In any case, ChatZilla (the client I was thinking of) is 
(now?) a separate download for Firefox.


Sean
Apr 17 2007
prev sibling parent Pragma <ericanderton yahoo.removeme.com> writes:
Dan wrote:

 Alexander Panek Wrote:
 

 Dan wrote:

 Alex, please try to contact me.  I'm interested in assisting with this
project, and am well able to contribute.


 Do you use IRC? If so, hakware would be glad to have assistance, I bet. 
 (#xana   Freenode)

 I'm not actively developing the OS at the moment, as I have my finals at 
 school now.. sucks up all my time, nerves and such. But I will continue 
 developing when I'm finished with school, which is about the 5th of June.

 Best regards,
 Alex


 
 Oh my... IRC?  I haven't used that for almost 12 years.  Like Windows 3.11.  :
)
 
 That said, what url is "Freenode", and/or how else can I contact this
"hakware"?
 
 : p  I feel like an old man, and I'm only 23.  : p


Dont' feel bad.

When I first logged into freenode it was only a year or two ago. Being in an
interactive text environment reminded me of 
time (mis)spent on local bulletin boards a long time ago.  I could almost swear
I heard my old modem connecting...

-- 
- EricAnderton at yahoo
Apr 17 2007
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Alexander Panek wrote:

 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.


Andrei suggested that the source code for Minix, which is fairly small, 
could be transliterated from C almost directly into D. This would neatly 
resolve the issue, and provide a starting point for anyone wanting to 
take it further.
Apr 17 2007
next sibling parent reply Alexander Panek <a.panek brainsware.org> writes:
Walter Bright wrote:

 Alexander Panek wrote:

 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.


 
 Andrei suggested that the source code for Minix, which is fairly small, 
 could be transliterated from C almost directly into D. This would neatly 
 resolve the issue, and provide a starting point for anyone wanting to 
 take it further.


True. On the other hand, a kernel written purely in D is something that 
really would (and will, arr) be cool! I just find it less exciting to 
code "C with D", so to speak.

I'll make sure to announce the project, once a few milestones are 
finished, on the newsgroups.
Apr 17 2007
next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
Alexander Panek wrote:

 Walter Bright wrote:

 Alexander Panek wrote:

 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.


 Andrei suggested that the source code for Minix, which is fairly 
 small, could be transliterated from C almost directly into D. This 
 would neatly resolve the issue, and provide a starting point for 
 anyone wanting to take it further.


 
 True. On the other hand, a kernel written purely in D is something that 
 really would (and will, arr) be cool! I just find it less exciting to 
 code "C with D", so to speak.
 
 I'll make sure to announce the project, once a few milestones are 
 finished, on the newsgroups.


Pardon the curmudgeon in me, but aside from being educational and being 
able to say 'see, someone's done it', what is to be achieved from 
inventing yet another kernel?  Anything beyond a toy kernel is an 
_enormous_ effort.

I don't mean to say "don't do it".  But I'm genuinely curious what the 
goals are.  There's so many more things that would have a higher long 
term benefit to the development community as a whole that it feels like 
a waste unless it really is just a 'I want to learn more about kernels' 
sort of project.

Later,
Brad
Apr 17 2007
next sibling parent reply Alexander Panek <alexander.panek brainsware.org> writes:
On Tue, 17 Apr 2007 18:59:04 -0700
Brad Roberts <braddr puremagic.com> wrote:


 [...]


 Pardon the curmudgeon in me, but aside from being educational and
 being able to say 'see, someone's done it', what is to be achieved
 from inventing yet another kernel?  Anything beyond a toy kernel is
 an _enormous_ effort.


It's meant to be a research project, which I have started with C,
anyways, out of personal interest. System design & development is my
future occupation, hopefully, so this is educational at least for me. :)
 

 I don't mean to say "don't do it".  But I'm genuinely curious what
 the goals are.  There's so many more things that would have a higher
 long term benefit to the development community as a whole that it
 feels like a waste unless it really is just a 'I want to learn more
 about kernels' sort of project.


It might look like a "hackish" (yes, I call anything that's not
properly designed and implemented - with pedantic attention for each
detail - hackish..no offense intended, that just applies to my own
code) kernel would has its benefits in the long-term, but I think a
proper research to see what features of D can ease the pain of
operating system development will gain more knowledge and will
hopefully help me invent A Good Thing (tm).

I'm more the innovation-at-all-cost guy.. so.. that might be the
reason.

Still, there's quite a broad code base already regarding the
initialization of an x86 CPU, and according to hakware, interrupts are
now working properly. This means, that things like memory management
(paging, and such) is not far. I hope to bring in some benefits for D
with the memory management, too, so it can colaborate with the GC.
Don't know how far that's optimizable, but it's worth some research
effort, imho.

Anyways, it seems like quite a few people in the D community seem to be
interested in kernel development. So if one project evolved enough to
have overcome the beginning obstacles (mostly CPU weirdness-issues..),
I bet it's possible to form a team with a bit more momentum.

Best regards,
Alex
Apr 18 2007
parent reply "David B. Held" <dheld codelogicconsulting.com> writes:
Alexander Panek wrote:

 [...]
 It might look like a "hackish" (yes, I call anything that's not
 properly designed and implemented - with pedantic attention for each
 detail - hackish..no offense intended, that just applies to my own
 code) kernel would has its benefits in the long-term, but I think a
 proper research to see what features of D can ease the pain of
 operating system development will gain more knowledge and will
 hopefully help me invent A Good Thing (tm).
 [...]


If you guys really are young college students and can afford to sink 
time into something like this, I say go for it 100%.  It really doesn't 
matter whether it amounts to anything, because you will learn a ton from 
it and it will be a fun project.  That's just the kind of thing to do at 
your age.  At my age, you have to do things that matter (read: pay the 
bills), which is to say that you simply cannot afford to do something 
like write an OS kernel, and I really envy you.  If you're really good, 
you can make a name for yourself with it, even if the kernel is a 
commercial failure, which is worth plenty to your career.

That being said, I think it would be *particularly* clever of you to use 
your start-from-scratch kernel to explore new ideas in multi-processing 
and massive multithreading, but this would imply having decent multi-CPU 
hardware to play with.  Most of the OS support we have is decades old 
technology hacked together to keep up with the hardware, but maybe you 
have an opportunity to do it right from the start.  That is, to create a 
kernel where multiprocessing is as fundamental as memory management or 
process scheduling.  We try to write lock-free algorithms using just one 
special assembly instruction, and look how far we have to bend over 
backwards to make it work.  If an MP-friendly kernel gave us more 
primitives than CAS, because it was designed that way from the start 
(meaning, the process scheduler was hacked up in a new way), I suspect 
that you could make MP a lot easier on users.  Just some stuff to think 
about...

Dave
Apr 18 2007
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to use 
 your start-from-scratch kernel to explore new ideas in multi-processing 
 and massive multithreading, but this would imply having decent multi-CPU 
 hardware to play with.


I'd also like to see better operating system support for garbage 
collection - either making gc an operating system service, or providing 
hooks to the virtual memory subsystem so the gc can tell when pages are 
dirty.
Apr 19 2007
next sibling parent reply Alexander Panek <alexander.panek brainsware.org> writes:
On Thu, 19 Apr 2007 01:07:22 -0700
Walter Bright <newshound1 digitalmars.com> wrote:


 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you
 to use your start-from-scratch kernel to explore new ideas in
 multi-processing and massive multithreading, but this would imply
 having decent multi-CPU hardware to play with.


 
 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or
 providing hooks to the virtual memory subsystem so the gc can tell
 when pages are dirty.


Well, something along the lines of that is what I was thinking of. I'm
not so sure how to make the GC an operating system service, though..?
Apr 19 2007
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Alexander Panek wrote:

 On Thu, 19 Apr 2007 01:07:22 -0700
 Walter Bright <newshound1 digitalmars.com> wrote:
 

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you
 to use your start-from-scratch kernel to explore new ideas in
 multi-processing and massive multithreading, but this would imply
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or
 providing hooks to the virtual memory subsystem so the gc can tell
 when pages are dirty.


 
 Well, something along the lines of that is what I was thinking of. I'm
 not so sure how to make the GC an operating system service, though..?


http://www-cs.canisius.edu/~hertzm/bc.html has a nice idea on support 
the OS could give to garbage collection: they modified Linux to notify 
the process when it was about to get some of its memory swapped out. 
Then the process could perform a quick GC (focusing on freeing up pages 
of memory) and tell the OS which pages to unmap or swap out. The GC 
freeing pages means the GC can tell the OS memory manager that certain 
pages can be thrown out without saving them to disk first (no disk 
writes is faster than disk writes :) ).
If it can't avoid swapping pages out, the GC checks the page about to be 
swapped out for pointers and marks the pointed-to objects as referenced 
by a swapped-out page, then tells the OS to put that page back at the 
head of the swap-out queue (since it has just been touched by the GC, it 
was removed from that queue). In this way, their GC also avoids touching 
swapped-out pages except in rare cases, so most GC cycles shouldn't 
cause a single page fault.
Pretty cool stuff.

About making it a system service: I'm not sure how much help that would 
be, though I guess it could cut down on privilege level switches (caused 
by warnings about swapping and/or system calls to nominate pages for 
swapping) if the kernel could perform something like the above GC itself.
Apr 19 2007
parent reply Sean Kelly <sean f4.ca> writes:
Frits van Bommel wrote:

 Alexander Panek wrote:

 On Thu, 19 Apr 2007 01:07:22 -0700
 Walter Bright <newshound1 digitalmars.com> wrote:


 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you
 to use your start-from-scratch kernel to explore new ideas in
 multi-processing and massive multithreading, but this would imply
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or
 providing hooks to the virtual memory subsystem so the gc can tell
 when pages are dirty.


 Well, something along the lines of that is what I was thinking of. I'm
 not so sure how to make the GC an operating system service, though..?


 
 http://www-cs.canisius.edu/~hertzm/bc.html has a nice idea on support 
 the OS could give to garbage collection: they modified Linux to notify 
 the process when it was about to get some of its memory swapped out. 
 Then the process could perform a quick GC (focusing on freeing up pages 
 of memory) and tell the OS which pages to unmap or swap out. The GC 
 freeing pages means the GC can tell the OS memory manager that certain 
 pages can be thrown out without saving them to disk first (no disk 
 writes is faster than disk writes :) ).


I can think of at least two papers on this topic for anyone interested. 
  One is "Reducing Garbage Collector Cache Missed" by Hans Boehm, and 
the other is "Garbage Collection Without Paging" by Hertz, Feng, and 
Berger.  Both should be available with a bit of googling.


Sean
Apr 19 2007
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Sean Kelly wrote:

 Frits van Bommel wrote:

 http://www-cs.canisius.edu/~hertzm/bc.html has a nice idea on support 
 the OS could give to garbage collection: they modified Linux to notify 
 the process when it was about to get some of its memory swapped out. 
 Then the process could perform a quick GC (focusing on freeing up 
 pages of memory) and tell the OS which pages to unmap or swap out. The 
 GC freeing pages means the GC can tell the OS memory manager that 
 certain pages can be thrown out without saving them to disk first (no 
 disk writes is faster than disk writes :) ).


 
 I can think of at least two papers on this topic for anyone interested. 
  One is "Reducing Garbage Collector Cache Missed" by Hans Boehm, and the 
 other is "Garbage Collection Without Paging" by Hertz, Feng, and 
 Berger.  Both should be available with a bit of googling.


The second one is actually available at the link I posted, but thanks 
for the first one, I don't think I read that one yet.
Apr 19 2007
prev sibling next sibling parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 
 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or providing 
 hooks to the virtual memory subsystem so the gc can tell when pages are 
 dirty.


Same here.  That the OS is aware of context switches and such should 
allow for some clever methods of garbage collection.  I'm actually a bit 
surprised that MS didn't do anything with GC support for Vista, given 
their push with CLI.


Sean
Apr 19 2007
parent reply Dave <Dave_member pathlink.com> writes:
Sean Kelly wrote:

 Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or 
 providing hooks to the virtual memory subsystem so the gc can tell 
 when pages are dirty.


 
 Same here.  That the OS is aware of context switches and such should 
 allow for some clever methods of garbage collection.  I'm actually a bit 
 surprised that MS didn't do anything with GC support for Vista, given 
 their push with CLI.
 


Not only that, but I've never even read/heard of any plans to add a GC to any
main-stream OS's.

Seems to me there's got to be a reason for this (not having an OS implement a
GC). I wonder what 
that would be, since now-a-days catering to application development is such a
vital thing for any 
commercial OS?


 
 Sean
Apr 20 2007
parent reply Dan <murpsoft hotmail.com> writes:
Dave Wrote:

 Seems to me there's got to be a reason for this (not having an OS implement a
GC). I wonder what 
 that would be, since now-a-days catering to application development is such a
vital thing for any 
 commercial OS?


Well, people are developing them.  The reason they aren't included by default
is a mystery to me, then most OS developers have left the entire process this
gigantic undocumented undescribable mess that explains why most new OS starts
just give up.  I think it also explains why currently widely-used OS's still
don't implement any additional features.

It's difficult, and they don't have to.

That said, I've been architecting a ring -1 program that multiplexes hardware
resources without any drivers, HAL or OS underneath it; taking the exokernel
principle to the farthest extreme.

My exokernel isn't even a kernel, it's just a gatekeeper.  I was trying to
think of a good name for it:

Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the
first that came to mind.
Apr 20 2007
next sibling parent reply Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Dan wrote:

 Dave Wrote:

 Seems to me there's got to be a reason for this (not having an OS implement a
GC). I wonder what 
 that would be, since now-a-days catering to application development is such a
vital thing for any 
 commercial OS?


 
 Well, people are developing them.  The reason they aren't included by default
is a mystery to me, then most OS developers have left the entire process this
gigantic undocumented undescribable mess that explains why most new OS starts
just give up.  I think it also explains why currently widely-used OS's still
don't implement any additional features.
 
 It's difficult, and they don't have to.
 
 That said, I've been architecting a ring -1 program that multiplexes hardware
resources without any drivers, HAL or OS underneath it; taking the exokernel
principle to the farthest extreme.
 
 My exokernel isn't even a kernel, it's just a gatekeeper.  I was trying to
think of a good name for it:
 
 Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the
first that came to mind.


Going the route of those first couple ideas (naming after psychopomps) there's
the 
possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the
way). 
Actually I think that's a perfect theme for selecting a name for something like
that. 
Sounds like an interesting project indeed.

-- Chris Nicholson-Sauls
Apr 20 2007
parent reply Dan <murpsoft hotmail.com> writes:
Chris Nicholson-Sauls Wrote:

 Dan wrote:

 Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the
first that came to mind.


 
 Going the route of those first couple ideas (naming after psychopomps) there's
the 
 possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the
way). 
 Actually I think that's a perfect theme for selecting a name for something
like that. 
 Sounds like an interesting project indeed.
 
 -- Chris Nicholson-Sauls


I had to look psychopomps up, thanks Chris, I learned something new today.  I
initially took 'he who opens the way' in a buddhist sense, which would be cool,
but Wepwawet was actually more scout-for-the-military sort of God.

Check the mythology for Nandi, that's not too bad either.

So, if I named it after Mr Pete, they catholics would probably hang me.
If I name it 'Garmies', the Mormons will.
If I name it 'Maat' I would probably get away with it;

Maat is the Egyptian god of justice, who's feather would be weighed against
one's heart upon death; and who ancient judges and even gods would seek
approval.

It's also unique enough that I don't think there'll be a name collision.  So
Maat it will be?

Now to keep working on Walnut.  : p
Apr 20 2007
parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Dan wrote:

 Chris Nicholson-Sauls Wrote:

 Dan wrote:

 Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the
first that came to mind.


 Going the route of those first couple ideas (naming after psychopomps) there's
the 
 possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the
way). 
 Actually I think that's a perfect theme for selecting a name for something
like that. 
 Sounds like an interesting project indeed.

 -- Chris Nicholson-Sauls


 
 I had to look psychopomps up, thanks Chris, I learned something new today.  I
initially took 'he who opens the way' in a buddhist sense, which would be cool,
but Wepwawet was actually more scout-for-the-military sort of God.
 
 Check the mythology for Nandi, that's not too bad either.
 
 So, if I named it after Mr Pete, they catholics would probably hang me.
 If I name it 'Garmies', the Mormons will.
 If I name it 'Maat' I would probably get away with it;
 
 Maat is the Egyptian god of justice, who's feather would be weighed against
one's heart upon death; and who ancient judges and even gods would seek
approval.
 
 It's also unique enough that I don't think there'll be a name collision.  So
Maat it will be?
 
 Now to keep working on Walnut.  : p


I just don't like anything else with a feather for its obvious logo.  I've got
a side 
project named Wren that intends to use that... ;)  Then again, Apache beat us
both to it, 
so who cares.

Okay, shutting up now.  (Du'a Netjer.)

-- Chris Nicholson-Sauls
Apr 20 2007
prev sibling parent reply torhu <fake address.dude> writes:
Dan wrote:

 My exokernel isn't even a kernel, it's just a gatekeeper.  I was trying to
think of a good name for it:
 
 Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the
first that came to mind.


I think you forgot Zuul. :)
Apr 21 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
torhu wrote:

 Dan wrote:

 My exokernel isn't even a kernel, it's just a gatekeeper.  I was
 trying to think of a good name for it:

 Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies
 were the first that came to mind.


 
 I think you forgot Zuul. :)


There is no kernel, only Zuul.

And if you remap StdOut to StdIn and vice-versa, you get total protonic
reversal.  That's bad.

	-- Daniel

-- 
int getRandomNumber()
{
    return 4; // chosen by fair dice roll.
              // guaranteed to be random.
}

http://xkcd.com/

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/
Apr 21 2007
parent Brad Anderson <brad dsource.org> writes:
Daniel Keep wrote:

 And if you remap StdOut to StdIn and vice-versa, you get total protonic
 reversal.  That's bad.


Right. That's bad. Okay. All right. Important safety tip. Thanks, Egon.
Apr 21 2007
prev sibling parent reply Lionello Lunesu <lio lunesu.remove.com> writes:
Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 
 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or providing 
 hooks to the virtual memory subsystem so the gc can tell when pages are 
 dirty.


I wish there were a function like VirtualRealloc that would grow a 
memory block without EVER having to physically move the data.

(That doesn't exist in Win32, does it?)

L.
Apr 19 2007
parent reply Sean Kelly <sean f4.ca> writes:
Lionello Lunesu wrote:

 Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or 
 providing hooks to the virtual memory subsystem so the gc can tell 
 when pages are dirty.


 
 I wish there were a function like VirtualRealloc that would grow a 
 memory block without EVER having to physically move the data.
 
 (That doesn't exist in Win32, does it?)


I don't think so.  If you have some data whose logical address range is 
bounded by other data, I don't think it would be possible for it to grow 
without being moved.


Sean
Apr 20 2007
next sibling parent reply "David B. Held" <dheld codelogicconsulting.com> writes:
Sean Kelly wrote:

 Lionello Lunesu wrote:

 [...]
 I wish there were a function like VirtualRealloc that would grow a 
 memory block without EVER having to physically move the data.

 (That doesn't exist in Win32, does it?)


 
 I don't think so.  If you have some data whose logical address range is 
 bounded by other data, I don't think it would be possible for it to grow 
 without being moved.


Well, there's at least two solutions to this.  One is to use infinite 
memory, in which case you can allocate at addresses that are infinitely 
far apart, allowing for unlimited growth in data structures.

The second is to use quantum superposition to allow storage of distinct 
data at the same location.  The trick is getting the state to collapse 
in a way that makes all the data recoverable, which is nothing more than 
an engineering detail, really.  We should get an intern to work on these 
projects over the summer.

Dave
Apr 20 2007
parent Sean Kelly <sean f4.ca> writes:
David B. Held wrote:

 Sean Kelly wrote:

 Lionello Lunesu wrote:

 [...]
 I wish there were a function like VirtualRealloc that would grow a 
 memory block without EVER having to physically move the data.

 (That doesn't exist in Win32, does it?)


 I don't think so.  If you have some data whose logical address range 
 is bounded by other data, I don't think it would be possible for it to 
 grow without being moved.


 
 Well, there's at least two solutions to this.  One is to use infinite 
 memory, in which case you can allocate at addresses that are infinitely 
 far apart, allowing for unlimited growth in data structures.
 
 The second is to use quantum superposition to allow storage of distinct 
 data at the same location.  The trick is getting the state to collapse 
 in a way that makes all the data recoverable, which is nothing more than 
 an engineering detail, really.  We should get an intern to work on these 
 projects over the summer.


And Google's "summer of code" has already stopped taking applications, 
too.  Darn it!


Sean
Apr 20 2007
prev sibling parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message 
news:f09ouq$2ium$1 digitalmars.com...

 Lionello Lunesu wrote:

 Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or providing 
 hooks to the virtual memory subsystem so the gc can tell when pages are 
 dirty.


 I wish there were a function like VirtualRealloc that would grow a memory 
 block without EVER having to physically move the data.

 (That doesn't exist in Win32, does it?)


 I don't think so.  If you have some data whose logical address range is 
 bounded by other data, I don't think it would be possible for it to grow 
 without being moved.


I don't care if there address changes, just as long as there's no data being 
copied physically. I mean, a block of continuous memory might already be 
all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
that returns the address of a new virtual memory block, but which is using 
the same physical memory.. 'Free' array resizes, surely this is worth 
something!

L.
Apr 20 2007
parent reply Sean Kelly <sean f4.ca> writes:
Lionello Lunesu wrote:

 "Sean Kelly" <sean f4.ca> wrote in message 
 news:f09ouq$2ium$1 digitalmars.com...

 Lionello Lunesu wrote:

 Walter Bright wrote:

 David B. Held wrote:

 That being said, I think it would be *particularly* clever of you to 
 use your start-from-scratch kernel to explore new ideas in 
 multi-processing and massive multithreading, but this would imply 
 having decent multi-CPU hardware to play with.


 I'd also like to see better operating system support for garbage 
 collection - either making gc an operating system service, or providing 
 hooks to the virtual memory subsystem so the gc can tell when pages are 
 dirty.


 I wish there were a function like VirtualRealloc that would grow a memory 
 block without EVER having to physically move the data.

 (That doesn't exist in Win32, does it?)


 I don't think so.  If you have some data whose logical address range is 
 bounded by other data, I don't think it would be possible for it to grow 
 without being moved.


 
 I don't care if there address changes, just as long as there's no data being 
 copied physically. I mean, a block of continuous memory might already be 
 all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
 that returns the address of a new virtual memory block, but which is using 
 the same physical memory.. 'Free' array resizes, surely this is worth 
 something!


Hm, assuming there is nothing else of the page(s) but the data to move, 
it should be possible to do so without explicit copying.  I have no idea 
is VirtualAlloc can do this, but it's definitely possible.


Sean
Apr 20 2007
parent reply Dan <murpsoft hotmail.com> writes:
Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data being 
 copied physically. I mean, a block of continuous memory might already be 
 all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
 that returns the address of a new virtual memory block, but which is using 
 the same physical memory.. 'Free' array resizes, surely this is worth 
 something!


 
 Hm, assuming there is nothing else of the page(s) but the data to move, 
 it should be possible to do so without explicit copying.  I have no idea 
 is VirtualAlloc can do this, but it's definitely possible.


Oh my!

On the x86, this sort of implementation would slaughter your performance, as
you'd get massive numbers of cache and page misses.

You could theoretically implement a system on the x86 that would virtualize
memory locations arbitrarily, but.... ewww...

I see two effective ways to realloc memory in a space too small for it:

1) move the data you are trying to reallocate to a larger space.

2) move the data that is confining it; which requires updating pointers and
such to that data, which you can't realistically track on the x86.
Apr 20 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Dan wrote:

 Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data being 
 copied physically. I mean, a block of continuous memory might already be 
 all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
 that returns the address of a new virtual memory block, but which is using 
 the same physical memory.. 'Free' array resizes, surely this is worth 
 something!


 Hm, assuming there is nothing else of the page(s) but the data to move, 
 it should be possible to do so without explicit copying.  I have no idea 
 is VirtualAlloc can do this, but it's definitely possible.


 
 Oh my!
 
 On the x86, this sort of implementation would slaughter your performance, as
you'd get massive numbers of cache and page misses.
 
 You could theoretically implement a system on the x86 that would virtualize
memory locations arbitrarily, but.... ewww...


Yeah.  I was thinking that the way to do this would be in the virtual 
memory manager.  When the pages are moved the page table is updated, etc.


Sean
Apr 20 2007
parent reply Dan <murpsoft hotmail.com> writes:
Sean Kelly Wrote:


 Dan wrote:

 Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data being 
 copied physically. I mean, a block of continuous memory might already be 
 all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
 that returns the address of a new virtual memory block, but which is using 
 the same physical memory.. 'Free' array resizes, surely this is worth 
 something!


 Hm, assuming there is nothing else of the page(s) but the data to move, 
 it should be possible to do so without explicit copying.  I have no idea 
 is VirtualAlloc can do this, but it's definitely possible.


 
 Oh my!
 
 On the x86, this sort of implementation would slaughter your performance, as
you'd get massive numbers of cache and page misses.
 
 You could theoretically implement a system on the x86 that would virtualize
memory locations arbitrarily, but.... ewww...


 
 Yeah.  I was thinking that the way to do this would be in the virtual 
 memory manager.  When the pages are moved the page table is updated, etc.
 
 
 Sean


That works Sean, if your arrays are only ever split if they're bigger than a
page; in which case letting the OS split them and 'handle it' is the best way. 
You *can* just declare a 64kb array and it'll handle it well enough.

If you're talking about arbitrarily splitting arrays whenever they happen to
collide, and hoping the paging system will handle it, you'll be grabbing 4kb
segments for each misplaced segment.  What's almost as bad, is that you'll have
to have additional code tracking what segments belong to what virtual array.
Apr 20 2007
parent Sean Kelly <sean f4.ca> writes:
Dan wrote:

 Sean Kelly Wrote:
 

 Dan wrote:

 Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data being 
 copied physically. I mean, a block of continuous memory might already be 
 all-over-the-place in physical memory. That's what I want: a VirtualRealloc 
 that returns the address of a new virtual memory block, but which is using 
 the same physical memory.. 'Free' array resizes, surely this is worth 
 something!


 Hm, assuming there is nothing else of the page(s) but the data to move, 
 it should be possible to do so without explicit copying.  I have no idea 
 is VirtualAlloc can do this, but it's definitely possible.


 Oh my!

 On the x86, this sort of implementation would slaughter your performance, as
you'd get massive numbers of cache and page misses.

 You could theoretically implement a system on the x86 that would virtualize
memory locations arbitrarily, but.... ewww...


 Yeah.  I was thinking that the way to do this would be in the virtual 
 memory manager.  When the pages are moved the page table is updated, etc.


 Sean


 
 That works Sean, if your arrays are only ever split if they're bigger than a
page; in which case letting the OS split them and 'handle it' is the best way. 
You *can* just declare a 64kb array and it'll handle it well enough.
 
 If you're talking about arbitrarily splitting arrays whenever they happen to
collide, and hoping the paging system will handle it, you'll be grabbing 4kb
segments for each misplaced segment.  What's almost as bad, is that you'll have
to have additional code tracking what segments belong to what virtual array.


This is basically how the DMD garbage collector already works, so it 
could be done almost "for free" if there were VMM calls to handle it.


Sean
Apr 21 2007
prev sibling parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Dan" <murpsoft hotmail.com> wrote in message 
news:f0b16i$2k5p$1 digitalmars.com...

 Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data 
 being
 copied physically. I mean, a block of continuous memory might already 
 be
 all-over-the-place in physical memory. That's what I want: a 
 VirtualRealloc
 that returns the address of a new virtual memory block, but which is 
 using
 the same physical memory.. 'Free' array resizes, surely this is worth
 something!


 Hm, assuming there is nothing else of the page(s) but the data to move,
 it should be possible to do so without explicit copying.  I have no idea
 is VirtualAlloc can do this, but it's definitely possible.


 Oh my!

 On the x86, this sort of implementation would slaughter your performance,
 as you'd get massive numbers of cache and page misses.


Is a cache using virtual addresses or physical addresses?


 You could theoretically implement a system on the x86 that would 
 virtualize memory locations arbitrarily, but.... ewww...


Isn't this what's happening already?

L.
Apr 21 2007
parent reply Dan <murpsoft hotmail.com> writes:
Lionello Lunesu Wrote:


 
 "Dan" <murpsoft hotmail.com> wrote in message 
 news:f0b16i$2k5p$1 digitalmars.com...

 Sean Kelly Wrote:

 Lionello Lunesu wrote:

 I don't care if there address changes, just as long as there's no data 
 being
 copied physically. I mean, a block of continuous memory might already 
 be
 all-over-the-place in physical memory. That's what I want: a 
 VirtualRealloc
 that returns the address of a new virtual memory block, but which is 
 using
 the same physical memory.. 'Free' array resizes, surely this is worth
 something!


 Hm, assuming there is nothing else of the page(s) but the data to move,
 it should be possible to do so without explicit copying.  I have no idea
 is VirtualAlloc can do this, but it's definitely possible.


 Oh my!

 On the x86, this sort of implementation would slaughter your performance,
 as you'd get massive numbers of cache and page misses.


 
 Is a cache using virtual addresses or physical addresses?


The cache loads in _blocks_ of physical memory, which can be specified using a
virtual address or a physical address regardless; "virtual memory" just means
you're using a translated address.


 You could theoretically implement a system on the x86 that would 
 virtualize memory locations arbitrarily, but.... ewww...


 
 Isn't this what's happening already?


Not arbitrarily sized ones.  If you have a buffer like so:

[slice0,slice1,slice2] = virtual address table
[minemineminemine][7Mb][minemineminemine][24b][minemine] = data buffer

Iterate through your virtual array, and you're going to get a hit on the
virtual address table, then on slice0, then on slice1 which will also load the
24b segment and probably slice2 for you.

So you get 3 requests to memory, which if they're not already in the cache,
will cause a cache miss.  If they're not in memory (swap), they'll cause a page
miss.

If you force that memory to stay in one place, it'll all load in 1 shot instead
of 3.

If you're curious, a cache miss costs several dozen cycles; while a page miss
will cost you thousands.

Additionally, your arbitrary virtual address table takes up alot of space if
someone has alot of small dynamic arrays.  Worst case, your virtual table can
be double the size of the data itself.

Sincerely,
Dan
Apr 21 2007
parent reply Dan <murpsoft hotmail.com> writes:
Dan Wrote:


 Lionello Lunesu Wrote:

 Isn't this what's happening already?


 
 Not arbitrarily sized ones.  If you have a buffer like so:
 
 [slice0,slice1,slice2] = virtual address table
 [minemineminemine][7Mb][minemineminemine][24b][minemine] = data buffer
 
 Iterate through your virtual array, and you're going to get a hit on the
virtual address table, then on slice0, then on slice1 which will also load the
24b segment and probably slice2 for you.


On another note, this is done by the x86 system already for memory pages (4kb
fixed block size).  If you need a 32kb array, you can take 8 pages from
anywhere in memory and call it yours, and iterate through the array and it'll
translate the address for you in the hardware via the "virtual memory" process.

: )
Apr 21 2007
parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Dan" <murpsoft hotmail.com> wrote in message 
news:f0di8h$q2e$1 digitalmars.com...

 Dan Wrote:


 Lionello Lunesu Wrote:

 Isn't this what's happening already?


 Not arbitrarily sized ones.  If you have a buffer like so:

 [slice0,slice1,slice2] = virtual address table
 [minemineminemine][7Mb][minemineminemine][24b][minemine] = data buffer

 Iterate through your virtual array, and you're going to get a hit on the 
 virtual address table, then on slice0, then on slice1 which will also 
 load the 24b segment and probably slice2 for you.


 On another note, this is done by the x86 system already for memory pages 
 (4kb fixed block size).  If you need a 32kb array, you can take 8 pages 
 from anywhere in memory and call it yours, and iterate through the array 
 and it'll translate the address for you in the hardware via the "virtual 
 memory" process.


Well, that was *exactly* what I was getting at! OK, granted, it's only done 
in 4KB increments, but that's when a regular array resize (copying the 
contents over to a new place) would actually start harming performance.

L.
Apr 22 2007
parent 0ffh <spam frankhirsch.net> writes:
Lionello Lunesu wrote:

 Well, that was *exactly* what I was getting at! OK, granted, it's only done 
 in 4KB increments, but that's when a regular array resize (copying the 
 contents over to a new place) would actually start harming performance.


That sounds enticing!
If you could put garbage collection into the kernel - even with tile-sized
granularity - that would make many things easier. And finally put the idle
thread to good use... :-)))


Regards, Frank
Apr 22 2007
prev sibling next sibling parent reply Alexander Panek <alexander.panek brainsware.org> writes:
On Wed, 18 Apr 2007 22:33:18 -0700
"David B. Held" <dheld codelogicconsulting.com> wrote:


 Alexander Panek wrote:

 [...]
 It might look like a "hackish" (yes, I call anything that's not
 properly designed and implemented - with pedantic attention for each
 detail - hackish..no offense intended, that just applies to my own
 code) kernel would has its benefits in the long-term, but I think a
 proper research to see what features of D can ease the pain of
 operating system development will gain more knowledge and will
 hopefully help me invent A Good Thing (tm).
 [...]


 
 If you guys really are young college students and can afford to sink 
 time into something like this, I say go for it 100%.  It really
 doesn't matter whether it amounts to anything, because you will learn
 a ton from it and it will be a fun project.  That's just the kind of
 thing to do at your age.  At my age, you have to do things that
 matter (read: pay the bills), which is to say that you simply cannot
 afford to do something like write an OS kernel, and I really envy
 you.  If you're really good, you can make a name for yourself with
 it, even if the kernel is a commercial failure, which is worth plenty
 to your career.
 
 That being said, I think it would be *particularly* clever of you to
 use your start-from-scratch kernel to explore new ideas in
 multi-processing and massive multithreading, but this would imply
 having decent multi-CPU hardware to play with.  Most of the OS
 support we have is decades old technology hacked together to keep up
 with the hardware, but maybe you have an opportunity to do it right
 from the start.  That is, to create a kernel where multiprocessing is
 as fundamental as memory management or process scheduling.  We try to
 write lock-free algorithms using just one special assembly
 instruction, and look how far we have to bend over backwards to make
 it work.  If an MP-friendly kernel gave us more primitives than CAS,
 because it was designed that way from the start (meaning, the process
 scheduler was hacked up in a new way), I suspect that you could make
 MP a lot easier on users.  Just some stuff to think about...
 
 Dave


Now those are some ideas and encouragement. Thanks! :)
Apr 19 2007
parent reply Dan <murpsoft hotmail.com> writes:
Alexander Panek Wrote:

 Now those are some ideas and encouragement. Thanks! :)


Indeed.

Lately a big thing in kernel development world is to move everything possible
out of the kernel.  Why?

The kernel is supposed to provide one thing and one thing only: securely
allowing multiple programs to run.

To that end, the best kernel is the smallest one that is proveably secure and
can efficiently use other programs to do hardware management stuff
(read:drivers)

So when I think of a kernel, I write an exokernel that controls the system by
means of controlling the:

1) memory
2) ports
3) thread/process switcher attached to timer interrupt

If you control memory, you control what agents can be loaded where and when. 
If an agent cannot get memory for a code block, they obviously cannot run it.

Just my view.  As I said, I'm interested in helping.  Unfortunately, I've been
pulling overtime the last few so I haven't had a chance to try getting on
freenode.net again.
Apr 19 2007
parent reply 0ffh <spam frankhirsch.net> writes:
Dan wrote:

 Lately a big thing in kernel development world is to move everything possible
out of the kernel.


I don't know how lately should be defined in that context, but when I 
did the research for my diploma thesis (design and implementation of a 
RTOS-Kernel on a Fujitsu microcontroller) in 2001 there where already
numerous papers to be found on the exokernel subject.
It was a fascinating idea then, and still am. I sure like the approach.


 Just my view.  As I said, I'm interested in helping.


Maybe I'd also be able to throw in a few hours of work... :)

Happy hacking, 0ffh
Apr 19 2007
parent Dan <murpsoft hotmail.com> writes:
0ffh Wrote:

 RTOS-Kernel on a Fujitsu microcontroller) in 2001 there where already
 numerous papers to be found on the exokernel subject.
 It was a fascinating idea then, and still....


Yeah, the MIT boys back then did a paper on it, and make Xok, which IMHO
demonstrated the concept but otherwise sucked because they used the wrong
strategies to 'import' code.

When you have an exokernel, you need to be able to delegate gc, thread/process
switching (cpu driver), paging and gc (memory driver), port drivers, dma and
such out.  Doing that efficiently IMHO requires using shared memory and
priveleges rather than strictly RPC.

I think I'll probably spend some time on this code this weekend; if I *ever*
get a URL.
Apr 19 2007
prev sibling parent Sean Kelly <sean f4.ca> writes:
David B. Held wrote:

 
 That being said, I think it would be *particularly* clever of you to use 
 your start-from-scratch kernel to explore new ideas in multi-processing 
 and massive multithreading, but this would imply having decent multi-CPU 
 hardware to play with.  Most of the OS support we have is decades old 
 technology hacked together to keep up with the hardware, but maybe you 
 have an opportunity to do it right from the start.  That is, to create a 
 kernel where multiprocessing is as fundamental as memory management or 
 process scheduling.  We try to write lock-free algorithms using just one 
 special assembly instruction, and look how far we have to bend over 
 backwards to make it work.  If an MP-friendly kernel gave us more 
 primitives than CAS, because it was designed that way from the start 
 (meaning, the process scheduler was hacked up in a new way), I suspect 
 that you could make MP a lot easier on users.  Just some stuff to think 
 about...


Great idea :-)  BeOS had very good multithreaded performance, but it 
didn't offer a new way to write multithreaded code.  If such a beast 
existed, I'd definitely be interested in playing with it.


Sean
Apr 19 2007
prev sibling parent reply Georg Wrede <georg nospam.org> writes:
Brad Roberts wrote:

 Alexander Panek wrote:
 

 Walter Bright wrote:


 Alexander Panek wrote:


 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.



 Andrei suggested that the source code for Minix, which is fairly 
 small, could be transliterated from C almost directly into D. This 
 would neatly resolve the issue, and provide a starting point for 
 anyone wanting to take it further.



 True. On the other hand, a kernel written purely in D is something 
 that really would (and will, arr) be cool! I just find it less 
 exciting to code "C with D", so to speak.

 I'll make sure to announce the project, once a few milestones are 
 finished, on the newsgroups.


 
 
 Pardon the curmudgeon in me, but aside from being educational and being 
 able to say 'see, someone's done it', what is to be achieved from 
 inventing yet another kernel? 


The advertising value is simply enormous.

A lot of car makers (the smaller, the likelier) have always had some top 
models that give the brand prestige. Those cars were sold in very small 
numbers, and they made losses with them. But that was on purpose, 
because the prestige value sold much more "regular cars". I've seen it 
with motorcycles, cameras, watches, liquor, perfumes, and in service 
marketing!

In the 80's many companies had doubts about Turbo Pascal. But when 
Borland demonstrated that it was not only possible but downright 
straightforward to write a TSR with it, the acceptance was huge. It 
became a Serious Programming Language.

Oh, and speaking of cameras, somebody buying a camera wants it to have 
all the bells and whistles, even when they never end up using them. But 
they /have to be there/.


  Anything beyond a toy kernel is an _enormous_ effort.


We don't have to beat Linux. All we need is to have a valid 
counterargument when the [yourLanguageHere]weenies start saying D is not 
for systems work.

Besides, the tinier the merrier. A small (e.g. Minix like) actually 
running OS (even if totally toy) is a good vehicle for trying out all 
sorts of things, without having to buy Annotated Linux Kernel Source 
Code ($200) or even worse, having to read it through only to test your 
little kernel heap algorithm.



---



For the less-than-midle-aged of us: a TSR (Terminate and Say Resident) 
was a killer technology of the time. Microcomputers (as "PCs" were 
called, emphasizing their laughable lack of memory and horsepower -- as 
opposed to Minicomputers, Computers and Mainframes) were single-user 
single-tasking, which effectively meant that you can't start a 
calculator while you're in the text editor, or make notes or look at 
your calendar while in e-mail (yes, we had e-mail at the time, in Europe 
too, before the WWW, or even before the Internet, or Usenet became 
available).

A TSR let you switch between two tasks, usually with some special key 
combination (like pressing both shifts or Ctrl-shift), and that was 
implemened with a wedge into the keyboard interrupt routine. Yes, you 
could have several TSRs running and switch between all of them -- if you 
were savvy. There was even a "multitasking" app, called Software 
Carousel, which let you run arbitrary programs while the others were 
suspended. But that came much later.

But the notion of TSR was common with the CP/M operating system (that 
the world used until PC-DOS (and later MS-DOS) usurped them) way before 
Bill got into the OS-game.
Apr 25 2007
parent BCS <BCS pathlink.com> writes:
Georg Wrede wrote:

 Brad Roberts wrote:
 

  Anything beyond a toy kernel is an _enormous_ effort.


 
 
 We don't have to beat Linux. All we need is to have a valid 
 counterargument when the [yourLanguageHere]weenies start saying D is not 
 for systems work.
 
 Besides, the tinier the merrier. A small (e.g. Minix like) actually 
 running OS (even if totally toy) is a good vehicle for trying out all 
 sorts of things, without having to buy Annotated Linux Kernel Source 
 Code ($200) or even worse, having to read it through only to test your 
 little kernel heap algorithm.
 
 


Besides, there is all sorts of fun to be had in the "really really small 
OS" department. I've been toying with the idea of how to architect a 
micro-kernel with only about 5 systems calls (remap memory, create 
process, kill process, switch to process, jump to on interrupt, fixed 
size IPC). If I wasn't already +3x booked for time, I would be figuring 
out how to wright it, in D.
Apr 25 2007
prev sibling parent Georg Wrede <georg nospam.org> writes:
Alexander Panek wrote:

 Walter Bright wrote:
 

 Alexander Panek wrote:


 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.



 Andrei suggested that the source code for Minix, which is fairly 
 small, could be transliterated from C almost directly into D. This 
 would neatly resolve the issue, and provide a starting point for 
 anyone wanting to take it further.


 
 
 True. On the other hand, a kernel written purely in D is something that 
 really would (and will, arr) be cool! I just find it less exciting to 
 code "C with D", so to speak.
 
 I'll make sure to announce the project, once a few milestones are 
 finished, on the newsgroups.


Linus T. started the same way. He had a Minix running on his machine, 
and piece by piece replaced stuff there, until no original code was 
left. (And then he /really/ started writing Linux.)

We should do the same thing. Figuratively speaking. So, do the thing in 
Cish D, and only when it gets bootable, start rewriting it in Real D (tm).
Apr 25 2007
prev sibling next sibling parent reply John Reimer <terminal.node gmail.com> writes:
On Tue, 17 Apr 2007 17:26:59 -0700, Walter Bright wrote:


 Alexander Panek wrote:

 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.


 
 Andrei suggested that the source code for Minix, which is fairly small, 
 could be transliterated from C almost directly into D. This would neatly 
 resolve the issue, and provide a starting point for anyone wanting to 
 take it further.



Minix3?  That would be a great idea!  I was thinking how translating Minix3
would be a good start for a D kernel.

-JJR
Apr 17 2007
parent Alexander Panek <alexander.panek brainsware.org> writes:
On Wed, 18 Apr 2007 04:06:40 +0000 (UTC)
John Reimer <terminal.node gmail.com> wrote:


 On Tue, 17 Apr 2007 17:26:59 -0700, Walter Bright wrote:
 

 Alexander Panek wrote:

 I'm sure going to spend some time getting a proper codebase for
 further development done, and write some documentation about how
 to actually get to the point of being able to write an operating
 system with D. I think that's a major weak point of D, as it
 claims to be a systems language, but there's no actual system
 written in it from scratch, neither documentation on how to
 achieve that.


 
 Andrei suggested that the source code for Minix, which is fairly
 small, could be transliterated from C almost directly into D. This
 would neatly resolve the issue, and provide a starting point for
 anyone wanting to take it further.


 
 
 Minix3?  That would be a great idea!  I was thinking how translating
 Minix3 would be a good start for a D kernel.
 
 -JJR


Well I think, when there's enough interest and a few people who'd like
to help, we (read as: the D community) could
start doing so, anyways. It'd at least help understanding the basic
principles of operating system development. :)
Apr 18 2007
prev sibling parent reply Dan <murpsoft hotmail.com> writes:
I went to freenode and nobody was in the #D channel.  : (

Any chance someone could point me to the source?  I really derive my feelings
about a project by how sexy the source looks.
Apr 18 2007
next sibling parent Lars Ivar Igesund <larsivar igesund.net> writes:
Dan wrote:


 
 I went to freenode and nobody was in the #D channel.  : (
 
 Any chance someone could point me to the source?  I really derive my
 feelings about a project by how sexy the source looks.


You sure you went to the right place? There haven't been less than 60 there
for a long time, 74 just now.

I am logged on to irc.freenode.org:8001

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango
Apr 18 2007
prev sibling parent Sean Kelly <sean f4.ca> writes:
Dan wrote:

 I went to freenode and nobody was in the #D channel.  : (


That's weird.  There are normally like 40 people in there.


Sean
Apr 18 2007
prev sibling parent reply 0ffh <spam frankhirsch.net> writes:
Davidl wrote:

 i would rather call it BrightOS


You can find a lot of Brights here:

   http://www.the-brights.net/

:-))))
Apr 13 2007
parent reply Walter Bright <newshound1 digitalmars.com> writes:
0ffh wrote:

 Davidl wrote:

 i would rather call it BrightOS


 
 You can find a lot of Brights here:
 
   http://www.the-brights.net/
 
 :-))))
 


Eh, never heard of those guys. My lineage is replete with nerds. 
Consider my grandfather's favorite picture of himself, with his telescope.
Apr 13 2007
parent reply Georg Wrede <georg nospam.org> writes:
Walter Bright wrote:

 0ffh wrote:

 Davidl wrote:


 i would rather call it BrightOS


 You can find a lot of Brights here:

   http://www.the-brights.net/


 
 Eh, never heard of those guys. My lineage is replete with nerds. 
 Consider my grandfather's favorite picture of himself, with his telescope.


Well, consider yourself lucky!

My grandfather's _non_-favorite picture was at a Baccarat table, in 
Wiesbaden or Monte Carlo (nobody seems to remember which), where he 
spent every holiday for 40 years, meticulously (and unwillingly) losing 
all of his fortune. When he died, it was the maid who owned even the 
furniture. A bit of a surprise to my Dad and his brother.

And yes, my father is and my grandfather was an incurable nerd. Maybe at 
the casino tables the women pretended not to see this. My Dad bought 
himself an Osbourne-1 at 50, without even a hint of a serious reason or 
excuse for buying it. Twenty-five years ago this was unheard of, 
especially when you could buy a sub-compact car for the same price. Last 
year he gave it to me. Alongside with the Atari Portfolio, whic is now 
so seriously outdated it's unusable. But hey, it runs MS-DOS.

Contrast that with my HP LX-95 pocket-size PC-compatible, a late 80's 
design (also Dad's surplus, since he bought the LX-200), which 
essentially was an entire 8080 IBM PC with 1MB mem and PC-cards as "hard 
drives". Even today it beats the pants off of smart phones, 
communicators and palms, when it comes to sheer raw utility. The 
calendar, the spreadsheet, the database and the word processor are just 
unbelievably intuitive while providing advanced features. Connect the 
thing to a modem, and you can edit your /etc/passwd like you were there. 
And the bundled RPN (+arith!) calculator let's you do off-the-hip math 
that nobody with an ordinary calculator could. And solving and graphs!

Sometimes I feel that the world is not advancing. And computer keyboards 
are still laid out for the illiterate, with Caps Lock where Control 
should be. If we nerds were as aggressive as the neuro normals, we'd 
take over the world in two weeks.

Oh well, let's just say I'm lucky I got born late enough to enjoy 
computers for most of my life!
Apr 13 2007
next sibling parent Don Clugston <dac nospam.com.au> writes:
Georg Wrede wrote:

 Sometimes I feel that the world is not advancing. And computer keyboards 
 are still laid out for the illiterate, with Caps Lock where Control 
 should be. If we nerds were as aggressive as the neuro normals, we'd 
 take over the world in two weeks.


The fact that QWERTY keyboards are overwhelmingly dominant shows that 
sometimes the worst possible technology can win. One of the major design 
rules was that you can type the word "typewriter" using only the top row 
of letters. (And basically the same layout is used in non-English 
speaking countries, where 'typewriter' isn't even a word). It's hard to 
imagine a worse design goal.
Apr 13 2007
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Georg Wrede wrote:

 Walter Bright wrote:

 0ffh wrote:

 Davidl wrote:


 i would rather call it BrightOS


 You can find a lot of Brights here:

   http://www.the-brights.net/


 Eh, never heard of those guys. My lineage is replete with nerds. 
 Consider my grandfather's favorite picture of himself, with his 
 telescope.


 
 Well, consider yourself lucky!


I do. The only unlucky part is he died before I was born, I never got to 
meet him. I'd give a lot just to be able to spend an afternoon with him.

When I was in high school, nerd was a perjorative term. But I've grown 
out of all that nonsense, and am happy to be what I am.


 My grandfather's _non_-favorite picture was at a Baccarat table, in 
 Wiesbaden or Monte Carlo (nobody seems to remember which), where he 
 spent every holiday for 40 years, meticulously (and unwillingly) losing 
 all of his fortune. When he died, it was the maid who owned even the 
 furniture. A bit of a surprise to my Dad and his brother.


Everyone has feet of clay.


 And yes, my father is and my grandfather was an incurable nerd. Maybe at 
 the casino tables the women pretended not to see this.
 My Dad bought 
 himself an Osbourne-1 at 50, without even a hint of a serious reason or 
 excuse for buying it. Twenty-five years ago this was unheard of, 
 especially when you could buy a sub-compact car for the same price. Last 
 year he gave it to me. Alongside with the Atari Portfolio, whic is now 
 so seriously outdated it's unusable. But hey, it runs MS-DOS.
 
 Contrast that with my HP LX-95 pocket-size PC-compatible, a late 80's 
 design (also Dad's surplus, since he bought the LX-200), which 
 essentially was an entire 8080 IBM PC with 1MB mem and PC-cards as "hard 
 drives". Even today it beats the pants off of smart phones, 
 communicators and palms, when it comes to sheer raw utility. The 
 calendar, the spreadsheet, the database and the word processor are just 
 unbelievably intuitive while providing advanced features. Connect the 
 thing to a modem, and you can edit your /etc/passwd like you were there. 
 And the bundled RPN (+arith!) calculator let's you do off-the-hip math 
 that nobody with an ordinary calculator could. And solving and graphs!


I still have my original IBM PC. With an 8080 chip!


 Sometimes I feel that the world is not advancing. And computer keyboards 
 are still laid out for the illiterate, with Caps Lock where Control 
 should be. If we nerds were as aggressive as the neuro normals, we'd 
 take over the world in two weeks.
 
 Oh well, let's just say I'm lucky I got born late enough to enjoy 
 computers for most of my life!


If I was born in the 1800's, I'd be happily immersed in designing 
locomotives. In the 1930's, I'd be building high performance aircraft. 
Every decade has its cool nerdly stuff to work on!
Apr 13 2007
next sibling parent reply "David B. Held" <dheld codelogicconsulting.com> writes:
Walter Bright wrote:

 [...]
 When I was in high school, nerd was a perjorative term. But I've grown 
 out of all that nonsense, and am happy to be what I am.
 [...]


When you were in high school, "nerds" weren't driving around in BMWs 
with millions in stock options...
Dave
Apr 13 2007
parent Walter Bright <newshound1 digitalmars.com> writes:
David B. Held wrote:

 Walter Bright wrote:

 [...]
 When I was in high school, nerd was a perjorative term. But I've grown 
 out of all that nonsense, and am happy to be what I am.
 [...]


 
 When you were in high school, "nerds" weren't driving around in BMWs 
 with millions in stock options...



True, the high school loser nerd who becomes a billionaire stereotype 
didn't exist yet. But then again, my grandfather did all right.
Apr 14 2007
prev sibling parent 0ffh <spam frankhirsch.net> writes:
Walter Bright wrote:

 Georg Wrede wrote:
 I still have my original IBM PC. With an 8080 chip!


Sniff... never had one of those (yet my dad does), but I
had an XT-Clone with a NEC V30 CPU going an amazing 8MHz
- with "Turbo" key depressed naturally, as it used to
break all hardcoded delay-loops... still before that I
had a C64 (bought 1986). As did many others, I suppose.

Regards, Frank
Apr 14 2007
prev sibling next sibling parent Dan <murpsoft hotmail.com> writes:
0ffh Wrote:


 Lionello Lunesu wrote:

 Well, that was *exactly* what I was getting at! OK, granted, it's only done 
 in 4KB increments, but that's when a regular array resize (copying the 
 contents over to a new place) would actually start harming performance.


 
 That sounds enticing!
 If you could put garbage collection into the kernel - even with tile-sized
 granularity - that would make many things easier. And finally put the idle
 thread to good use... :-)))


It would be somewhat practical to, before starting to use swap, call the GC to
collect garbage and compress what's used in memory, freeing up pages by the
block...

(squeeze the data from 48% of 6 blocks into 3 blocks, freeing up 3?)

Starting to swap (from hard disk) is extremely expensive, and would warrant the
overhead from compression, as it would most likely perform faster.

This has been something operating systems of the present cannot do - they can't
tell a program to clean up it's shit, it's getting crowded.
Apr 22 2007
prev sibling parent reply Dan <murpsoft hotmail.com> writes:
Georg Wrede Wrote:


 Alexander Panek wrote:

 Walter Bright wrote:
 

 Alexander Panek wrote:


 I'm sure going to spend some time getting a proper codebase for further
 development done, and write some documentation about how to actually
 get to the point of being able to write an operating system with D. I
 think that's a major weak point of D, as it claims to be a systems
 language, but there's no actual system written in it from scratch,
 neither documentation on how to achieve that.



 Andrei suggested that the source code for Minix, which is fairly 
 small, could be transliterated from C almost directly into D. This 
 would neatly resolve the issue, and provide a starting point for 
 anyone wanting to take it further.


 
 
 True. On the other hand, a kernel written purely in D is something that 
 really would (and will, arr) be cool! I just find it less exciting to 
 code "C with D", so to speak.
 
 I'll make sure to announce the project, once a few milestones are 
 finished, on the newsgroups.


 
 Linus T. started the same way. He had a Minix running on his machine, 
 and piece by piece replaced stuff there, until no original code was 
 left. (And then he /really/ started writing Linux.)
 
 We should do the same thing. Figuratively speaking. So, do the thing in 
 Cish D, and only when it gets bootable, start rewriting it in Real D (tm).
 


Umm... you're an Object Oriented programmer, aren't you?

I personally love D's structs, functions and asm{}, it's much much cleaner than
C and equally as powerful.
Apr 25 2007
parent Alexander Panek <alexander.panek brainsware.org> writes:
On Wed, 25 Apr 2007 22:17:23 -0400
Dan <murpsoft hotmail.com> wrote:

 [...]
 Umm... you're an Object Oriented programmer, aren't you?


If this is addressed to me (?)..

Yes, pretty much. But I don't actually hesitate to use other
techniques, either, such as structs in D (I *do* think structs in D are
reeeaaaallly great. They just lack some things like proper ctor syntax,
and parameter passing by reference.).


 I personally love D's structs, functions and asm{}, it's much much
 cleaner than C and equally as powerful


Aye. Still, for some things in osdev even classes are useful!
Apr 26 2007