www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.dwt - Would GIT submodules be a good idea?

reply Jacob Carlborg <doob me.com> writes:
Since it seems that everyone is ok with git I will move the DWT2 
repository to github. That leads to the next question: Would git 
submodules be a good idea? I haven't use submodules myself my it sounds 
good in theory. I'm thinking about having sub repository for basically 
every top level directory in the current repository. One repsository for 
jface, one for dwt-win, one for dwt-linux and so on. What do you think?


-- 
/Jacob Carlborg
Aug 15 2011
next sibling parent torhu <no spam.invalid> writes:
On 15.08.2011 19:26, Jacob Carlborg wrote:

 Since it seems that everyone is ok with git I will move the DWT2
 repository to github. That leads to the next question: Would git
 submodules be a good idea? I haven't use submodules myself my it sounds
 good in theory. I'm thinking about having sub repository for basically
 every top level directory in the current repository. One repsository for
 jface, one for dwt-win, one for dwt-linux and so on. What do you think?


If it simplifies things instead of complicating them, sure.  I have no 
experience with neither submodules nor the Mercurial subrepos.  I hope 
that hg-git supports them, but wasn't able to find a repository for testing.
Aug 15 2011
prev sibling parent reply Jesse Phillips <jessekphillips+d gmail.com> writes:
On Mon, 15 Aug 2011 19:26:57 +0200, Jacob Carlborg wrote:


 Since it seems that everyone is ok with git I will move the DWT2
 repository to github. That leads to the next question: Would git
 submodules be a good idea? I haven't use submodules myself my it sounds
 good in theory. I'm thinking about having sub repository for basically
 every top level directory in the current repository. One repsository for
 jface, one for dwt-win, one for dwt-linux and so on. What do you think?


My experience with submodules has been great. There is a little bit of 
overhead which I'll explain below, but cloning just becomes

$ git clone url
$ git submodule init
$ git submodule update

(I of course don't know hg-git's support for this)

I'd recommend separate repositories, I guess it isn't feasible to support 
all the OS specific stuff under one source base? I'd think that would be 
easier than supporting D1/D2/Phobos/Tango in one source tree.

But do you really need submodules? If you want dwt you'll either pull in 
dtw-win dwt-linux as a submodule of your own or just dwt-linux and jface.

Personally I think the OS's should be brought together and the library/
language separation should be their own repos (jface its own repo). In 
fact if that sounds reasonable I'll volunteer to try and make it happen 
(as you've done dwt-mac you might know this is a bad idea)?

Back to submodules. These create clones of a completely different repo in 
a subdirectory of the project. The commit hash is stored in the parent, 
not the files. You can commit/branch/push/pull/merge in your submodules 
just like any other local repository. As long as your working directory 
is within the submodule directory.

The .gitsubmodules config file is tracked in the repository just like any 
other file. This file stores the information about the repository 
locations and directories to use. The first time setup can be done from 
the command line but say for example someone forks dwt-linux and jface.

Assuming we are using a Hub project (or whatever the formal name would 
be). There is DWT which has submodules of jface, dwt-linux... If the user 
wants to have their own for of DWT that points to their forks of jface... 
they would need edit the .gitsubmodules so that the URL locations point 
to their git repositories.

$ git submodule init just takes the information found in the 
configuration file and moves them into .git/config So once 
the .gitsubmodules is updated you'll have easy checkout.

I don't think a Hub project is needed.
Aug 15 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-16 03:37, Jesse Phillips wrote:

 On Mon, 15 Aug 2011 19:26:57 +0200, Jacob Carlborg wrote:


 Since it seems that everyone is ok with git I will move the DWT2
 repository to github. That leads to the next question: Would git
 submodules be a good idea? I haven't use submodules myself my it sounds
 good in theory. I'm thinking about having sub repository for basically
 every top level directory in the current repository. One repsository for
 jface, one for dwt-win, one for dwt-linux and so on. What do you think?


 My experience with submodules has been great. There is a little bit of
 overhead which I'll explain below, but cloning just becomes

 $ git clone url
 $ git submodule init
 $ git submodule update

 (I of course don't know hg-git's support for this)

 I'd recommend separate repositories, I guess it isn't feasible to support
 all the OS specific stuff under one source base? I'd think that would be
 easier than supporting D1/D2/Phobos/Tango in one source tree.


I'm not exactly sure what you mean with "one source base", because 
currently all OS specific stuff are in one repository.


 But do you really need submodules? If you want dwt you'll either pull in
 dtw-win dwt-linux as a submodule of your own or just dwt-linux and jface.


The problem is that you also always need the "base" repository. Which 
contains implementations of the some Java classes and other utility 
functions used by dwt.


 Personally I think the OS's should be brought together and the library/
 language separation should be their own repos (jface its own repo). In
 fact if that sounds reasonable I'll volunteer to try and make it happen
 (as you've done dwt-mac you might know this is a bad idea)?


Currently everything is in one repository but it's in separate top level 
directories. Having all the OS's merge in the same directory would be 
very very bad. First, it's just too much code for that. Second it's not 
how SWT is arranged and it would be very difficult to port future 
versions of SWT.

That was why I thought submodules could take the best of both 
approaches, that is, one repository for everything (like we have now) 
versus individual repositories for each project.


 Back to submodules. These create clones of a completely different repo in
 a subdirectory of the project. The commit hash is stored in the parent,
 not the files. You can commit/branch/push/pull/merge in your submodules
 just like any other local repository. As long as your working directory
 is within the submodule directory.

 The .gitsubmodules config file is tracked in the repository just like any
 other file. This file stores the information about the repository
 locations and directories to use. The first time setup can be done from
 the command line but say for example someone forks dwt-linux and jface.

 Assuming we are using a Hub project (or whatever the formal name would
 be). There is DWT which has submodules of jface, dwt-linux... If the user
 wants to have their own for of DWT that points to their forks of jface...
 they would need edit the .gitsubmodules so that the URL locations point
 to their git repositories.

 $ git submodule init just takes the information found in the
 configuration file and moves them into .git/config So once
 the .gitsubmodules is updated you'll have easy checkout.

 I don't think a Hub project is needed.


I think I need to read more about submodules. What I was hoping for was 
that you could have individual repositories for each project and then a 
super project. When the super project could contain the build script and 
when it's cloned the sub respiratory will be cloned as well.

-- 
/Jacob Carlborg
Aug 15 2011
parent reply Jesse Phillips <jessekphillips+d gmail.com> writes:
On Tue, 16 Aug 2011 08:54:16 +0200, Jacob Carlborg wrote:


 I'm not exactly sure what you mean with "one source base", because
 currently all OS specific stuff are in one repository.
 

 But do you really need submodules? If you want dwt you'll either pull
 in dtw-win dwt-linux as a submodule of your own or just dwt-linux and
 jface.


 
 The problem is that you also always need the "base" repository. Which
 contains implementations of the some Java classes and other utility
 functions used by dwt.


Oh, that makes sense then yeah submodules is not a bad approach.
 

 Personally I think the OS's should be brought together and the library/
 language separation should be their own repos (jface its own repo). In
 fact if that sounds reasonable I'll volunteer to try and make it happen
 (as you've done dwt-mac you might know this is a bad idea)?


 
 Currently everything is in one repository but it's in separate top level
 directories. Having all the OS's merge in the same directory would be
 very very bad. First, it's just too much code for that. Second it's not
 how SWT is arranged and it would be very difficult to port future
 versions of SWT.
 
 That was why I thought submodules could take the best of both
 approaches, that is, one repository for everything (like we have now)
 versus individual repositories for each project.
 
 I think I need to read more about submodules. What I was hoping for was
 that you could have individual repositories for each project and then a
 super project. When the super project could contain the build script and
 when it's cloned the sub respiratory will be cloned as well.


It is almost like that. It's just that the cloning also requires the 
initialization and cloning of the submodules. You'll have to keep them in 
sync and I've never submoduled a project that had submodules it self.

I'm not user if there is much point in separating them if you can use dwt-
linux or win in something else. But jface could be its own repository as 
that isn't required to used DWT at all. And making that a submodule is 
fine.
Aug 16 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-16 16:30, Jesse Phillips wrote:

 It is almost like that. It's just that the cloning also requires the
 initialization and cloning of the submodules. You'll have to keep them in
 sync and I've never submoduled a project that had submodules it self.


Yeah, I've read that now.


 I'm not user if there is much point in separating them if you can use dwt-
 linux or win in something else. But jface could be its own repository as
 that isn't required to used DWT at all. And making that a submodule is
 fine.


Not all top level directories need to be submodules, that was just a 
thought. I could include everything to make it possible to build DWT in 
one repository (the super repository) and have everything else in 
submodules.

-- 
/Jacob Carlborg
Aug 16 2011
next sibling parent reply Jesse Phillips <jessekphillips+d gmail.com> writes:
On Tue, 16 Aug 2011 18:46:28 +0200, Jacob Carlborg wrote:


 Not all top level directories need to be submodules, that was just a
 thought. I could include everything to make it possible to build DWT in
 one repository (the super repository) and have everything else in
 submodules.


Looking a little closer at the stuff, I've remembered part of DWT is a 
port of Java, and looks like part of Eclipse.

My thought would be that those are separate porting efforts, and you'd 
have dwt (which includes the the Windows and Linux code together, ie the 
swt source) which has the Java/Eclipse ports as submodules.

Though it seems that DWT isn't laid out the same as SWT 3.8's repo:
http://git.eclipse.org/c/platform/eclipse.platform.swt.git/tree/

Anyway, I hope I'm contributing to you making a decision rather than just 
adding confusion.
Aug 16 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-17 03:36, Jesse Phillips wrote:

 On Tue, 16 Aug 2011 18:46:28 +0200, Jacob Carlborg wrote:


 Not all top level directories need to be submodules, that was just a
 thought. I could include everything to make it possible to build DWT in
 one repository (the super repository) and have everything else in
 submodules.


 Looking a little closer at the stuff, I've remembered part of DWT is a
 port of Java, and looks like part of Eclipse.


DWT is a port of SWT which is Java and part of Eclipse. To ease the 
porting some parts of Java is ported as well.


 My thought would be that those are separate porting efforts, and you'd
 have dwt (which includes the the Windows and Linux code together, ie the
 swt source) which has the Java/Eclipse ports as submodules.


Ok.


 Though it seems that DWT isn't laid out the same as SWT 3.8's repo:
 http://git.eclipse.org/c/platform/eclipse.platform.swt.git/tree/


I have no idea of SWT 3.8 is laid out. DWT is quite far behind and is a 
port of SWT 3.4.


 Anyway, I hope I'm contributing to you making a decision rather than just
 adding confusion.


No no, you're helping :)

-- 
/Jacob Carlborg
Aug 16 2011
prev sibling parent reply torhu <no spam.invalid> writes:
On 16.08.2011 18:46, Jacob Carlborg wrote:

 On 2011-08-16 16:30, Jesse Phillips wrote:

  It is almost like that. It's just that the cloning also requires the
  initialization and cloning of the submodules. You'll have to keep them in
  sync and I've never submoduled a project that had submodules it self.


 Yeah, I've read that now.


  I'm not user if there is much point in separating them if you can use dwt-
  linux or win in something else. But jface could be its own repository as
  that isn't required to used DWT at all. And making that a submodule is
  fine.


 Not all top level directories need to be submodules, that was just a
 thought. I could include everything to make it possible to build DWT in
 one repository (the super repository) and have everything else in
 submodules.


What problem is splitting things into submodules supposed to solve?  The 
current Mercurial repository with everything in it is just 50 MB.  I 
suppose Git repositories take up more space, but assume that's not why 
you want to split things up?

If submodules are meant for the same use cases as SVN externals, they 
are primarily for automatically pulling in stuff that is separate 
because it is actually a separate project, probably maintained by 
someone else, or maybe just hosted elsewhere for some reason.  Not for 
splitting a project into parts.  Which is why I'm wondering what exactly 
you want to achieve here.
Aug 17 2011
next sibling parent Jesse Phillips <jessekphillips+d gmail.com> writes:
On Wed, 17 Aug 2011 11:46:23 +0200, torhu wrote:


 If submodules are meant for the same use cases as SVN externals, they
 are primarily for automatically pulling in stuff that is separate
 because it is actually a separate project, probably maintained by
 someone else, or maybe just hosted elsewhere for some reason.  Not for
 splitting a project into parts.  Which is why I'm wondering what exactly
 you want to achieve here.


With SVN you will create large projects/multi-projects in one repository, 
this can include multiple tools, libraries, and everything else. SVN also 
provides the ability to checkout subdirectories without pulling 
everything else.

My suggest for submoduling is to separate parts that could be reused by 
something not DWT. For example the base directory that contains base Java 
components. The parts of Eclipse that aren't part of SWT (as it is 
handled by the SWT repository). These can be used by others that are 
porting something from Java/Eclipse that isn't DWT. (not likely but still 
different).
Aug 17 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-17 11:46, torhu wrote:

 On 16.08.2011 18:46, Jacob Carlborg wrote:

 On 2011-08-16 16:30, Jesse Phillips wrote:

 It is almost like that. It's just that the cloning also requires the
 initialization and cloning of the submodules. You'll have to keep
 them in
 sync and I've never submoduled a project that had submodules it self.


 Yeah, I've read that now.


 I'm not user if there is much point in separating them if you can use
 dwt-
 linux or win in something else. But jface could be its own repository as
 that isn't required to used DWT at all. And making that a submodule is
 fine.


 Not all top level directories need to be submodules, that was just a
 thought. I could include everything to make it possible to build DWT in
 one repository (the super repository) and have everything else in
 submodules.


 What problem is splitting things into submodules supposed to solve? The
 current Mercurial repository with everything in it is just 50 MB. I
 suppose Git repositories take up more space, but assume that's not why
 you want to split things up?

 If submodules are meant for the same use cases as SVN externals, they
 are primarily for automatically pulling in stuff that is separate
 because it is actually a separate project, probably maintained by
 someone else, or maybe just hosted elsewhere for some reason. Not for
 splitting a project into parts. Which is why I'm wondering what exactly
 you want to achieve here.


In addition to what Jesse Phillips replied to your post I think that 
these projects actually should be separate repositories. Another reason 
is that currently I have only interest in DWT and JFace. I see no reason 
to have the other projects in the same repository.

-- 
/Jacob Carlborg
Aug 17 2011
parent reply torhu <no spam.invalid> writes:
On 17.08.2011 18:42, Jacob Carlborg wrote:

 On 2011-08-17 11:46, torhu wrote:

  On 16.08.2011 18:46, Jacob Carlborg wrote:

  On 2011-08-16 16:30, Jesse Phillips wrote:

  It is almost like that. It's just that the cloning also requires the
  initialization and cloning of the submodules. You'll have to keep
  them in
  sync and I've never submoduled a project that had submodules it self.


  Yeah, I've read that now.


  I'm not user if there is much point in separating them if you can use
  dwt-
  linux or win in something else. But jface could be its own repository as
  that isn't required to used DWT at all. And making that a submodule is
  fine.


  Not all top level directories need to be submodules, that was just a
  thought. I could include everything to make it possible to build DWT in
  one repository (the super repository) and have everything else in
  submodules.


  What problem is splitting things into submodules supposed to solve? The
  current Mercurial repository with everything in it is just 50 MB. I
  suppose Git repositories take up more space, but assume that's not why
  you want to split things up?

  If submodules are meant for the same use cases as SVN externals, they
  are primarily for automatically pulling in stuff that is separate
  because it is actually a separate project, probably maintained by
  someone else, or maybe just hosted elsewhere for some reason. Not for
  splitting a project into parts. Which is why I'm wondering what exactly
  you want to achieve here.


 In addition to what Jesse Phillips replied to your post I think that
 these projects actually should be separate repositories. Another reason
 is that currently I have only interest in DWT and JFace. I see no reason
 to have the other projects in the same repository.


Okay, I think I get the picture now.  Sounds like a good plan, then.

Would DWT become a single repository with all platform implementations 
and snippets in it, that has the base Java library as a submodule?  Or 
would it be dwt-win, dwt-linux, dwt-snippets, etc.?  The former sounds 
simpler, but the latter could make it possible to get rid of the extra 
directory levels that DWT2 uses currently.
Aug 17 2011
next sibling parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
torhu Wrote:


 Okay, I think I get the picture now.  Sounds like a good plan, then.
 
 Would DWT become a single repository with all platform implementations 
 and snippets in it, that has the base Java library as a submodule?  Or 
 would it be dwt-win, dwt-linux, dwt-snippets, etc.?  The former sounds 
 simpler, but the latter could make it possible to get rid of the extra 
 directory levels that DWT2 uses currently.


My thought is:
Java runtime
Eclipse runtime
jface
DWT (Submodules for Java, Eclipse, Jface)
snippets (dwt and jface) (submodules for DWT)
Aug 17 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-17 20:18, Jesse Phillips wrote:

 torhu Wrote:


 Okay, I think I get the picture now.  Sounds like a good plan, then.

 Would DWT become a single repository with all platform implementations
 and snippets in it, that has the base Java library as a submodule?  Or
 would it be dwt-win, dwt-linux, dwt-snippets, etc.?  The former sounds
 simpler, but the latter could make it possible to get rid of the extra
 directory levels that DWT2 uses currently.


 My thought is:
 Java runtime
 Eclipse runtime
 jface
 DWT (Submodules for Java, Eclipse, Jface)
 snippets (dwt and jface) (submodules for DWT)


That looks reasonable. But shouldn't it look something like this, 
submodules in parentheses:

Java runtime
Eclipse runtime (java runtime)
DWT (Eclipse runtime)
JFace (DWT)
snippets (jface)

This is how the dependencies look like. Read "->" as depends on

Java runtime ->
Eclipse runtime -> java runtime
DWT -> Eclipse runtime, java runtime
JFace -> DWT, Eclipse runtime, java runtime
snippets -> JFace, DWT, Eclipse runtime, java runtime

All of these dependencies are not direct dependencies.

I mean, if the Eclipse runtime repository doesn't contain a submodule 
for the Java runtime it can't work on its own.

-- 
/Jacob Carlborg
Aug 17 2011
parent Jesse Phillips <jessekphillips+d gmail.com> writes:
On Wed, 17 Aug 2011 21:26:14 +0200, Jacob Carlborg wrote:


 Java runtime
 Eclipse runtime (java runtime)
 DWT (Eclipse runtime)
 JFace (DWT)
 snippets (jface)
 
 This is how the dependencies look like. Read "->" as depends on
 
 Java runtime ->
 Eclipse runtime -> java runtime
 DWT -> Eclipse runtime, java runtime
 JFace -> DWT, Eclipse runtime, java runtime snippets -> JFace, DWT,
 Eclipse runtime, java runtime
 
 All of these dependencies are not direct dependencies.
 
 I mean, if the Eclipse runtime repository doesn't contain a submodule
 for the Java runtime it can't work on its own.


This does look right. Though I'll bring up this thought which also goes 
against my proposal. Not that I'm against the idea.

Suppose you have a project which uses DWT, then later I want to make use 
of JFace. Since JFace comes with DWT you'd rework your build to include 
the new location rather than just add JFace too it. However as you don't 
have to pull in JFaces DWT you can just leave it. But what about the 
other direction, you start by using JFace and decide to phase it out, 
either to use DWT itself or another layer library.

Anyway these are uncommon, already require proper redesign and yadda so I 
like your reasoning.
Aug 17 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-08-17 18:58, torhu wrote:

 Okay, I think I get the picture now. Sounds like a good plan, then.

 Would DWT become a single repository with all platform implementations
 and snippets in it, that has the base Java library as a submodule? Or
 would it be dwt-win, dwt-linux, dwt-snippets, etc.? The former sounds
 simpler, but the latter could make it possible to get rid of the extra
 directory levels that DWT2 uses currently.


I haven't decided this yet and it's this I'm trying to figure out, what 
will be the best. This is also why I'm asking here what people think.

I'm not sure what you mean with "extra directory levels that DWT2 uses 
currently". Do you mean that org.eclipse.swt.gtk.linux.x86 contains:

src
   org
     eclipse
       swt
         ...

This will most likely not change. When all platforms were in their own 
repositories (DWT1) we just had "dwt" as the root directory. The current 
package layout, in DWT2, is what the original SWT sources use and it was 
decided that it was best to follow the original sources as closely as 
possible to ease porting future versions of SWT.

In DWT1 most classes containing "SWT" was renamed to "DWT" but this just 
caused more problems. I guess it would be easier to rename the root 
package to DWT then renaming all classes containing "SWT". But it would 
be an additional step when porting future versions. I do agree that this 
deep package level is annoying.

-- 
/Jacob Carlborg
Aug 17 2011