www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - GitHub behind proxy servers among other questions

reply "JohnnyK" <johnnykinsey comcast.net> writes:
Hi all,
   I have searched everywhere over the Internet and I have yet to 
find a way to clone a project using git when my workstation is 
behind a company proxy.  Can you guys clone your projects to a 
single zip file that I can download?  This would be easier 
instead of working with some strange command-line tool that does 
not recognize modern networks.  Honestly I just need the DWT 
binary with the help files so I can use the api.  I have spent 
weeks searching for a way to download DWT to my windows 
workstation at work and have yet figured out how to make GIT work.
Jul 07 2013
next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
   I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my workstation 
 is behind a company proxy.  Can you guys clone your projects to 
 a single zip file that I can download?  This would be easier 
 instead of working with some strange command-line tool that 
 does not recognize modern networks.  Honestly I just need the 
 DWT binary with the help files so I can use the api.  I have 
 spent weeks searching for a way to download DWT to my windows 
 workstation at work and have yet figured out how to make GIT 
 work.


Git provides a download by zip; Right side, bottom.

This is what I found on getting Git to work with a proxy:

http://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy
Jul 07 2013
next sibling parent reply "JohnnyK" <johnnykinsey comcast.net> writes:
On Monday, 8 July 2013 at 03:35:03 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
  I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my workstation 
 is behind a company proxy.  Can you guys clone your projects 
 to a single zip file that I can download?  This would be 
 easier instead of working with some strange command-line tool 
 that does not recognize modern networks.  Honestly I just need 
 the DWT binary with the help files so I can use the api.  I 
 have spent weeks searching for a way to download DWT to my 
 windows workstation at work and have yet figured out how to 
 make GIT work.


 Git provides a download by zip; Right side, bottom.

 This is what I found on getting Git to work with a proxy:

 http://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy


I appreciate your response.  I have tried these.  I think the 
real issue is that I am not sure on the IP and port needed for 
the proxy here at work.  The company uses WPAD in the browser and 
I cannot figure out what the IP and port the browser is using to 
connect through the proxy.  If I knew that I probably could make 
it work.  It would be nice if GitHUB would change their Downlaod 
Zip button such that it does a recursive zip to include all the 
subfolders.
Jul 08 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 8 July 2013 at 16:24:53 UTC, JohnnyK wrote:

 On Monday, 8 July 2013 at 03:35:03 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
 I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my 
 workstation is behind a company proxy.  Can you guys clone 
 your projects to a single zip file that I can download?  This 
 would be easier instead of working with some strange 
 command-line tool that does not recognize modern networks.  
 Honestly I just need the DWT binary with the help files so I 
 can use the api.  I have spent weeks searching for a way to 
 download DWT to my windows workstation at work and have yet 
 figured out how to make GIT work.


 Git provides a download by zip; Right side, bottom.

 This is what I found on getting Git to work with a proxy:

 http://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy


 I appreciate your response.  I have tried these.  I think the 
 real issue is that I am not sure on the IP and port needed for 
 the proxy here at work.  The company uses WPAD in the browser 
 and I cannot figure out what the IP and port the browser is 
 using to connect through the proxy.  If I knew that I probably 
 could make it work.  It would be nice if GitHUB would change 
 their Downlaod Zip button such that it does a recursive zip to 
 include all the subfolders.


I would just ssh to somewhere outside the firewall, git clone 
there, tar the folder, then scp it back. Assuming that is 
possible through your firewall.

If you're on windows then there are the always useful putty and 
winscp to do your ssh and scp work respectively.
Jul 08 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 8 July 2013 at 16:49:05 UTC, John Colvin wrote:

 On Monday, 8 July 2013 at 16:24:53 UTC, JohnnyK wrote:

 On Monday, 8 July 2013 at 03:35:03 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
 I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my 
 workstation is behind a company proxy.  Can you guys clone 
 your projects to a single zip file that I can download?  
 This would be easier instead of working with some strange 
 command-line tool that does not recognize modern networks.  
 Honestly I just need the DWT binary with the help files so I 
 can use the api.  I have spent weeks searching for a way to 
 download DWT to my windows workstation at work and have yet 
 figured out how to make GIT work.


 Git provides a download by zip; Right side, bottom.

 This is what I found on getting Git to work with a proxy:

 http://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy


 I appreciate your response.  I have tried these.  I think the 
 real issue is that I am not sure on the IP and port needed for 
 the proxy here at work.  The company uses WPAD in the browser 
 and I cannot figure out what the IP and port the browser is 
 using to connect through the proxy.  If I knew that I probably 
 could make it work.  It would be nice if GitHUB would change 
 their Downlaod Zip button such that it does a recursive zip to 
 include all the subfolders.


 I would just ssh to somewhere outside the firewall, git clone 
 there, tar the folder, then scp it back. Assuming that is 
 possible through your firewall.

 If you're on windows then there are the always useful putty and 
 winscp to do your ssh and scp work respectively.


Well, the thing to remember is that he did say "company", so such 
approaches to bypass a firewall could very well be a violation of 
his IT policies. I know doing this would be *major* violation of 
my companie's policies, and could be cause for termination.
Jul 08 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 8 July 2013 at 18:09:44 UTC, monarch_dodra wrote:

 On Monday, 8 July 2013 at 16:49:05 UTC, John Colvin wrote:

 On Monday, 8 July 2013 at 16:24:53 UTC, JohnnyK wrote:

 On Monday, 8 July 2013 at 03:35:03 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
 I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my 
 workstation is behind a company proxy.  Can you guys clone 
 your projects to a single zip file that I can download?  
 This would be easier instead of working with some strange 
 command-line tool that does not recognize modern networks.  
 Honestly I just need the DWT binary with the help files so 
 I can use the api.  I have spent weeks searching for a way 
 to download DWT to my windows workstation at work and have 
 yet figured out how to make GIT work.


 Git provides a download by zip; Right side, bottom.

 This is what I found on getting Git to work with a proxy:

 http://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy


 I appreciate your response.  I have tried these.  I think the 
 real issue is that I am not sure on the IP and port needed 
 for the proxy here at work.  The company uses WPAD in the 
 browser and I cannot figure out what the IP and port the 
 browser is using to connect through the proxy.  If I knew 
 that I probably could make it work.  It would be nice if 
 GitHUB would change their Downlaod Zip button such that it 
 does a recursive zip to include all the subfolders.


 I would just ssh to somewhere outside the firewall, git clone 
 there, tar the folder, then scp it back. Assuming that is 
 possible through your firewall.

 If you're on windows then there are the always useful putty 
 and winscp to do your ssh and scp work respectively.


 Well, the thing to remember is that he did say "company", so 
 such approaches to bypass a firewall could very well be a 
 violation of his IT policies. I know doing this would be 
 *major* violation of my companie's policies, and could be cause 
 for termination.


I don't see how it is problematic? The firewall doesn't allow a 
particular type of connection, so you cache the result somewhere 
else and then access it via something that is allowed.

It's the same as downloading the zip file from github, only 
you're doing the "zipping" yourself, then using scp to do the 
download instead of http.

Is it common to have ssh connections banned in the IT policy but 
*not* blocked by the firewall?
Jul 08 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 8 July 2013 at 19:20:46 UTC, John Colvin wrote:

 I don't see how it is problematic? The firewall doesn't allow a 
 particular type of connection, so you cache the result 
 somewhere else and then access it via something that is allowed.

 It's the same as downloading the zip file from github, only 
 you're doing the "zipping" yourself, then using scp to do the 
 download instead of http.

 Is it common to have ssh connections banned in the IT policy 
 but *not* blocked by the firewall?


I'm just saying that maybe the firwall is blocking that protocol 
on purpose? Github can very easily be seen as "File Sharing" 
and/or "Online Storage". The sole act of downloading/uploading 
anything on there is possible violations of policy. You could 
download viruses, or be stealing your company's code to publish 
it online. Even if you *can* download the zip via http, it 
doesn't mean you are actually allowed to. If your company's 
firewall doesn't block a porn site, does that means its OK to 
surf it?

Now, most companies wouldn't really care that much about it, if 
you connect every now and then, no one will care. Worst case 
scenario, you'll get a reminder not to download/upload crap on 
internet.

But, if you start doing all this via ssh, or other alternative 
routes, then you are *activelly* bypassing protections, and in 
particular, doing something you *know* you shouldn't be doing. 
That's a more serious offense.

Now, do what you want. I'm just saying before going it and 
deploying such tools to download the files on git, please make 
sure you aren't violating your company's policies.
Jul 09 2013
parent "JohnnyK" <johnnykinsey comcast.net> writes:
On Tuesday, 9 July 2013 at 08:55:29 UTC, monarch_dodra wrote:

 On Monday, 8 July 2013 at 19:20:46 UTC, John Colvin wrote:

 I don't see how it is problematic? The firewall doesn't allow 
 a particular type of connection, so you cache the result 
 somewhere else and then access it via something that is 
 allowed.

 It's the same as downloading the zip file from github, only 
 you're doing the "zipping" yourself, then using scp to do the 
 download instead of http.

 Is it common to have ssh connections banned in the IT policy 
 but *not* blocked by the firewall?


 I'm just saying that maybe the firwall is blocking that 
 protocol on purpose? Github can very easily be seen as "File 
 Sharing" and/or "Online Storage". The sole act of 
 downloading/uploading anything on there is possible violations 
 of policy. You could download viruses, or be stealing your 
 company's code to publish it online. Even if you *can* download 
 the zip via http, it doesn't mean you are actually allowed to. 
 If your company's firewall doesn't block a porn site, does that 
 means its OK to surf it?

 Now, most companies wouldn't really care that much about it, if 
 you connect every now and then, no one will care. Worst case 
 scenario, you'll get a reminder not to download/upload crap on 
 internet.

 But, if you start doing all this via ssh, or other alternative 
 routes, then you are *activelly* bypassing protections, and in 
 particular, doing something you *know* you shouldn't be doing. 
 That's a more serious offense.

 Now, do what you want. I'm just saying before going it and 
 deploying such tools to download the files on git, please make 
 sure you aren't violating your company's policies.


   I do not encourage anyone to bypass network security 
protections and I do encourage letting your conscious be your 
guide on this topic.

   The issue is that it is just more complicating than I think it 
should be to download libraries and such.  Authors of helper 
libraries and other D related code should take their queue from 
dlang.org and actually have binaries that users can download and 
just use without the need to compile before they can use the 
wares.  It is not a firewall issue or anything like that.  It was 
just GIT not supporting modern proxy authorization schemes.  
Problem resolved but it could have been avoided all together for 
those of us that are not to the point of contributing to these 
projects and are just trying to get their feet wet without 
drowning first.
On another note I was able to successfully compile the DWT lib 
and I was able to compile the snippet example programs to test 
DWT.  I did notice for those that are interested DWT exe's are 
over 2MB which is really what I was wanted to find out.  I will 
redirect DWT questions to the appropriate forum.

   The workaround for GIT via proxy is just not as pretty as I 
would have liked it.  However as I posted previously I found a 
suitable workaround for the issue.  No other discussion is needed 
for this issue.
Jul 09 2013
prev sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 8 July 2013 at 16:24:53 UTC, JohnnyK wrote:


 It would be nice if GitHUB would change their Downlaod Zip 
 button such that it does a recursive zip to include all the 
 subfolders.


Ehhh, well that is useless. Sorry I couldn't hook you up with a 
solution.
Jul 08 2013
parent reply "JohnnyK" <johnnykinsey comcast.net> writes:
On Monday, 8 July 2013 at 18:34:33 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 16:24:53 UTC, JohnnyK wrote:


 It would be nice if GitHUB would change their Downlaod Zip 
 button such that it does a recursive zip to include all the 
 subfolders.


 Ehhh, well that is useless. Sorry I couldn't hook you up with a 
 solution.


Thanks for all the input.  I found a tool called NTLMAPS at 
http://ntlmaps.sourceforge.net/ which allows one to create a pass 
through proxy which understands NTLM proxy servers and are able 
to connect and forward HTTP requests and return responses so long 
story short I can now git clone as I need to NTLMAPS config file 
is pretty easy to configure with your proxy authentication 
information.  However I did have to change the following 
named/value pairs.

from this

LM_PART:1
NT_PART:0





NTLM_FLAGS: 06820000

to this

LM_PART:1
NT_PART:1





NTLM_FLAGS: 07820000

The proxy that I am behind was not happy until I did that.  
Anyway it is working now and I hope this post serves to help 
others with the same issue.

also note that I had to use http:// in place of git:// in the git 
paths and in the case of DWT I had to go to every major folder 
and git clone them individually because for some reason git 
wanted to use git:// for subsequent requests.  It is strange that 
it did not do that for the Win-Res folder just for the others.  
So I had to do commands like the following

C:\GITClones>cd dwt

C:\GITClones\dwt>git clone --recursive 
http://github.com/d-widget-toolkit/base.g
it
Cloning into 'base'...
remote: Counting objects: 806, done.
remote: Compressing objects: 100% (326/326), done.
remote: Total 806 (delta 402), reused 803 (delta 401)
Receiving objects: 100% (806/806), 310.62 KiB | 261.00 KiB/s, 
done.
Resolving deltas: 100% (402/402), done.

C:\GITClones\dwt>git clone --recursive 
http://github.com/d-widget-toolkit/org.ec
lipse.swt.gtk.linux.x86.git
Cloning into 'org.eclipse.swt.gtk.linux.x86'...
remote: Counting objects: 1691, done.
remote: Compressing objects: 100% (634/634), done.
remote: Total 1691 (delta 951), reused 1659 (delta 944)
Receiving objects: 100% (1691/1691), 1.71 MiB | 361.00 KiB/s, 
done.
Resolving deltas: 100% (951/951), done.

C:\GITClones\dwt>git clone --recursive 
http://github.com/d-widget-toolkit/org.ec
lipse.swt.win32.win32.x86.git
Cloning into 'org.eclipse.swt.win32.win32.x86'...


The only other thing I wish git could do is compile it for me so 
that I wouldn't have too but until we have HDgit I guess I am 
stuck with all this command line busy work.  Thanks again for 
every ones time.
Jul 08 2013
parent reply "JohnnyK" <johnnykinsey comcast.net> writes:
On Monday, 8 July 2013 at 19:37:08 UTC, JohnnyK wrote:
Another thing about GitHub's Download Zip button and this process 
as a whole.  While the Download Zip button does allow you to 
download the master folders with recursive directories I do find 
it somewhat cumbersome or should I say awkward since you have to 
extract these in their correct locations upon download.  This is 
not good when you are trying to follow someones instructions on 
how to download, build, and install their wares.  I just don't 
get git?  Just compile the code and put a binary up there for 
people to use.  If I want to contribute to the code base then I 
will I guess go through all this hassle but for those of us not 
smart enough to contribute and just want to use it show me the 
binary please.  In the amount of time it is taking to get this 
stuff I could have made major strides at building my own.  Also I 
don't know what this fragmented downloading is going to do in the 
future when I want an updated version of the software.  Please 
remember many of us are simpletons and don't use all these fancy 
version control systems and just need the binary so that we can 
use it.  I guess you could say we are your customers and not your 
contributors.  Please everyone in the D community try and follow 
the KISS methodology and if you do I bet you could get many more 
users of your wares and not people banging their heads until they 
give up.  Yeah I was only 2 clicks away from giving up on this 
whole D thing thinking it was just too darned hard to make it 
worth my while.  Who knows I still may give up since I have not 
attempted to compile it yet.  I think I will wait until tomorrow 
before I try to tackle that part of the destruction's.  For now I 
think I have it downloaded.  BTW did I mention that I really hate 
git and compiling other people's code just to use an edit box and 
button on a dialog???
Jul 08 2013
parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 8 July 2013 at 19:57:52 UTC, JohnnyK wrote:

 On Monday, 8 July 2013 at 19:37:08 UTC, JohnnyK wrote:
 Another thing about GitHub's Download Zip button and this 
 process as a whole.  While the Download Zip button does allow 
 you to download the master folders with recursive directories I 
 do find it somewhat cumbersome or should I say awkward since 
 you have to extract these in their correct locations upon 
 download.  This is not good when you are trying to follow 
 someones instructions on how to download, build, and install 
 their wares.  I just don't get git?  Just compile the code and 
 put a binary up there for people to use.  If I want to 
 contribute to the code base then I will I guess go through all 
 this hassle but for those of us not smart enough to contribute 
 and just want to use it show me the binary please.  In the 
 amount of time it is taking to get this stuff I could have made 
 major strides at building my own.  Also I don't know what this 
 fragmented downloading is going to do in the future when I want 
 an updated version of the software.  Please remember many of us 
 are simpletons and don't use all these fancy version control 
 systems and just need the binary so that we can use it.  I 
 guess you could say we are your customers and not your 
 contributors.  Please everyone in the D community try and 
 follow the KISS methodology and if you do I bet you could get 
 many more users of your wares and not people banging their 
 heads until they give up.  Yeah I was only 2 clicks away from 
 giving up on this whole D thing thinking it was just too darned 
 hard to make it worth my while.  Who knows I still may give up 
 since I have not attempted to compile it yet.  I think I will 
 wait until tomorrow before I try to tackle that part of the 
 destruction's.  For now I think I have it downloaded.  BTW did 
 I mention that I really hate git and compiling other people's 
 code just to use an edit box and button on a dialog???


Ar you talking about D specifically? You *can* just download the 
package you know:
http://dlang.org/download.html
I'm actually wondering how you even reached git?
Jul 09 2013
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 8 July 2013 at 03:35:03 UTC, Jesse Phillips wrote:

 On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
  I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my workstation 
 is behind a company proxy.  Can you guys clone your projects 
 to a single zip file that I can download?  This would be 
 easier instead of working with some strange command-line tool 
 that does not recognize modern networks.  Honestly I just need 
 the DWT binary with the help files so I can use the api.  I 
 have spent weeks searching for a way to download DWT to my 
 windows workstation at work and have yet figured out how to 
 make GIT work.


 Git provides a download by zip; Right side, bottom.


Holly crap! Is this new? I'm in need of this.
Jul 08 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 July 2013 at 02:42:50 UTC, JohnnyK wrote:

 Hi all,
   I have searched everywhere over the Internet and I have yet 
 to find a way to clone a project using git when my workstation 
 is behind a company proxy.  Can you guys clone your projects to 
 a single zip file that I can download?  This would be easier 
 instead of working with some strange command-line tool that 
 does not recognize modern networks.  Honestly I just need the 
 DWT binary with the help files so I can use the api.  I have 
 spent weeks searching for a way to download DWT to my windows 
 workstation at work and have yet figured out how to make GIT 
 work.


git can clone via HTTP(S):
git clone http://github.com/D-Programming-Language/dmd.git
Jul 08 2013