www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Symmetry Investments and the D Language Foundation are Hiring

reply Mike Parker <aldacron gmail.com> writes:
Looking for a full-time or part-time gig? Not only is Symmetry 
Investments hiring D programmers, they are also generously 
funding two positions for ecosystem work under the D Language 
Foundation. And they've put up a bounty for a new DUB feature. 
Read all about it here:

https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/
Aug 30 2020
next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
 permanent position


This is huge! Thank you Symmetry!

— Bastiaan.
Aug 30 2020
prev sibling next sibling parent reply Arjan <arjan ask.me.to> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


Fantastic! Thanks Symmetry!
Aug 30 2020
parent Tove <tove fransson.se> writes:
On Sunday, 30 August 2020 at 20:12:41 UTC, Arjan wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


 Fantastic! Thanks Symmetry!


Epic, this is a gamechanger!
Aug 31 2020
prev sibling next sibling parent bachmeier <no spam.net> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


Thanks to Symmetry and the Foundation for this.

This is a big deal. I hope there will be announcements posted and 
promoted after the hires are made. From a marketing perspective, 
it is initiatives like this that give others the confidence to 
invest in the language.
Aug 31 2020
prev sibling next sibling parent reply Arun <aruncxy gmail.com> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/

 SHA-1 hashing throughout


Why not SHA-256?
Aug 31 2020
parent reply starcanopy <starcanopy protonmail.com> writes:
On Monday, 31 August 2020 at 19:55:32 UTC, Arun wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/

 SHA-1 hashing throughout


 Why not SHA-256?


I think the majority opinion is that sha1 is fine for 
non-security purposes like monitoring files for changes to 
obviate superfluous work in a build system; for which DUB 
seemingly would use the sha1 checksums. I have (had) a script 
that would backup files, and I'd use sha1 to detect changes 
instead of sha256 because the latter's storage and computational 
requirements were not insignificant compared to the former. This 
is just my perspective, however. I could be talking nonsense.
Aug 31 2020
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 31 August 2020 at 21:24:53 UTC, starcanopy wrote:

 I think the majority opinion is that sha1 is fine for 
 non-security purposes like monitoring files for changes to 
 obviate superfluous work in a build system;


Not a big fan of using hashes, since collisions do arise when you 
scale things up, but there are fast noncryptographic hashes. Some 
are like 10-500x faster than SHA1:

https://github.com/rurban/smhasher
Sep 01 2020
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 1 September 2020 at 11:53:25 UTC, Ola Fosheim Grøstad 
wrote:

 you scale things up, but there are fast noncryptographic 
 hashes. Some are like 10-500x faster than SHA1:


More like 10-50x...
Sep 01 2020
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


As an alternative to use SHA-1 hashing. There's the option to 
have a daemon running the background listing on filesystem events.

BTW, is timestamps vs SHA-1 hashing really the most pressing 
issue with Dub?

--
/Jacob Carlborg
Sep 01 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


 As an alternative to use SHA-1 hashing. There's the option to 
 have a daemon running the background listing on filesystem 
 events.

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?

 --
 /Jacob Carlborg


We think that not recompiling certain modules which have not 
changed will improve our build times.
And the task proposed is actually something that can go in 
without too much struggle.
Whereas deeper issues in dub likely take much longer.
Sep 01 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/1/20 5:38 AM, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg wrote:

 BTW, is timestamps vs SHA-1 hashing really the most pressing issue 
 with Dub?


 
 We think that not recompiling certain modules which have not changed 
 will improve our build times.
 And the task proposed is actually something that can go in without too 
 much struggle.
 Whereas deeper issues in dub likely take much longer.


I have to agree with Jacob -- what common situation is changing the 
timestamps of your files but not the data?

-Steve
Sep 01 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 1 September 2020 at 13:28:07 UTC, Steven 
Schveighoffer wrote:

 On 9/1/20 5:38 AM, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
 wrote:

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?


 
 We think that not recompiling certain modules which have not 
 changed will improve our build times.
 And the task proposed is actually something that can go in 
 without too much struggle.
 Whereas deeper issues in dub likely take much longer.


 I have to agree with Jacob -- what common situation is changing 
 the timestamps of your files but not the data?

 -Steve


git update. on CI.
Sep 01 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 1 September 2020 at 13:30:33 UTC, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 13:28:07 UTC, Steven 
 Schveighoffer wrote:

 On 9/1/20 5:38 AM, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
 wrote:

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?


 
 We think that not recompiling certain modules which have not 
 changed will improve our build times.
 And the task proposed is actually something that can go in 
 without too much struggle.
 Whereas deeper issues in dub likely take much longer.


 I have to agree with Jacob -- what common situation is 
 changing the timestamps of your files but not the data?

 -Steve


 git update. on CI.


at least I think that was the problem.
I have not looked into this personally.
Sep 01 2020
prev sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Tuesday, 1 September 2020 at 13:28:07 UTC, Steven 
Schveighoffer wrote:

 On 9/1/20 5:38 AM, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
 wrote:

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?


 
 We think that not recompiling certain modules which have not 
 changed will improve our build times.
 And the task proposed is actually something that can go in 
 without too much struggle.
 Whereas deeper issues in dub likely take much longer.


 I have to agree with Jacob -- what common situation is changing 
 the timestamps of your files but not the data?


git checkout branch
git checkout -
Sep 01 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/1/20 2:38 PM, Patrick Schluter wrote:

 On Tuesday, 1 September 2020 at 13:28:07 UTC, Steven Schveighoffer wrote:

 On 9/1/20 5:38 AM, Stefan Koch wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg wrote:

 BTW, is timestamps vs SHA-1 hashing really the most pressing issue 
 with Dub?


 We think that not recompiling certain modules which have not changed 
 will improve our build times.
 And the task proposed is actually something that can go in without 
 too much struggle.
 Whereas deeper issues in dub likely take much longer.


 I have to agree with Jacob -- what common situation is changing the 
 timestamps of your files but not the data?


 git checkout branch
 git checkout -


Is that a part of normal development process? Typically when I want 
incremental building, I'm editing a file, then rebuilding.

I mean, you check out a different branch, but you don't want to rebuild 
everything? I would. And with D, where there are so many templates, 
almost everything is going to need rebuilding anyway. This update to dub 
might replace a build-time problem with a build inconsistency problem 
(hopefully linker error, but possibly code generation differences).

-Steve
Sep 02 2020
parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Wednesday, 2 September 2020 at 12:50:35 UTC, Steven 
Schveighoffer wrote:

 On 9/1/20 2:38 PM, Patrick Schluter wrote:

 On Tuesday, 1 September 2020 at 13:28:07 UTC, Steven 
 Schveighoffer wrote:

 On 9/1/20 5:38 AM, Stefan Koch wrote:

 [...]


 I have to agree with Jacob -- what common situation is 
 changing the timestamps of your files but not the data?


 git checkout branch
 git checkout -


 Is that a part of normal development process? Typically when I 
 want incremental building, I'm editing a file, then rebuilding.

 I mean, you check out a different branch, but you don't want to 
 rebuild everything? I would. And with D, where there are so 
 many templates, almost everything is going to need rebuilding 
 anyway. This update to dub might replace a build-time problem 
 with a build inconsistency problem (hopefully linker error, but 
 possibly code generation differences).


Yes, it happens from time to time and on makefile based it 
involves recompilations of unchanged files but granted it could 
be just me.
Sep 02 2020
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


 As an alternative to use SHA-1 hashing. There's the option to 
 have a daemon running the background listing on filesystem 
 events.

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?

 --
 /Jacob Carlborg


Agreed. A server approach would probably scale much better,
if the intent is to speed up the developer's CTR cycle.
But in any case, thanks to Symmetry for doing this! This is huge.
Sep 01 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 1 September 2020 at 12:59:00 UTC, Mathias LANG wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
 wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is 
 Symmetry Investments hiring D programmers, they are also 
 generously funding two positions for ecosystem work under the 
 D Language Foundation. And they've put up a bounty for a new 
 DUB feature. Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


 As an alternative to use SHA-1 hashing. There's the option to 
 have a daemon running the background listing on filesystem 
 events.

 BTW, is timestamps vs SHA-1 hashing really the most pressing 
 issue with Dub?

 --
 /Jacob Carlborg


 Agreed. A server approach would probably scale much better,
 if the intent is to speed up the developer's CTR cycle.
 But in any case, thanks to Symmetry for doing this! This is 
 huge.


We need both a fs-watcher daemon support and "offline" 
incremental build support, based on SHA-256 (note that git is 
moving away from SHA1 to SHA256 [1]). I'd say SHA-256 is cheap 
enough these days [2] that I don't see a reason not to use it 
even for "online" fs-watcher daemon compilation.

[1]: https://git-scm.com/docs/hash-function-transition/
[2]: https://bench.cr.yp.to/impl-hash/sha256.html
Sep 01 2020
parent reply drug <drug2004 bk.ru> writes:
On 9/1/20 7:34 PM, Petar Kirov [ZombineDev] wrote:

 On Tuesday, 1 September 2020 at 12:59:00 UTC, Mathias LANG wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously funding 
 two positions for ecosystem work under the D Language Foundation. 
 And they've put up a bounty for a new DUB feature. Read all about it 
 here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-fo
ndation-are-hiring/ 


 As an alternative to use SHA-1 hashing. There's the option to have a 
 daemon running the background listing on filesystem events.

 BTW, is timestamps vs SHA-1 hashing really the most pressing issue 
 with Dub?

 -- 
 /Jacob Carlborg


 Agreed. A server approach would probably scale much better,
 if the intent is to speed up the developer's CTR cycle.
 But in any case, thanks to Symmetry for doing this! This is huge.


 
 We need both a fs-watcher daemon support and "offline" incremental build 
 support, based on SHA-256 (note that git is moving away from SHA1 to 
 SHA256 [1]). I'd say SHA-256 is cheap enough these days [2] that I don't 
 see a reason not to use it even for "online" fs-watcher daemon compilation.
 
 [1]: https://git-scm.com/docs/hash-function-transition/
 [2]: https://bench.cr.yp.to/impl-hash/sha256.html


We can easily use the following option:
```
dub build --hash=sha1
dub build --hash=sha256
```
and let the user to make the final choice
Sep 01 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 1 September 2020 at 16:45:55 UTC, drug wrote:

 On 9/1/20 7:34 PM, Petar Kirov [ZombineDev] wrote:

 On Tuesday, 1 September 2020 at 12:59:00 UTC, Mathias LANG 
 wrote:

 On Tuesday, 1 September 2020 at 09:09:36 UTC, Jacob Carlborg 
 wrote:

 [...]


 Agreed. A server approach would probably scale much better,
 if the intent is to speed up the developer's CTR cycle.
 But in any case, thanks to Symmetry for doing this! This is 
 huge.


 
 We need both a fs-watcher daemon support and "offline" 
 incremental build support, based on SHA-256 (note that git is 
 moving away from SHA1 to SHA256 [1]). I'd say SHA-256 is cheap 
 enough these days [2] that I don't see a reason not to use it 
 even for "online" fs-watcher daemon compilation.
 
 [1]: https://git-scm.com/docs/hash-function-transition/
 [2]: https://bench.cr.yp.to/impl-hash/sha256.html


 We can easily use the following option:
 ```
 dub build --hash=sha1
 dub build --hash=sha256
 ```
 and let the user to make the final choice


I understand your idea to make this configurable, but this will 
introduce more complexity than necessary because then 
$HOME/.dub/packages will have build artifacts based on both SHA1 
and SHA256, and so dub will need to support mixed mode file 
integrity checking.
Sep 02 2020
parent reply drug <drug2004 bk.ru> writes:
On 9/2/20 12:53 PM, Petar Kirov [ZombineDev] wrote:

 
 I understand your idea to make this configurable, but this will 
 introduce more complexity than necessary because then 
 $HOME/.dub/packages will have build artifacts based on both SHA1 and 
 SHA256, and so dub will need to support mixed mode file integrity checking.


Yesterday I thought about it and now I believe that if you change sha1 
to sha256 or vice versa then dub just rebuild all because sha1 and 
sha256 has different sizes so cached hashes will be different to 
calculated ones and collisions are impossible. No need to support mixed 
mode. The drawback is total rebuild on algorithm changing but I suppose 
it will be seldom operation. Another way is to having several hashes 
simultaneously and select one of them depending on option value. 
Implementation of this will be simple.
Sep 02 2020
parent reply drug <drug2004 bk.ru> writes:
On 9/2/20 1:18 PM, drug wrote:

 it will be seldom operation. Another way is to having several hashes 
 simultaneously and select one of them depending on option value. 


I meant having several hashes in dub cache. Then if the user changes the 
hash algorithm dub checks calculated hashes against cached ones and 
avoids total rebuilding.
Sep 02 2020
parent drug <drug2004 bk.ru> writes:
On 9/2/20 1:21 PM, drug wrote:

 On 9/2/20 1:18 PM, drug wrote:

 it will be seldom operation. Another way is to having several hashes 
 simultaneously and select one of them depending on option value. 


 
 I meant having several hashes in dub cache. Then if the user changes the 
 hash algorithm dub checks calculated hashes against cached ones and 
 avoids total rebuilding.
 


Hmm, probably I was too confident. I've implemented support for both 
sha1 and sha256. It takes about  30 lines of additional code but what is 
worse the code uses classes now and dynamic memory allocation. I would 
avoid it. But in other it works as I expected:
```
dub build --hash=sha1
dub build --hash=sha256
```
I do not test it in big projects but building dub does not show any 
difference in performance
Sep 02 2020
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


This is a great initiative!

It's also a great opportunity for up-and-coming D contributors. I 
know that if I had this chance back many years ago when I was 
unemployed I would have jumped at the opportunity. So if you're 
in a similar position don't hesitate to apply!
Sep 02 2020
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously 
 funding two positions for ecosystem work under the D Language 
 Foundation. And they've put up a bounty for a new DUB feature. 
 Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-foundation-are-hiring/


One other thing that hashing enables is caching builds. So if you 
build, then change a file, then build again, and then revert the 
file again it would be great if the next build just restored the 
cached build from the very first build.

The less time spent waiting on builds, the better.
Sep 05 2020
parent drug <drug2004 bk.ru> writes:
On 9/5/20 3:55 PM, Andrej Mitrovic wrote:

 On Sunday, 30 August 2020 at 14:13:36 UTC, Mike Parker wrote:

 Looking for a full-time or part-time gig? Not only is Symmetry 
 Investments hiring D programmers, they are also generously funding two 
 positions for ecosystem work under the D Language Foundation. And 
 they've put up a bounty for a new DUB feature. Read all about it here:

 https://dlang.org/blog/2020/08/30/symmetry-investments-and-the-d-language-fo
ndation-are-hiring/ 


 
 One other thing that hashing enables is caching builds. So if you build, 
 then change a file, then build again, and then revert the file again it 
 would be great if the next build just restored the cached build from the 
 very first build.
 
 The less time spent waiting on builds, the better.


Could you test this feature https://github.com/dlang/dub/pull/2077?

The simple way to do it would be adding this file to the project root 
directory:
```json
{
     "buildCachePolicy" : "sha1"
}
```
This enables hash dependent build based on sha1. Get ready to spent more 
storage space for your builds, of course.
Jan 13 2021