• Ethan (113/113) Jun 03 2018 Hey dlang community.
• rikki cattermole (2/16) Jun 03 2018 Make that a final switch, that should generate better assembly.
• Kagamin (4/18) Jun 05 2018 Why message detection is in receiver instead of infrastructure?
• Ethan (25/28) Jun 05 2018 Because recursion. One of the messages I've written is a wrapper

Ethan <gooberman gmail.com> writes:

Hey dlang community.

I've already been thinking in advance for DConf next year. If 
things keep going well for me, I may not have anything I can 
publicly talk about. So I've been thinking about what kind of 
capacity I can contribute to the conference.

The easy way would be to raise my hand for volunteering to be MC. 
I get good feedback about my humour and energy levels during my 
talks, and translating that to an entire event means I could 
probably relax for a change and not worry about making mistakes 
in my talk/actually getting everything ready in time.

Another thing that's becoming apparent though is that there needs 
to be more widely-available, simple to understand tutorials for 
the kind of work I do. There's Andrei's book, which is the first 
stop shop. And then there's Adam's D Cookbook. But I can't think 
of much else.

So. Being the kind of programming communist I am, I write 
something cool and think "I need to talk about this." Hence the 
posts I've made in here lately. But it also got me thinking. 
Maybe these kind of posts would make a great DConf talk. Or a 
website. Or whatever, I'm only coming in to this with the 
mentality of DConf next year.

This is something I've emailed to friends/posted to Facebook (but 
slightly sanitised so that the troll from Melbourne hiding behind 
a Tor connection won't derail the thread with "sex talk" 
complaints). It discusses stuff at a high level, with only a 
handful of specific examples. And I can go in to details. But 
keeping things understandable at a high level and using a bit of 
humour at the same time goes a very long way in my experience.

So if stuff like the following, but presented for a proper 
audience, seems like it'll be valuable, I'll add all these 
examples I'm writing to a presentation as I go throughout the 
year and submit that for my talk when the time comes.

===

I've been writing a client/server architecture. In D.

And I harp on about D, I know. But wow is it making things easy.

The kind of programming paradigms I'm using to get it down to 
"simple as pie usability" aren't widely known outside of the D 
community's best and brightest. But knowing what the language is 
capable of, I can just go ahead and do it and it works.

So you're a client and a server. Which means you're going to want 
to send and receive messages to each other. How would you do this 
in C++? Maybe an enum for message type so you can switch on it? 
Then some way of casting the data from the bytestream? And what 
about variable length messages? So rather than cast you need a 
proper deserialisation. Just send everything as JSON? Yeah, 
goodbye network bandwidth, but at least you don't have to mess 
around with binary.

The only method there that is low-maintenance for futureproofing 
is the JSON method I mentioned. And many people will stop there 
and be happy. But I'm not stopping there. I'm writing a gaming 
middleware, which implicitly means there's going to be large 
quantities of data I'll need to deal with.

So, how do you make maintenance a breeze when you write such a 
system from scratch?

Step one: All your messages are structs. "Value types" for you 
modern programmers.

Step two: Apply a user-defined attribute to your struct. I have 
ClientMsg for messages sent by the client, and ServerMsg for 
messages sent by the server. What if both client and server send 
the message?

 ClientMsg  ServerMsg struct Ping { }

Done.

Step three: Collect all messages from the message modules and 
stick them in a tuple. No need to register your message. It's 
done for you.

Step four: Filter that tuple out in to separate client and server 
message tuples.

Step five: Write a template that generates a size_t that hashes 
together the names of all members of any given message. This has 
the built-in benefit of loosely versioning your structures. And 
thanks to CTFE and collecting all message types earlier, we can 
check at compile time for hash collisions.

Step six: Serialisation of your structures starts out by sticking 
that size_t in a buffer, and then parsing each member of your 
struct and either copying in to the buffer directly or specifying 
a length before copying N elements from the array you've 
encountered. Deserialisation works in the exact opposite manner.

Step seven: In your receive function that takes a byte stream, 
put in a switch statement that looks a little bit like the 
following:

switch( msg.GetID )
{
     static foreach( Message; ServerMessages )
     {
     case ObjectIDOf!Message:
         Message deserialised = msg.FromBinary!Message;
         this.receive( deserialised );
     }
     default:
         break;
}

Yes, we basically generate at compile time an entire switch 
statement from structures. This currently will generate a fairly 
inefficient jump table, but you know, we can add an indexing 
declarator to that foreach statement and let the compiler 
optimise down to a proper jump table with a little bit more 
effort.

Step eight: Anyone that wants to add a message to your system 
will now immediately get a compile error if they don't implement 
void receive( ref MessageType msg ) in their server or client.


representation from all this. So I can keep my server highly 
performant and write a frontend that won't make people want to 
send death threats to the company's Twitter account.

I'm writing this code for my start-up tech company, so being a 
rock solid base that is maintainable will be a key feature for 
reducing time and maintenance costs in the future. And honestly, 
I'm an expert level C++ programmer these days and know a thing or 
two about a few other languages, and none of the languages I'm 
familiar with can make things as simple as "write a struct". 
Getting anything like this out of C++, for example, requires you 
jump language hoops and then give up and write an external 
pre-processor.

rikki cattermole <rikki cattermole.co.nz> writes:

On 04/06/2018 4:25 AM, Ethan wrote:
 Step seven: In your receive function that takes a byte stream, put in a 
 switch statement that looks a little bit like the following:
 
 switch( msg.GetID )
 {
      static foreach( Message; ServerMessages )
      {
      case ObjectIDOf!Message:
          Message deserialised = msg.FromBinary!Message;
          this.receive( deserialised );
      }
      default:
          break;
 }

Make that a final switch, that should generate better assembly.

Kagamin <spam here.lot> writes:

On Sunday, 3 June 2018 at 16:25:29 UTC, Ethan wrote:
 Step seven: In your receive function that takes a byte stream, 
 put in a switch statement that looks a little bit like the 
 following:

 switch( msg.GetID )
 {
     static foreach( Message; ServerMessages )
     {
     case ObjectIDOf!Message:
         Message deserialised = msg.FromBinary!Message;
         this.receive( deserialised );
     }
     default:
         break;
 }

Why message detection is in receiver instead of infrastructure? 
And why not gather message types from receiver's interface with 
DbI (like WCF does)?

Ethan <gooberman gmail.com> writes:

On Tuesday, 5 June 2018 at 13:33:18 UTC, Kagamin wrote:
 Why message detection is in receiver instead of infrastructure?

Because recursion. One of the messages I've written is a wrapper 
message for a multi-packet split message, and calls receive with 
the reconstructed byte buffer. Fairly elegant way to not 
special-case the thing that much.

 And why not gather message types from receiver's interface with 
 DbI (like WCF does)?

There already is design by introspection. But I don't parse a 
type, I parse an entire module. The switch statement is being 
built through the results of an introspective pass.

This is quite deliberate. I'm writing a large-scale maintainable 
codebase. Having everything in one file is a sure way to reduce 
maintainability and thus productivity of the programmers 
maintaining it. Getting D to favour lots of smaller files means 
getting creative.

I *could* put all the messages in an interface and inherit from 
it... but that's a fairly old-school way of thinking. I don't 
need a giant virtual function table to enforce implementing 
message types when I can use outside-the-box introspective tricks 
and compile down to nothing.

There's also a design advantage to going message-first here. I'm 
forcing the maintainers to think of the data before they get to 
implementation. The existence of a struct already explicitly 
creates one piece of data - the message ID. Anything else you put 
in there is up to you. And being structs, means that you don't 
constantly have to maintain function signatures each time you 
want to add a value to a message for example.