• Adam D. Ruppe (224/224) Apr 13 2017 I haven't posted in announce for a while, so I figured I'd give
• H. S. Teoh via Digitalmars-d-announce (64/105) Apr 13 2017 I've been using terminal.d for a lot of my projects. It's a really nice
• Adam D. Ruppe (75/101) Apr 13 2017 Hah.... though actually, my terminal emulator has support for
• Basile B. (3/9) Apr 13 2017 Thanks particularly for this one, which is certainly a reference
• Adam D. Ruppe (5/7) Apr 14 2017 I actually don't think X is that bad for what it does..... just
• Jerry (2/2) Apr 14 2017 You have mixed spaces and tabs, not in the good way either.
• Adam D. Ruppe (6/8) Apr 14 2017 Some code is copied from C headers, some come from open source

Adam D. Ruppe <destructionator gmail.com> writes:

I haven't posted in announce for a while, so I figured I'd give 
you guys an update on what's been happening in my libs. This 
message is a big long, so feel free to just skim to see if you 
are interested in anything.

Here's my three main public repos:

https://github.com/adamdruppe/arsd
https://github.com/adamdruppe/terminal-emulator
https://github.com/adamdruppe/adrdox

  * * * * *

Going bottom to top: adrdox is the documentation generator for my 
http://dpldocs.info/ website. In the last few months, it:

* Is easier to use by third parties. If you download it and hit 
`make`, then run it from its own source directory (that's 
important right now) `./doc2 path/to/your/project`, it will 
generate your docs. So you don't have to be on my computer to run 
it anymore!

* It has a javascript search option, so you can use it 
out-of-the-box with search enabled on a static site, like  github 
pages.

* It has gained a lot of automatic cross referencing in source 
code.

* The documentation generating language got a few more features 
like footnotes. See the syntax doc here: 
http://dpldocs.info/experimental-docs/adrdox.syntax.html

  * * * * *

terminal-emulator is my custom Linux terminal emulator. It 
includes four programs:

* a Linux program that works like xterm, just with my preferences 
and features. For example, it adjusts the palette so if a program 
prints yellow on white, it will adjust to a darker yellow to make 
it easier to read.

* a Windows program that works just like Linux, but instead of 
running a local shell, it ssh connects to a Linux box and runs 
the shell there.

* a Linux program that enables tabbing and detaching of terminal 
sessions, similar to GNU Screen, but forwarding my custom 
features. For example, scrollback with shift+pageup works through 
all levels of nesting, and it can do bracketed pastes through the 
levels.

* and a nested terminal emulator that I rarely use, but could be 
used to split screens and such.

It also holds the stand-alone terminalemulator.d core, which is 
independent of any UI.

It has mostly gotten bug fixes over the last several months, 
including a major one on scrolling back long lines. I use it for 
all my terminal emulation need (which is almost everything I do 
on computers!), so the more I use it, the more fixes it gets.

  * * * * *

arsd contains my generally useful library modules. See the docs 
here: http://dpldocs.info/experimental-docs/arsd.html

Some of the modules are obsolete or experimental, but many of 
them are stable for many years, including:

* cgi.d can be used for D web programs using existing libraries - 
it uses simple process and thread model rather than the 
newfangled async event loops, which means it is compatible with 
almost any code you write. It works with cgi, fastcgi, scgi, or 
you can run an embedded http server in it. It stands alone, 
though web.d (I haven't used it for a while so it might need some 
updating) and others can build on top of it.

It hasn't changed for a while, no new updates here.

* dom.d is a html or xml dom parser, capable of reading tag soup. 
Its API is similar to Javascript, so if you have used the browser 
apis before, you should be able to dive in pretty quickly. It has 
no required dependencies.

In recent months, it has had a few small bug fixes and gained a 
new api: `optionSelector` 
http://dpldocs.info/experimental-docs/arsd.dom.Element.optionSelector.html

It uses the mostly-CSS3-compatible selector syntax, same as 
querySelector and requireSelector, but returns a null-safe 
object, meaning you can use it for scrapes.

optionSelector(".foo").innerText; // instead of throwing or 
segfaulting if .foo does not exist, it just returns an empty 
string

It has also gained a Document.fromUrl function which will 
download and parse some web site. This requires 
characterencodings.d and http2.d as well.


dom.d has also received a few more bug fixes in reading tag soup 
as I use it for more and more web scraping. See also: 
htmltotext.d which uses dom.d to convert html to plain text and 
html.d for more html helper functions. It includes a css expand 
thing similar to scss. It hasn't changed much recently, either.


* database.d, postgres.d, sqlite.d, mysql.d, and mssql.d are 
wrappers of the C libraries. They haven't changed much recently.


* http2.d is a http/1.1 client library. (It is the successor to 
my http.d lib and does not yet have http/2 support. And maybe 
never will since 1.1 client is plenty good enough for everything 
I use it for).

It has gotten some new stuff recently:

   - HttpClient is a class that works kinda like a browser, with 
cookie storage, connection reuse, and relative link handling. It 
has received a number of bug fixes in the last few months, 
especially with regard to SSL (for which it requires OpenSSL 
library btw).

   - The HttpApiClient template has convenience methods for 
working with Oauth2 json apis. It is still a work in progress but 
you can read some about it here: 
http://dpldocs.info/experimental-docs/arsd.http2.HttpApiClient.html It requires
jsvar.d


* jsvar.d and script.d provide a Javascript-esque type for D with 
easy interop into the rest of the language. jsvar itself just 
provides the data type, and script.d expands it with a dynamic 
scripting language interpreter too (no outside dependencies btw). 
They've changed a bit recently:

   - jsvar is even more loosely typed than it was before: 
accessing a member of a null object simply returns null too. This 
is convenient for accessing nested JSON objects where they may 
not be present (similar to the use case for optionSelector in 
dom.d), though makes errors harder to find. I'll probably go back 
to some kind of middle ground on that eventually.

   - script has more bug fixes and gains from every jsvar change. 
Notably, it can now more reliably wrap native D class objects, 
but it does so in an opaque way, just holding and returning the 
object on demand, unless the object's specific methods are marked 
with a UDA ` "arsd_jsvar_compatible"`. That UDA has nicer aliases 
in a few of my modules like ` scriptable`. The reason it is a 
string though is I can mark it in other modules without 
introducing a dependency on jsvar so the user doesn't pay for it 
if they don't use it.

   - The script language now allows identifiers-with-dashes, since 
I like how they look in scripts a lot. To do subtraction, you 
must put spaces around the - character.


* terminal.d is a terminal client library, similar to ncurses. It 
allows you to move the cursor around the terminal, go to full 
screen mode, output color, input all keys and mouse events.

In recent months, it has gained 24-bit color output support (with 
automatic approximation to lower color fallback), though I 
generally suggest you keep to 8 color output for best results in 
terminal programs.

It also got a `getline` function with user editing and 
autocompletion available, unified some of the key input events, 
and a ScrollbackBuffer which is clunky to use, but allows you to 
have individual scroll buffers scattered around the screen.

It also received a few small bug fixes for Mac OS X.


* simpleaudio, joystick, and gamehelpers haven't been changed 
yet, but it is my plan to almost completely redo them. I am 
reasonably happy with joystick.d, but simpleaudio.d is probably 
going to be gutted and rewritten. If you actually use it as-is, 
let me know and I'll work with you on compatibility, but I doubt 
that...


* simpledisplay.d has gotten a LOT of work lately.

   - Its ScreenPainter is now more reliable if you pass it to 
other functions. It does refcounting but also sets the impl 
values back in dtors so it acts like a hybrid value/reference 
type.

   - The NotificationAreaIcon now works on Windows and Linux, with 
balloon notifications.

   - There's a new postEvent API with collapsing and timeouts, 
allowing multithreaded programs to run efficiently.

   - It has support for creating OpenGL 3+ contexts.

   - There are more exposed low level events - simpledisplay aims 
to be simple for many things, but not get in your way if you want 
to do more with it than it was specifically designed for.

   - It has timer and Linux file read support in its event loop 
now, though postEvent can be better than Timer for game purposes, 
and the Linux file read is use-at-your-own-risk experimental.

   - ScreenPainter can do clipping and transformations.

   - Sprite can now be drawn upon, just like SimpleWindow. Sprite 
is a class that contains a display-side image for quick blitting 
(kinda like a 2d version of a opengl texture).

   - SimpleWindow can construct more window types like popup 
menus, for use building custom UIs, among other things.

   - OperatingSystemFont gives a way to load a font off the OS and 
draw with it in a ScreenPainter.

   - You can captureInput to a window, getting mouse tracking or 
even confinement.

   - There are platform-specific functions for handling global 
hotkeys. Still work in progress in unifying the APIs.

   - simpledisplay.d can now survive an X server disconnect, and 
can even migrate windows from one X server to another! Think GNU 
Screen for GUI apps.

   - The documentation has been expanded, though it still needs 
cleanup, it includes more and more examples. See: 
http://dpldocs.info/experimental-docs/arsd.simpledisplay.html

   - Other little features and bug fixes.


I haven't used SDL or anything similar for years thanks to 
simpledisplay.d. Its only dependency is color.d, leading to 
simple builds.


* And the huge one in the last month, minigui.d is a small GUI 
widget library that aims to use native widgets on Windows and a 
small set of custom ones on Linux (in the future, it might also 
support GTK widgets, though I don't plan to write that myself, 
there is just another user who said he might). It has no 
dependencies except for simpledisplay.d and color.d and the 
operating system - no complicated library build process, no huge 
dlls to distribute with your program.

http://dpldocs.info/experimental-docs/arsd.minigui.html

Before now, it was fairly usable on Windows, but only had buttons 
on Linux. Now, it is about equally featured on both with widgets 
like scroll areas, checkboxes, buttons, progress bars, menus, 
drop down selects, textareas, and more. It aims to cover the 
basic boxes that HTTP 4 forms did, while also being easy to 
define your own widgets - you just subclass it and draw in the 
area it assigns you. It now even has an OpenGL widget.

It automatically sizes and lays out your widgets and has an event 
system similar to Javascript. With add on modules like 
minigui_xml, you can create windows from XML definitions, and it 
is script.d compatible. When I'm ready, I'll post that too, and 
hopefully you can take basic web knowledge over to desktop 
land... without just embedding browsers and calling them desktop 
apps. My largest program is about 800 KB, and that's all it needs.

Since it is built on simpledisplay, it also gets X connection 
disconnect survival or migration!

Old screenshot: http://arsdnet.net/guitest.png
New screenshot: http://arsdnet.net/minigui.windows.new.png
New screenshot, Linux version: 
http://arsdnet.net/minigui.linux.d.png
an app I'm writing with it: http://arsdnet.net/minigui-sprite.png
xml app screenshot: http://arsdnet.net/minigui-scripted-app.png


Demo program source code: http://arsdnet.net/guitest.d


I'll write more about minigui.d and do a separate announcement 
for it when it reaches version 1.0, which will probably be in 
about another month.

The look on Linux is based on Windows 95, I think it is elegant 
and usable, and is also very efficient for remote X connections, 
so I can use them from my laptop on slow wifi.

minigui also has more add on widgets: 
https://github.com/adamdruppe/arsd/tree/master/minigui_addons

In the core is a file picker, which is very fast on Linux and 
native on Windows. In addons, we add a color picker and a 
terminal emulator widget (depends on the terminal emulator core, 
mentioned above in this message).


Look forward to more on this, it is close to being ready for 
general use, and then we'll have another chapter in the D gui 
ecosystem. dlangui <https://github.com/buggins/dlangui>, I hope 
you like some friendly competition :)

"H. S. Teoh via Digitalmars-d-announce" writes:

On Thu, Apr 13, 2017 at 04:04:18PM +0000, Adam D. Ruppe via
Digitalmars-d-announce wrote:
[...]
 * terminal.d is a terminal client library, similar to ncurses. It
 allows you to move the cursor around the terminal, go to full screen
 mode, output color, input all keys and mouse events.

I've been using terminal.d for a lot of my projects.  It's a really nice
terminal library that has a much better (IMO) API than the crawling mess
that is ncurses. Thanks so much for sharing it!


 In recent months, it has gained 24-bit color output support (with
 automatic approximation to lower color fallback), though I generally
 suggest you keep to 8 color output for best results in terminal
 programs.

Hooray!  Finally!  Maybe now I can play videos over SSH (albeit
ultra-pixelated). :-P


 It also got a `getline` function with user editing and autocompletion
 available, unified some of the key input events, and a
 ScrollbackBuffer which is clunky to use, but allows you to have
 individual scroll buffers scattered around the screen.

Very nice. So far I've been rolling my own for user input, but it's nice
to have built-in support.

And on that note, I've been playing around with my own range-like
abstractions over Terminal (variously called Display or Canvas), that
lets you treat rectangular subsets of the Terminal as sub-terminals in
their own right (as least as far as output is concerned).  Basically,
there's an isCanvas template that acknowledges anything that has moveTo
and writef as a canvas-like type (and obviously designed so that
Terminal automatically qualifies), along with a series of introspection
templates to identify if a canvas has a clear method, a flush method,
etc.. On top of this there's a subcanvas UFCS function that lets you
create arbitrary rectangular views over any canvas (with automatic
moveTo coordinate translations, etc.).  So I can write code that will
work either directly with a Terminal, or any other Terminal-like
backends as well as arbitrary rectangular subregions.


[...]
 * simpledisplay.d has gotten a LOT of work lately.

I've been eyeing SDL (via Derelict) recently... but maybe if
simpledisplay.d does what I need, I might just use it instead.  Leaner
and meaner is always better, I think!


[...]
   - simpledisplay.d can now survive an X server disconnect, and can
   even migrate windows from one X server to another! Think GNU Screen
   for GUI apps.

Now *this* really caught my eye.  I'm gonna hafta check it out just for
this feature alone.  It has always annoyed me that although such a thing
has always been *possible* for a network-oriented GUI protocol like X11,
hardly any programs (if at all!) are written in a way that would work
this way.  This is a really awesome feature and now I'm *really* tempted
to try it out!


[...]
 I haven't used SDL or anything similar for years thanks to
 simpledisplay.d.  Its only dependency is color.d, leading to simple
 builds.

Now I'm very tempted to look into simpledisplay.d instead of SDL.


 * And the huge one in the last month, minigui.d is a small GUI widget
 library that aims to use native widgets on Windows and a small set of
 custom ones on Linux (in the future, it might also support GTK
 widgets, though I don't plan to write that myself, there is just
 another user who said he might). It has no dependencies except for
 simpledisplay.d and color.d and the operating system - no complicated
 library build process, no huge dlls to distribute with your program.

Awesome!!!  I always hated those bulky toolkit libraries that includes
everything and the kitchen sink -- 90% of which you don't actually need
in your application, which usually means you have a ton of dependencies
that aren't actually necessary and library dependency hell when you ship
your binaries.


[...]
 It automatically sizes and lays out your widgets and has an event
 system similar to Javascript. With add on modules like minigui_xml,
 you can create windows from XML definitions, and it is script.d
 compatible. When I'm ready, I'll post that too, and hopefully you can
 take basic web knowledge over to desktop land... without just
 embedding browsers and calling them desktop apps. My largest program
 is about 800 KB, and that's all it needs.

That's cool, man.  Now I gotta check this out too! :-P


 Since it is built on simpledisplay, it also gets X connection
 disconnect survival or migration!

This is a killer feature, IMO.


[...]
 I'll write more about minigui.d and do a separate announcement for it
 when it reaches version 1.0, which will probably be in about another
 month.

Awesome!


[...]
 Look forward to more on this, it is close to being ready for general
 use, and then we'll have another chapter in the D gui ecosystem.
 dlangui <https://github.com/buggins/dlangui>, I hope you like some
 friendly competition :)

This looks like a potential awesome trend, especially that bit about X
server disconnection survival / migration.  I think we could write a
killer app on top of this.  Say a terminal that travels with you
wherever you go -- migratable from desktop to laptop on wifi and back,
perhaps even to/from your {smart/dumb}phone or tablet, so don't have to
bother with reconnecting your current code hacking session, but can just
pick up and keep going from where you left off instantaneously.

Or a lightweight mini-browser that follows you wherever you go. Etc.

We could start a new trend based on migratable apps that can move around
from device to device seamlessly. Sure, browser apps have this to some
extent, but we're talking about not even needing to start your browser
and relogin and all that jazz -- just resume in exactly the same state
you left it on the previous device.  Or ping-pong a terminal window
between you and a colleague for instantaneous collaboration.  Best of
all, the programmers only need to put in minimal effort to support it.


T

-- 
VI = Visual Irritation

Adam D. Ruppe <destructionator gmail.com> writes:

On Thursday, 13 April 2017 at 18:11:10 UTC, H. S. Teoh wrote:
 Hooray!  Finally!  Maybe now I can play videos over SSH (albeit
 ultra-pixelated). :-P

Hah.... though actually, my terminal emulator has support for 
outputting png images...

But truth is, this is one reason why I prefer things like 
`attach` or the new simpledisplay feature to VNC and the like - I 
can then combine programs from different computers on one 
desktop. So if I do need to play videos, I get them in the 
compressed format over whatever link and play them locally on the 
laptop.

I rarely run browsers remotely, even though sometimes I wish I 
could just because the local one tends to be much easier to use.


 And on that note, I've been playing around with my own 
 range-like abstractions over Terminal (variously called Display 
 or Canvas), that lets you treat rectangular subsets of the 
 Terminal as sub-terminals in their own right (as least as far 
 as output is concerned).

Nice. I actually thought about doing something like that 
(basically nested ScrollbackBuffers with a nicer api) too, but 
haven't gotten around to it yet.

 Now *this* really caught my eye.  I'm gonna hafta check it out 
 just for this feature alone.  It has always annoyed me that 
 although such a thing has always been *possible* for a 
 network-oriented GUI protocol like X11, hardly any programs (if 
 at all!) are written in a way that would work this way.


Indeed. So the way it works right now is there's an 
XDisconnectException that is thrown. If you want to recover, you 
must catch it OUTSIDE the event loop, then adjust the display 
string if appropriate and run 
XDisplayConnection.discardAndRecreate. If it succeeds, you can 
reenter your event loop and pick up where you left off.

Any X resources you had may need to be recreated. The library 
will automatically handle things it has inside (SimpleWindow 
right now, I'll add support to the other classes soon) but if you 
go outside it, you might have to do it yourself.

Server-side fonts, in particular, may need attention because they 
might no longer be available. But I might automate that too (keep 
a list of fallbacks inside the class).

Well, anyway, the big point is it isn't 100% automatic, you do 
need to catch the exception, possibly fix some stuff, and 
reconnect, but it is almost automatic; in simple apps, it is as 
simple as `catch(...) discardAndRecreate(); loopAgain;`

But it DOES work! :P

 Awesome!!!  I always hated those bulky toolkit libraries that 
 includes everything and the kitchen sink -- 90% of which you 
 don't actually need in your application, which usually means 
 you have a ton of dependencies that aren't actually necessary 
 and library dependency hell when you ship your binaries.

Yeah, I avoid distributing most programs because I can't stand 
having 20 MB + of stuff and the poor users have to manage that 
crap. Gah.

And I avoid *compiling* most gui programs because that's a huge 
pain too. But not with minigui, you've used my libs, you know how 
it is

dmd yourapp.d minigui simpledisplay color

Boom, it works. There's a few tweaks you can do, you might want 
some of the optional add on modules (which I am keeping each one 
strictly independent - they are not allowed to import each other 
to keep that decoupled), but the basic of it is right there.

And it builds in a fraction of a second. No meta-object compiler, 
no bindings generation, no dlls. And the generated file can be as 
small as 200 KB if you strip it right, though my test programs 
tend to hover closer to 600 on Windows, ~800 on Linux. I can 
handle that a lot easier than 20,000 KB.

Adding widgets is just newing the class. The automatic layout is 
usually ok. Custom widgets? Just subclass (actually until 
yesterday, it was all done via delegates, but subclassing is 
actually a bit easier, thanks D's anonymous classes) and 
implement paint and events. Again, no multiple files to generate 
and compile.


 I think we could write a killer app on top of this.  Say a 
 terminal that travels with you wherever you go -- migratable 
 from desktop to laptop on wifi and back, perhaps even to/from 
 your

Actually, I already have that: `attach` in the terminal emulator 
repo (it is similar to GNU screen, just my style, including 
things like scrollback, window icons, mouse support, and more).

My "IDE" tends to be an attach session running everything I want 
in the terminal tabs and I grab them by name when moving around.

That's the big reason why I have spent more time on terminal.d 
than minigui.d until now: my terminal sessions follow me on the 
laptop and such, so I wrote a LOT of terminal programs. But now, 
I can do GUI too! So we'll see how it goes.

The one problem I have with a quick migration is that the gui one 
is still somewhat bandwidth inefficient. Just rolling your mouse 
over a window generates several KB of network traffic. On a real 
connection... no big deal, but sometimes I use my laptop on a 
mobile metered connection, and I pay by the kilobyte there.

The good news though is I should be able to optimize that. I'll 
keep the api being fat by default, but have the ability to turn 
off mouse motion reporting on demand, and will optimize the 
redraws and atom querying and others (that's a reason why the 
styles are so simple btw, other people transfer big pngs over, I 
transfer over small XDrawRectangle packets).


 just resume in exactly the same state you left it on the 
 previous device.  Or ping-pong a terminal window between you 
 and a colleague for instantaneous collaboration.  Best of all, 
 the programmers only need to put in minimal effort to support 
 it.

Indeed.

Basile B. <b2.temp gmx.com> writes:

On Thursday, 13 April 2017 at 16:04:18 UTC, Adam D. Ruppe wrote:
 I haven't posted in announce for a while, so I figured I'd give 
 you guys an update on what's been happening in my libs. This 
 message is a big long, so feel free to just skim to see if you 
 are interested in anything.

 [...]

 * simpledisplay.d has gotten a LOT of work lately.

Thanks particularly for this one, which is certainly a reference 
about how to use the nightmare-ish X11 API in D.

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 14 April 2017 at 06:23:00 UTC, Basile B. wrote:
 Thanks particularly for this one, which is certainly a 
 reference about how to use the nightmare-ish X11 API in D.

I actually don't think X is that bad for what it does..... just 
that it is ridiculously underpowered compared to the competition.

If you want a real nightmare, take a look at terminal.d's 
innards. Eeek.

Jerry <hurricane hereiam.com> writes:

You have mixed spaces and tabs, not in the good way either. 
https://github.com/adamdruppe/arsd/blob/master/simpledisplay.d#L3642

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 14 April 2017 at 19:55:54 UTC, Jerry wrote:
 You have mixed spaces and tabs, not in the good way either. 
 https://github.com/adamdruppe/arsd/blob/master/simpledisplay.d#L3642

Some code is copied from C headers, some come from open source 
contributors who may use different styles than I do. I accept 
these gifts in the spirit in which they are given.

If I edit that code, I might reformat it, but it isn't important 
to me.