www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Background thread, async and GUI (dlangui)

reply Bagomot <bagomot gmail.com> writes:
Hello! I have a few questions about multithreading, asynchrony 
and how it works with the GUI.

1) How to make asynchronous HTTP requests with curl in order to 
receive progress and status? And how do you know that the request 
is completed? This question is related to the next one, so it 
should be answered in the context of working with the GUI.

2) How to get and update information in the GUI (I use dlangui) 
about the execution of an asynchronous http request (for example, 
downloading a file)? Something like a progress bar.

3) My application must perform some work in a separate thread (I 
do this through inheritance of the worker from 
core.thread.Thread, is that correct?). I need to be able to 
manage data in the GUI from this thread without blocking GUI. I 
found that you can use RunnableEvent for dlangui, but I can’t 
figure out how to do it (and I didn’t find any examples). Could 
you suggest how to do it?

Sorry if the questions here are a bit from different topics, but 
they are all related to dlangui.
Jul 06 2022
next sibling parent GrimMaple <grimmaple95 gmail.com> writes:
On Wednesday, 6 July 2022 at 09:26:35 UTC, Bagomot wrote:

 Hello! I have a few questions about multithreading, asynchrony 
 and how it works with the GUI.

 1) How to make asynchronous HTTP requests with curl in order to 
 receive progress and status? And how do you know that the 
 request is completed? This question is related to the next one, 
 so it should be answered in the context of working with the GUI.

 2) How to get and update information in the GUI (I use dlangui) 
 about the execution of an asynchronous http request (for 
 example, downloading a file)? Something like a progress bar.

 3) My application must perform some work in a separate thread 
 (I do this through inheritance of the worker from 
 core.thread.Thread, is that correct?). I need to be able to 
 manage data in the GUI from this thread without blocking GUI. I 
 found that you can use RunnableEvent for dlangui, but I can’t 
 figure out how to do it (and I didn’t find any examples). Could 
 you suggest how to do it?

 Sorry if the questions here are a bit from different topics, 
 but they are all related to dlangui.


I can't say anything about making asynchronous requests, but in 
dlangui simply updating any of the properties should cause a 
redraw. This means, if you set a new value to a progress bar, 
there isn't much else to do. Ideally, your library will provide a 
callback in which you can update anything you need.

If you don't receive a callback from your requests lib, you can 
poll it using timers in dlangui. For that you'd need to subclass 
any widget and override the `onTimer` method:
```d
class CustomCanvas : CanvasWidget
{
     this(string ID = null)
     {
         super(ID);
     }

     override bool onTimer(ulong id)
     {
         window.showMessageBox("OnTimer"d, "OnTimer"d);
         return true;
     }
}
```

And then set a timer in your code somewhere
```d
auto window = ...
auto canvas = new CustomCanvas("canvas");
window.setTimer(canvas, 1000); // will call onTimer every 1000ms
```
Jul 06 2022
prev sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 7/6/22 02:26, Bagomot wrote:


 1) How to make asynchronous HTTP requests with curl in order to receive
 progress and status? And how do you know that the request is completed?


I don't know how dlangui or others automate this but I had fun writing 
the following program that uses std.concurrency and std.net.curl:


 3) My application must perform some work in a separate thread (I do this
 through inheritance of the worker from core.thread.Thread, is that
 correct?).


core.thread.Thread is low level. I would consider std.parallelism and 
std.concurrency first. (I use the latter for message passing in the 
program below.)

import std.concurrency;
import std.net.curl;
import std.conv;
import std.stdio;
import std.algorithm;
import std.range;
import std.format;

// Signifies that the main thread requests the getter to stop
// serving. (Not used here.)
struct Done {}

// Represents download progress.
struct Progress {
   size_t downloaded;
   size_t total;
}

// Represents a URL
struct Url {
   string value;
}

// The thread entry point for the getter
void getterFunc() {
   receive(
     (Done _) {
       // Not used here but this is for clean termination.
       return;
     },

     (Url url) {
       // Received a URL to get.
       auto http = HTTP(url.value);

       // std.net.curl.HTTP will call this delegate with progress
       // information...
       http.onProgress((size_t dl, size_t dln, size_t ul, size_t uln) {
         if (dl != 0) {
           // ... and we will send it along to the main thread.
           ownerTid.send(Progress(dln, dl));
         }
         return 0;
       });

       // std.net.curl.HTTP will call this delegate with
       // downloaded parts...
       http.onReceive((ubyte[] data) {
         // ... and we will send it along to the main thread.
         ownerTid.send((cast(char[])data).to!string);
         return data.length;
       });

       // Everything is set up. Let's do it.
       http.perform();
     },
   );
}

void main() {
   auto getter = spawnLinked(&getterFunc);

   getter.send(Url("dlang.org"));

   string result;
   bool done = false;
   while (!done) {
     receive(
       (LinkTerminated msg) {
         // (This may or may not be expected.)
         stderr.writefln!"The getter thread terminated.";
         done = true;
       },

       (Progress progress) {
         writeln(progress);
         if (!result.empty &&
             (progress.downloaded == progress.total)) {
           done = true;
         }
       },

       (string part) {
         result ~= part;
       },
     );
   }

   writefln!"Downloaded %s bytes:\n%s"(result.length, result);
}

Ali
Jul 06 2022
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 7/6/22 16:17, Ali Çehreli wrote:


 I would consider std.parallelism


And it looks more natural with a std.parallelism.Task:

struct Progress {
   size_t percent_;

   void set(size_t downloaded, size_t total) {
     if (total != 0) {
       import core.atomic: atomicStore;

       const value = cast(size_t)(float(downloaded) / float(total) * 100);
       atomicStore(percent_, value);
     }
   }

   size_t get() const {
     import core.atomic: atomicLoad;

     return atomicLoad(percent_);
   }
}

struct Request {
   string url;
   string result;
   Progress progress;
}

void download(Request * request) {
   import std.net.curl: HTTP;

   auto http = HTTP(request.url);

   http.onProgress((size_t dl, size_t dln, size_t ul, size_t uln) {
       if (dl != 0) {
         request.progress.set(dln, dl);
       }
       return 0;
     });

   http.onReceive((ubyte[] data) {
       request.result ~= (cast(char[])data);
       return data.length;
     });

   http.perform();
}

void main() {
   import std.parallelism : task;
   import std.stdio: writefln;

   auto request = Request("dlang.org");
   auto downloadTask = task!download(&request);
   downloadTask.executeInNewThread;

   foreach (i; 0 .. 10) {
     writefln!"Doing work on the side (%s)"(i);
     writefln!"Checking download progress: %s%%"(request.progress.get());

     import core.thread;
     Thread.sleep(100.msecs);
   }

   // Now we need the result before continuing:
   downloadTask.yieldForce();

   writefln!"Downloaded %s bytes:\n%s"(request.result.length, 
request.result);
}

Ali
Jul 07 2022
parent reply Bagomot <bagomot gmail.com> writes:
On Thursday, 7 July 2022 at 18:26:15 UTC, Ali Çehreli wrote:

Summing up your help and GrimMaple, I understood how to make the 
file download progress bar beautifully. Thank you!

I only have a question with how I can work with the GUI from 
other threads. For example, in my application I have a button 
(GUI) that starts a long running job. As I understand it, it 
should run it in a separate thread so as not to block the GUI.

I did it through Thread, but now I realized that there are more 
convenient ways through Task.

But I do not understand how I can manage the GUI from that 
separate task. For example, I start the task with the button and 
block the button press (without blocking the entire GUI), it does 
its job and unblocks the button.

This is where help is needed. How is it fundamentally correct to 
do this in D, and how to do it in general with dlangui?
Jul 08 2022
parent reply Bagomot <bagomot gmail.com> writes:
On Friday, 8 July 2022 at 08:04:45 UTC, Bagomot wrote:

 On Thursday, 7 July 2022 at 18:26:15 UTC, Ali Çehreli wrote:

 Summing up your help and GrimMaple, I understood how to make 
 the file download progress bar beautifully. Thank you!

 I only have a question with how I can work with the GUI from 
 other threads. For example, in my application I have a button 
 (GUI) that starts a long running job. As I understand it, it 
 should run it in a separate thread so as not to block the GUI.

 I did it through Thread, but now I realized that there are more 
 convenient ways through Task.

 But I do not understand how I can manage the GUI from that 
 separate task. For example, I start the task with the button 
 and block the button press (without blocking the entire GUI), 
 it does its job and unblocks the button.

 This is where help is needed. How is it fundamentally correct 
 to do this in D, and how to do it in general with dlangui?


Let's even look at the example of downloading a file.

Here in my application I have a button and its click event. I can 
set it to work on click. But I do not know how to do it so that 
the GUI does not block for the duration of the work. I do as in 
Ali's example, but the GUI is still blocked:
```d
auto btn = new Button("Run"d);
btn.click = delegate(Widget src) {
     auto request = Request("dlang.org");
     auto downloadTask = task!download(&request);
     downloadTask.executeInNewThread;
     return true;
};
```
How to do it right? Not only for downloading, but for any 
background work in dlangui? Maybe someone knows?
Jul 08 2022
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 7/8/22 06:32, Bagomot wrote:


 I do as in Ali's example, but
 the GUI is still blocked:


I don't know exactly why but something looks suspicious.


 ```d
 auto btn = new Button("Run"d);
 btn.click = delegate(Widget src) {
      auto request = Request("dlang.org");


That is a local variable which will die after the return statement...


      auto downloadTask = task!download(&request);


... and that pointer will be invalid, still pointing on function call stack.

At least, I would move the 'auto request' line up, next to 'btn' so that 
it lives long and that you can get the 'result' later on.

But then, you need access to 'task' as well so that you can ensure it 
has finished with downloadTask.yieldForce().


      downloadTask.executeInNewThread;
      return true;
 };


Aside: What does the return value 'true' mean?

I am not sure whether my explicit way of starting a thread (albeit with 
std.parallelism) is the right way here. As you've been suspecting, 
dlangui may provide async execution. (?)

Ali
Jul 08 2022
parent reply Bagomot <bagomot gmail.com> writes:
On Friday, 8 July 2022 at 21:47:32 UTC, Ali Çehreli wrote:

 Aside: What does the return value 'true' mean?


Docs says:

 Handler of click signal should return true if signal is 
 processed.



 I am not sure whether my explicit way of starting a thread 
 (albeit with std.parallelism) is the right way here. As you've 
 been suspecting, dlangui may provide async execution. (?)


Docs says:

 Non thread safe - all UI operations should be preformed in 
 single thread.


But the library contains this method for window: 
'executeInUiThread(void delegate() runnable)'.

Unfortunately, there are no examples of how to use this.
Jul 09 2022
parent reply Bagomot <bagomot gmail.com> writes:
Based on Thread, I managed to do what I intended. I have not yet 
been able to figure out how to do the same through the Task.

Here in the example, when you click on the Start button, a worker 
is launched that updates the progress bar.

```d
import dlangui;
import core.thread;
import std.conv;

class MyWorker : Thread {
     private {
         bool _status;
         int _counter;
         Window _window;
     }

     this(Window window) {
         super(&run);
         counter = 1;
         _window = window;
     }

     public void run() {
         _status = true;

         while (_status) {
             _window.executeInUiThread(() {
                 
_window.mainWidget.childById!ProgressBarWidget("progressbar")
                     .progress(_counter * 10);
             });
             _window.update(true);
             _counter++;
             Thread.sleep(3000.msecs);
         }
     }

     public void stop() {
         _status = false;
         _counter = 1;
     }

     public bool status() {
         return _status;
     }
}

class MyWidget : VerticalLayout {
     private {
         Button _btnStart;
         Button _btnStop;
         ProgressBarWidget _progressBar;
         MyWorker _worker;
     }

     this(string id, Window window) {
         super(id);
         styleId = id;

         _worker = new MyWorker(window);

         _btnStart = new Button("btn_start", "Start"d);
         _btnStop = new Button("btn_stop", "Stop"d);
         _progressBar = new ProgressBarWidget("progressbar");
         _progressBar.animationInterval(100);

         _btnStart.click = delegate(Widget src) {

             if (!_worker.status)
                 _worker.start;

             return true;
         };

         _btnStop.click = delegate(Widget src) {
             if (_worker.status)
                 _worker.stop;

             return true;
         };

         addChild(_btnStart);
         addChild(_btnStop);
         addChild(_progressBar);
     }
}

mixin APP_ENTRY_POINT;

extern (C) int UIAppMain(string[] args) {
	
     auto window = Platform.instance.createWindow("Test", null, 
WindowFlag.Resizable, 480, 480);

     auto root = new MyWidget("my_widget");

     window.mainWidget = root;
     window.show();

     return Platform.instance.enterMessageLoop();
}
```

Do you have any suggestions how to do it more beautifully and not 
through the Thread? In particular, I would like to see the 
comments of the dlangui developers ( GrimMaple).
Jul 10 2022
parent reply evilrat <evilrat666 gmail.com> writes:
On Sunday, 10 July 2022 at 09:15:59 UTC, Bagomot wrote:

 Based on Thread, I managed to do what I intended. I have not 
 yet been able to figure out how to do the same through the Task.

 Here in the example, when you click on the Start button, a 
 worker is launched that updates the progress bar.

 ...

 Do you have any suggestions how to do it more beautifully and 
 not through the Thread? In particular, I would like to see the 
 comments of the dlangui developers ( GrimMaple).


Since you've already used `executeInUiThread` you can now just 
move your Thread.run() implementation to a free function and run 
it using task instead.
`executeInUiThread` will ensure that passed delegate will be 
called in UI thread making it safe to call from another thread or 
task.

Not sure about default widgets, but in some cases for your custom 
properties you might need to `invalidate()` widget to trigger 
redraw.

Anyway you will need a reference to your widget to update 
progress bar value, options are:
- It can be done in place where you start your task (e.g. inside 
button click event, delegates can access call site scope)
- get widget reference directly by using 
`getWidgetById("WIDGET_ID")` (or how it was called, though you 
will need to assign ID to your progress bar and remember it)
- use signal-slot mechanism (you still need widget reference to 
bind events)


btw, have you tried googling this forum? I think there was exact 
same question with curl & dlangui a year or two ago, maybe it has 
an answer or at least tips you need?

Anyway for curl notify progress delegate you can just do it 
inside that delegate using `executeInUiThread` there.

like in example here 
https://dlang.org/library/std/net/curl/curl.on_progress.html
adjusting to somewhat above will now look like (not tested)

```d
// download button click event
void downloadbutton_Click()
{
   auto progressBar = (get progress bar reference);

   Curl curl;
   curl.initialize();
   curl.set(CurlOption.url, "http://dlang.org");
   curl.onProgress = delegate int(size_t dltotal, size_t dlnow, 
size_t ultotal, size_t ulnow)
   {
     // even though onProgress can be run in another thread this 
delegate will be run on next UI tick
     executeInUiThread( (){ progressBar.setProgress(dlnow/dltotal 
* 100); } );
     return 0;
   };
   curl.perform();
}
```
Jul 11 2022
parent reply Bagomot <bagomot gmail.com> writes:
On Monday, 11 July 2022 at 09:32:46 UTC, evilrat wrote:

   curl.onProgress = delegate int(size_t dltotal, size_t dlnow, 
 size_t ultotal, size_t ulnow)
   {
     // even though onProgress can be run in another thread this 
 delegate will be run on next UI tick
     executeInUiThread( (){ 
 progressBar.setProgress(dlnow/dltotal * 100); } );
     return 0;
   };
   curl.perform();
 }
 ```


Yes. Thanks, I did just that for the download.

I now have a couple more questions about `Task`:
1) How to run a non-static class method through `task`?
2) How to use `taskPool` to run a series of tasks of the same 
type (from question 1)?
Jul 12 2022
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 7/12/22 11:47, Bagomot wrote:


 I now have a couple more questions about `Task`:
 1) How to run a non-static class method through `task`?
 2) How to use `taskPool` to run a series of tasks of the same type (from
 question 1)?


As a friendly reminder, these questions could be more useful in a 
separate forum thread. :)

import std.stdio;
import std.parallelism;
import std.algorithm;
import std.range;

interface Animal {
   string song();
}

class Dog : Animal {
   string voice_;

   this(string voice) {
     this.voice_ = voice;
   }

   string song() {
     return voice_ ~ " " ~ voice_;
   }
}

void main() {
   auto voices = [ "hav", "woof", "bark", "gav", "grrr" ];
   auto dogs = voices.map!(voice => new Dog(voice)).array;

   // No need to specify this; just being silly...
   const workerCount = totalCPUs + 7;

   auto tp = new TaskPool(workerCount);
   scope (exit) tp.finish();

   // a) Classic foreach loop
   foreach (dog; tp.parallel(dogs)) {
     writeln(dog.song);
   }

   // b) Adding individual tasks (could be a foreach loop)
   dogs.each!(dog => tp.put(task!(d => writeln(d.song))(dog)));
}

Ali
Jul 12 2022
parent reply Bagomot <bagomot gmail.com> writes:
On Tuesday, 12 July 2022 at 19:25:42 UTC, Ali Çehreli wrote:

 On 7/12/22 11:47, Bagomot wrote:


 I now have a couple more questions about `Task`:
 1) How to run a non-static class method through `task`?
 2) How to use `taskPool` to run a series of tasks of the same


 type (from

 question 1)?


 As a friendly reminder, these questions could be more useful in 
 a separate forum thread. :)

 import std.stdio;
 import std.parallelism;
 import std.algorithm;
 import std.range;

 interface Animal {
   string song();
 }

 class Dog : Animal {
   string voice_;

   this(string voice) {
     this.voice_ = voice;
   }

   string song() {
     return voice_ ~ " " ~ voice_;
   }
 }

 void main() {
   auto voices = [ "hav", "woof", "bark", "gav", "grrr" ];
   auto dogs = voices.map!(voice => new Dog(voice)).array;

   // No need to specify this; just being silly...
   const workerCount = totalCPUs + 7;

   auto tp = new TaskPool(workerCount);
   scope (exit) tp.finish();

   // a) Classic foreach loop
   foreach (dog; tp.parallel(dogs)) {
     writeln(dog.song);
   }

   // b) Adding individual tasks (could be a foreach loop)
   dogs.each!(dog => tp.put(task!(d => writeln(d.song))(dog)));
 }

 Ali


Thank you! I will take into account the advice and will not 
inflate the forum thread.
Jul 12 2022
parent reply Bagomot <bagomot gmail.com> writes:
I had this question: how can I get the value from the `task`, 
like how I can get from the `spawnLinked`(`ownerTid.send` and 
`receive`)?

I'm using a `taskPool` through `arr.parallel`, but it became 
necessary to collect the progress from all tasks into one 
variable.

In this case, I can't use `spawnLinked` because the worker is 
mutable, I get the "Aliases to mutable thread-local data not 
allowed" error.

Also, I could make a new `Thread` for each task, but I think this 
is a bad idea. Also, I don't know how to send messages from child 
thread to parent thread.
Jul 21 2022
parent reply frame <frame86 live.com> writes:
On Thursday, 21 July 2022 at 13:27:49 UTC, Bagomot wrote:

 I had this question: how can I get the value from the `task`, 
 like how I can get from the `spawnLinked`(`ownerTid.send` and 
 `receive`)?

 I'm using a `taskPool` through `arr.parallel`, but it became 
 necessary to collect the progress from all tasks into one 
 variable.

 In this case, I can't use `spawnLinked` because the worker is 
 mutable, I get the "Aliases to mutable thread-local data not 
 allowed" error.



The module creators want you to prevent from non-threadsafe 
actions. Merging into one variable can be such a non-threadsafe 
action, but it has not to be.

For example, if you have an array of fixed length or an already 
allocated one and each worker thread uses a fixed index to write 
into this result array variable then this operation may be 
threadsafe because each thread writes in different positions in 
the memory and the variable itself doesn't change.

If you are depending to overwrite an value instead, the operation 
is not considered threadsafe. For example just adding 1 + 2. 
Adding a value to an associative array is also not thread safe 
(it may re-allocate or re-index it's data)

We have basic tools to achieve this task. One is to lock actions 
with a synchronized block so each thread need to wait the other 
thread to complete before it can continue to execute the 
particular code position - and the other one are atomic 
operations that provide a better performance for simple 
operations because they are lock-free.

Why it's needed to know this? Because you may run into situations 
where have to deal with it, even the compiler keeps silent about.

https://dlang.org/spec/statement.html#synchronized-statement

```d
// parent:
int result;
// ...
// worker/parallel:
// each thread can only run this code if no other thread is 
currently running this code section => the OS is locking, this 
costs time
synchronized {
    result += 1;
}
```

https://dlang.org/phobos/core_atomic.html#.atomicOp

Usage of `atomaticOp` is simple, eg.

```d
// parent:
shared int result;
// ...
// worker/parallel:
// lock free, D-runtime ensures for a threadsafe operation
atomicOp!"+="(result, 1);
```

Looking on `atomicOp` you will see it want you to use a `shared` 
variable. And this is an easy way to make the compiler happy if 
you have mutual data. If you know what you are doing (not making 
thread unsafe-operations) then it's fine to just cast your data 
to `shared` that now can be sent or received.

```d
tid.send(cast(shared)mutualData);
```

A better approach is always to avoid such things completely and 
design your workflow to send and receive only simple data types 
(or immutable ones, which is also fine).


 Also, I could make a new `Thread` for each task, but I think 
 this is a bad idea. Also, I don't know how to send messages 
 from child thread to parent thread.


The messagebox system from `std.concurrency` should also work 
with parallelism tools, meaning inside a worker thread use 
`std.concurrency.ownerTid` to get the parent Tid.

Looking for your task you might want to use a 
`WorkerLocalStorage` solution instead, look at the example:
https://dlang.org/phobos/std_parallelism.html#.TaskPool.WorkerLocalStorage
Jul 21 2022
parent Bagomot <bagomot gmail.com> writes:
On Thursday, 21 July 2022 at 17:26:07 UTC, frame wrote:

 The module creators want you to prevent from non-threadsafe 
 actions. Merging into one variable can be such a non-threadsafe 
 action, but it has not to be.

 For example, if you have an array of fixed length or an already 
 allocated one and each worker thread uses a fixed index to 
 write into this result array variable then this operation may 
 be threadsafe because each thread writes in different positions 
 in the memory and the variable itself doesn't change.

 If you are depending to overwrite an value instead, the 
 operation is not considered threadsafe. For example just adding 
 1 + 2. Adding a value to an associative array is also not 
 thread safe (it may re-allocate or re-index it's data)

 We have basic tools to achieve this task. One is to lock 
 actions with a synchronized block so each thread need to wait 
 the other thread to complete before it can continue to execute 
 the particular code position - and the other one are atomic 
 operations that provide a better performance for simple 
 operations because they are lock-free.

 Why it's needed to know this? Because you may run into 
 situations where have to deal with it, even the compiler keeps 
 silent about.

 https://dlang.org/spec/statement.html#synchronized-statement


I'm considering thread safety.


 Looking for your task you might want to use a 
 `WorkerLocalStorage` solution instead, look at the example:
 https://dlang.org/phobos/std_parallelism.html#.TaskPool.WorkerLocalStorage


This looks interesting, thanks, I'll give it a try.
Jul 21 2022