www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Process in parallel and output result to stdout theread-safely

reply Dr.No <jckj33 gmail.com> writes:
so I'im doing an expansive operation with a file, to try speed 
up, i switch to using parallel but keeping in the otuput printing 
thread-safe. But for some reason, even using synchonized, it 
doesn't work as expected, for example, it output multiples 
results on same time, as in the example below.

the code:


	stdout.flush();
	foreach(string fn; parallel(files))
	{
		auto res = doSomething(fn);
		synchronized
		{
		   stdout.writefln("outjson = %s", res.serializeToJson);
		   stdout.flush();
		}

     }



the expeced output is like that (one per line):


 outjson = {"barCode":"XXXX1","ade":"1"}
 outjson = {"barCode":"XXXX2","ade":"2"}
 outjson = {"barCode":"XXXX3","ade":"3"}


// and so on...

But it in the middle of output, I got output like this:


 outjson = {"barCode":"XXXX20","ade":"20"}♪◙outjson = 
 {"barCode":"XXXXX21","ade":"21"}


also there's that extra ♪◙ character. Thos sounds memory 
violation somewhere.
This only happens when using parallel. Any guess what's possibily 
happeing?
Sep 03 2018
next sibling parent Dr.No <jckj33 gmail.com> writes:
Does anyone have some tips to try trace the error with debug or 
so?
I haven't fixed this issue yet... any help is very appreciated
Sep 08 2018
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 09/03/2018 08:13 PM, Dr.No wrote:

 But it in the middle of output, I got output like this:
 

 outjson = {"barCode":"XXXX20","ade":"20"}♪◙outjson = 
 {"barCode":"XXXXX21","ade":"21"}


 
 also there's that extra ♪◙ character. Thos sounds memory violation 
 somewhere.
 This only happens when using parallel. Any guess what's possibily happeing?


If that only ever happens at line breaks, then that doesn't necessarily 
look like memory corruption to me.

I looked around a bit and found code page 437 [1]. It has those 
characters at 0xD and 0xA which is where ASCII has \r and \n. So it 
might be that code page 437 is used when displaying that particular 
entry. Why that would happen, I have no idea.

To get better help, you should post a complete test case that people can 
just copy/paste. That includes imports, a `main` function, and the 
command you use to compile. Also add information about your environment 
(OS, compiler version).


[1] https://en.wikipedia.org/wiki/Code_page_437
Sep 08 2018
parent reply Dr.No <jckj33 gmail.com> writes:
On Saturday, 8 September 2018 at 14:26:45 UTC, ag0aep6g wrote:

 On 09/03/2018 08:13 PM, Dr.No wrote:

 But it in the middle of output, I got output like this:
 

 outjson = {"barCode":"XXXX20","ade":"20"}♪◙outjson = 
 {"barCode":"XXXXX21","ade":"21"}


 
 also there's that extra ♪◙ character. Thos sounds memory 
 violation somewhere.
 This only happens when using parallel. Any guess what's 
 possibily happeing?


 If that only ever happens at line breaks, then that doesn't 
 necessarily look like memory corruption to me.


Yes, it does only happens at line breaks. I hadn't realized that 
until you mentioned. It does gets in place of \r and \n, that's 
why there's all in one line when this happens. Thankfor for the 
notice.


 I looked around a bit and found code page 437 [1]. It has those 
 characters at 0xD and 0xA which is where ASCII has \r and \n. 
 So it might be that code page 437 is used when displaying that 
 particular entry. Why that would happen, I have no idea.


So I guessed that something changed the console code page. I've 
tried to call call chcp 65001 at program's start up. Not worked. 
Tried inside the main loop. Also not worked.


 To get better help, you should post a complete test case that 
 people can just copy/paste. That includes imports, a `main` 
 function, and the command you use to compile. Also add 
 information about your environment (OS, compiler version).


I made a reduced version where you can do just dmd -run hello.d. 
The application does convert alot of PDF files to text then do 
some parsing (I've removed this not needed part). In my code, at 
iteration 28 I got those character (♪◙), which is the error. I 
hope you can reproduce this error on your machine as well. I got 
some PDF files and duplicated them just for testing purpose. If 
for some reason you didn't get the same error immediately, try 
duplicate the PDFs and try again. I believe this can reproduce 
the erro.


Here's the resources to compile the application:

source code: https://pastebin.com/RwrUikQS
PDF files + pdf to text application: 
https://drive.google.com/file/d/1uKjJX4pQIEWVK4vujUsm0ln2yHS7z5ZZ/view?usp=sharing
OS: Windows 10 64-bit
compiler: DMD32 D Compiler v2.080.0 (same issue happens on ldc)
command line: dmd -run hello.d

Thank you very much for your time.
Sep 10 2018
parent Dr.No <jckj33 gmail.com> writes:
On Monday, 10 September 2018 at 20:30:52 UTC, Dr.No wrote:

 On Saturday, 8 September 2018 at 14:26:45 UTC, ag0aep6g wrote:

 [...]


 Yes, it does only happens at line breaks. I hadn't realized 
 that until you mentioned. It does gets in place of \r and \n, 
 that's why there's all in one line when this happens. Thankfor 
 for the notice.

 [...]


I noticied this happens with everything I print in the loop, even 
using C's printf:



	__gshared Mutex m;
	m = new Mutex();
	foreach(string fn; parallel(files))
	{
		try
		{
			auto res = f(fn);
			synchronized(m)
			{
				import std.c.stdio : 				printf;
				printf("hello!\n");
			}

 [...]



Output:

https://imgur.com/a/Mq9X4c3
Sep 11 2018