www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - vibe.d: How to get the conent of a file upload ?

reply wjoe <invalid example.com> writes:
I found this [1] but unfortunately the post this refers to is a 
dead link and the content, unfortunately, didn't tell me anything 
that I didn't already find in the docs.

What I can get from the form is the form fields with content, the 
field name for the file upload and the file name. But the file 
name is useless to me because I need the file content.

Where is it stored ?

[1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/
Sep 17 2020
parent reply WebFreak001 <d.forum webfreak.org> writes:
On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:

 I found this [1] but unfortunately the post this refers to is a 
 dead link and the content, unfortunately, didn't tell me 
 anything that I didn't already find in the docs.

 What I can get from the form is the form fields with content, 
 the field name for the file upload and the file name. But the 
 file name is useless to me because I need the file content.

 Where is it stored ?

 [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/


hi, you can access HTTPServerRequest.files which contains the 
uploaded file.

Example in real code: 
https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159

Documentation: https://vibed.org/api/vibe.inet.webform/FilePart

Note: the file is only downloaded from the client / stored on 
disk once you access the files or the form property, though this 
isn't documented.

I don't believe the temp file behavior is customizable as it is 
hardcoded here to write to a temporary file (which is called on 
form parsing): 
https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311

However if you really wanted to (but I'd advise against it) you 
could parse the form data from the HTTPServerRequest.bodyReader 
yourself
Sep 17 2020
next sibling parent reply wjoe <invalid example.com> writes:
On Thursday, 17 September 2020 at 16:32:55 UTC, WebFreak001 wrote:

 On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:

 I found this [1] but unfortunately the post this refers to is 
 a dead link and the content, unfortunately, didn't tell me 
 anything that I didn't already find in the docs.

 What I can get from the form is the form fields with content, 
 the field name for the file upload and the file name. But the 
 file name is useless to me because I need the file content.

 Where is it stored ?

 [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/


 hi, you can access HTTPServerRequest.files which contains the 
 uploaded file.

 Example in real code: 
 https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159

 Documentation: https://vibed.org/api/vibe.inet.webform/FilePart

 Note: the file is only downloaded from the client / stored on 
 disk once you access the files or the form property, though 
 this isn't documented.


Every post or example I found copies the file, like your code 
does, too. Why is that ? The content of the file upload is 
embedded in the form data. There's no need for temporary files or 
copying at all.

On top of that, if I upload a file to a server which is on a 
different PC on a different file system, how am I supposed to 
read the file from disk on a remote file system ?

This makes no sense.

What I want is something like this:


 ~$ cat /var/log/temperatures.log

 temp_1=22;temp_2=28
 temp_1=21;temp_2=25




 ~$ curl -F "temperature_log= /var/log/temperatures.log" 
 192.168.1.1:20222/temperature_upload



 ~$ nc -l 127.0.0.1 20222

 POST /temperature_upload HTTP/1.1
 Host: 192.168.1.1:20222
 User-Agent: curl/7.72.0
 Accept: */*
 Content-Length: 245
 Content-Type: multipart/form-data; 
 boundary=------------------------c73c71472ff9e7d5

 --------------------------c73c71472ff9e7d5
 Content-Disposition: form-data; name="temperature_log"; 
 filename="/var/log/temperatures.log"
 Content-Type: application/octet-stream

 temp_1=22;temp_2=28
 temp_1=21;temp_2=25

 --------------------------c73c71472ff9e7d5--



void upload(HttpRequest.. req, blah)
{
    auto file = "temperature_log" in req.files;
    if (file) {
       string temp_data_raw = file.data;
       assert ("temp_1=22;temp_2=28\ntemp_1=21;temp_2=25" == 
temp_data_raw);
    }
}
Sep 17 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/17/20 1:08 PM, wjoe wrote:

 Every post or example I found copies the file, like your code does, too. 
 Why is that ? The content of the file upload is embedded in the form 
 data. There's no need for temporary files or copying at all.
 
 On top of that, if I upload a file to a server which is on a different 
 PC on a different file system, how am I supposed to read the file from 
 disk on a remote file system ?
 
 This makes no sense.
 
 What I want is something like this:
 

 ~$ cat /var/log/temperatures.log

 temp_1=22;temp_2=28
 temp_1=21;temp_2=25


 
 

 ~$ curl -F "temperature_log= /var/log/temperatures.log" 
 192.168.1.1:20222/temperature_upload


 

 ~$ nc -l 127.0.0.1 20222

 POST /temperature_upload HTTP/1.1
 Host: 192.168.1.1:20222
 User-Agent: curl/7.72.0
 Accept: */*
 Content-Length: 245
 Content-Type: multipart/form-data; 
 boundary=------------------------c73c71472ff9e7d5

 --------------------------c73c71472ff9e7d5
 Content-Disposition: form-data; name="temperature_log"; 
 filename="/var/log/temperatures.log"
 Content-Type: application/octet-stream

 temp_1=22;temp_2=28
 temp_1=21;temp_2=25

 --------------------------c73c71472ff9e7d5--


 
 
 void upload(HttpRequest.. req, blah)
 {
     auto file = "temperature_log" in req.files;
     if (file) {
        string temp_data_raw = file.data;
        assert ("temp_1=22;temp_2=28\ntemp_1=21;temp_2=25" == 
 temp_data_raw);
     }
 }
 


the `files` property actually does the processing only when you call it.

If you access the `bodyReader` property directly, you can process that 
data yourself. You can even register a web interface function with an 
`InputStream` parameter type, and it will be bound to the body data.

I've done this with my REST interface, though that's not form data.

That's not a great API, though. I would love to see vibe.d allow a 
direct call to vibe.inet.webform.parseFormData with a specific handler 
for files and form data.

I think you can agree that it's not feasible to store arbitrary sized 
file contents in memory. But it certainly can provide a mechanism to 
handle it as it's read.

-Steve
Sep 17 2020
parent reply aberba <karabutaworld gmail.com> writes:
On Thursday, 17 September 2020 at 21:57:37 UTC, Steven 
Schveighoffer wrote:

 On 9/17/20 1:08 PM, wjoe wrote:

 [...]


 the `files` property actually does the processing only when you 
 call it.

 If you access the `bodyReader` property directly, you can 
 process that data yourself. You can even register a web 
 interface function with an `InputStream` parameter type, and it 
 will be bound to the body data.


I'm not sure I understand how to do this and parser the files in 
memory.


 I've done this with my REST interface, though that's not form 
 data.

 That's not a great API, though. I would love to see vibe.d 
 allow a direct call to vibe.inet.webform.parseFormData with a 
 specific handler for files and form data.


Can we file an issue for this? Because I'm very interested in 
having this resolved


 I think you can agree that it's not feasible to store arbitrary 
 sized file contents in memory. But it certainly can provide a 
 mechanism to handle it as it's read.

 -Steve


There's potential to results in out of memory condition. Its a 
know issues. A complete parser (like multer in nodejs) allowance 
you to limit file size as well for error handling.


 I've done this with my REST interface, though that's not form 
 data.


Can you share your code for this?
Sep 17 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/17/20 6:13 PM, aberba wrote:

 On Thursday, 17 September 2020 at 21:57:37 UTC, Steven Schveighoffer wrote:

 On 9/17/20 1:08 PM, wjoe wrote:

 [...]


 the `files` property actually does the processing only when you call it.

 If you access the `bodyReader` property directly, you can process that 
 data yourself. You can even register a web interface function with an 
 `InputStream` parameter type, and it will be bound to the body data.


 
 I'm not sure I understand how to do this and parser the files in memory.


So an HTTP request with form data will come in with the headers parsed, 
but the data is still on the network stream.

The first time you access `files`, it processes the stream data, and 
splits it into form data and file data, saves the files, and then gives 
you back the file dictionary so you can use them.

If instead, you access `bodyReader`, YOU get to process the form data 
and file data.


 I've done this with my REST interface, though that's not form data.

 That's not a great API, though. I would love to see vibe.d allow a 
 direct call to vibe.inet.webform.parseFormData with a specific handler 
 for files and form data.


 Can we file an issue for this? Because I'm very interested in having 
 this resolved


You can always file an issue! https://github.com/vibe-d/vibe.d/issues

There may already be one in there.


 There's potential to results in out of memory condition. Its a know 
 issues. A complete parser (like multer in nodejs) allowance you to limit 
 file size as well for error handling.


Meh, this is D :) we should be able to just process the data and do 
whatever we want with it. What I would like to see is vibe provide the 
parsing of form data, and just give me the data as it comes (kind of 
like a SAX parser). Maybe just a property in the HTTPServerRequest that 
I can set that says "use this callback when you get Form File data".


 I've done this with my REST interface, though that's not form data.


 
 Can you share your code for this?


Heh, this is not form data, it's just file data, raw on the stream. So I 
have a function like:

```
class FileRestImpl
{
      path(":cat/:id/:uuid/upload")
      getAuth
     void postUpload(HTTPServerResponse res, string _cat, int _id, 
string _uuid, InputStream stream, Nullable!string md5sum, NRMAuthInfo 
_authInfo)
     {
         ...
     }
}
```

You can see, I take an InputStream as a parameter -- the data comes in 
there. I just read it and save it to a file (in the correct location) 
anyway, verifying the md5sum is valid.

-Steve
Sep 17 2020
parent reply wjoe <invalid example.com> writes:
On Thursday, 17 September 2020 at 22:33:46 UTC, Steven 
Schveighoffer wrote:

 On 9/17/20 6:13 PM, aberba wrote:

 On Thursday, 17 September 2020 at 21:57:37 UTC, Steven 
 Schveighoffer wrote:

 On 9/17/20 1:08 PM, wjoe wrote:

 [...]


 the `files` property actually does the processing only when 
 you call it.

 If you access the `bodyReader` property directly, you can 
 process that data yourself. You can even register a web 
 interface function with an `InputStream` parameter type, and 
 it will be bound to the body data.


 
 I'm not sure I understand how to do this and parser the files 
 in memory.


 So an HTTP request with form data will come in with the headers 
 parsed, but the data is still on the network stream.

 The first time you access `files`, it processes the stream 
 data, and splits it into form data and file data, saves the 
 files, and then gives you back the file dictionary so you can 
 use them.

 If instead, you access `bodyReader`, YOU get to process the 
 form data and file data.


 I've done this with my REST interface, though that's not form 
 data.

 That's not a great API, though. I would love to see vibe.d 
 allow a direct call to vibe.inet.webform.parseFormData with a 
 specific handler for files and form data.


 Can we file an issue for this? Because I'm very interested in 
 having this resolved


 You can always file an issue! 
 https://github.com/vibe-d/vibe.d/issues

 There may already be one in there.


 There's potential to results in out of memory condition. Its a 
 know issues. A complete parser (like multer in nodejs) 
 allowance you to limit file size as well for error handling.


 Meh, this is D :) we should be able to just process the data 
 and do whatever we want with it. What I would like to see is 
 vibe provide the parsing of form data, and just give me the 
 data as it comes (kind of like a SAX parser). Maybe just a 
 property in the HTTPServerRequest that I can set that says "use 
 this callback when you get Form File data".


 I've done this with my REST interface, though that's not form 
 data.


 
 Can you share your code for this?


 Heh, this is not form data, it's just file data, raw on the 
 stream. So I have a function like:

 ```
 class FileRestImpl
 {
      path(":cat/:id/:uuid/upload")
      getAuth
     void postUpload(HTTPServerResponse res, string _cat, int 
 _id, string _uuid, InputStream stream, Nullable!string md5sum, 
 NRMAuthInfo _authInfo)
     {
         ...
     }
 }
 ```

 You can see, I take an InputStream as a parameter -- the data 
 comes in there. I just read it and save it to a file (in the 
 correct location) anyway, verifying the md5sum is valid.

 -Steve


Not a reply to this post in particular but to all the ones I've 
read so far.

If I understand correctly. Vibe parses the form data and writes 
all files to disk. Where to ?
Can I configure it ? I don't want libraries to just write data to 
my file systems without me setting this up. Nowhere did I find 
this behavior described in the docs.
And if not, how is data processed with a 10mb file upload 
followed by a few number fields ?
It needs to read all of the file data to get to the other data 
fields, doesn't it ?

I'm sorry this is completely counter intuitive. I can understand 
the memory/security risks and all but I have no intention to 
hack, DOS or however else disrupt my private server in my private 
network with garbage data. I just want to get the data in a 
byte[].

Why does the lib not simply reject files that are unreasonably 
(configurable) big ?
Writing files to disk in order to then needing to copy them 
somewhere else or to read them back into memory for further 
processing sounds, above all else, incredibly inefficient.
I might not even want to keep the file and drop it.

I guess it's no problem to parse the data myself, but then what's 
the point in using a framework ?

Are there other frameworks besides vibe that can do what I want ?

I'm sorry for the rant, developing this kind of software is a 
pain in the drain and stresses me out to no end. It sucked hard 
in the past with php and decades later with python, ruby, D, you 
name it it still sucks ;)

Still, thanks a lot for all the replies everybody. Very much 
appreciated :)
Sep 17 2020
next sibling parent reply Atwork <fwafwa fwfafa.com> writes:
On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:

 And if not, how is data processed with a 10mb file upload 
 followed by a few number fields ?
 It needs to read all of the file data to get to the other data 
 fields, doesn't it ?


Yes and no
Sep 18 2020
parent wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 11:44:39 UTC, Atwork wrote:

 On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:

 And if not, how is data processed with a 10mb file upload 
 followed by a few number fields ?
 It needs to read all of the file data to get to the other data 
 fields, doesn't it ?


 Yes and no


Consider this:


 ~$ curl -F "temperature_log= /var/log/temperatures.log" -F 
 "field1=a" -F "field2+foo" 192.168.1.1:20222/temperature_upload



 ~$ nc -l 127.0.0.1 20222

 POST /temperature_upload HTTP/1.1
 Host: 192.168.1.1:20222
 User-Agent: curl/7.72.0
 Accept: */*
 Content-Length: 10486005
 Content-Type: multipart/form-data; 
 boundary=------------------------c73c71472ff9e7d5

 --------------------------c73c71472ff9e7d5
 Content-Disposition: form-data; name="temperature_log"; 
 filename="/var/log/temperatures.log"
 Content-Type: application/octet-stream

 temp_1=22;temp_2=28
 temp_1=21;temp_2=25

 [ ... 10 MB of data ... ]

 --------------------------c73c71472ff9e7d5
 Content-Disposition: form-data; name="field1"

 a
 --------------------------c73c71472ff9e7d5
 Content-Disposition: form-data; name="field2"

 foo
 --------------------------c73c71472ff9e7d5--


How is vibe going to extract field1 and field2 without processing 
the temperature_log field ?
Sep 18 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/17/20 8:07 PM, wjoe wrote:

 
 Not a reply to this post in particular but to all the ones I've read so 
 far.
 
 If I understand correctly. Vibe parses the form data and writes all 
 files to disk. Where to ?


See the code here: 
https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311


 Can I configure it ? I don't want libraries to just write data to my 
 file systems without me setting this up. Nowhere did I find this 
 behavior described in the docs.


No, not at the moment. Which is why I was saying, it could be an 
enhancement request to vibe.


 And if not, how is data processed with a 10mb file upload followed by a 
 few number fields ?


All the data is processed before the accessor to the form data or the 
file data. It HAS to be this way, as the data is still on the incoming 
network socket.


 It needs to read all of the file data to get to the other data fields, 
 doesn't it ?


Yes


 
 I'm sorry this is completely counter intuitive. I can understand the 
 memory/security risks and all but I have no intention to hack, DOS or 
 however else disrupt my private server in my private network with 
 garbage data. I just want to get the data in a byte[].


Again, enhancement request needed. The code currently is hard-coded to 
write to disk.


 
 Why does the lib not simply reject files that are unreasonably 
 (configurable) big ?


If you had 1000 requests being processed simultaneously, and each of 
those requests provided 10MB files, then you now need potentially 10GB 
of RAM to hold those requests. This doesn't scale when the application 
is unknown to vibe.

But again, solved with an enhancement that allows you to process the 
data in your code. I'll file the enhancement request for you, as I think 
it's a nice addition.


 Writing files to disk in order to then needing to copy them somewhere 
 else or to read them back into memory for further processing sounds, 
 above all else, incredibly inefficient.


Agreed. In my case, it was an actual copy, as the location of the stored 
data was on a different filesystem than the temporary files.


 I might not even want to keep the file and drop it.


Yep, it's a waste in that case.


 
 I guess it's no problem to parse the data myself, but then what's the 
 point in using a framework ?


Agreed.


 Are there other frameworks besides vibe that can do what I want ?


In D, I'm not sure what the other frameworks do. I believe there are 
others if you search on code.dlang.org, you should be able to find some.


 I'm sorry for the rant, developing this kind of software is a pain in 
 the drain and stresses me out to no end. It sucked hard in the past with 
 php and decades later with python, ruby, D, you name it it still sucks ;)


web development sucks in general ;) Yet, we all still do it.

-Steve
Sep 18 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/18/20 8:39 AM, Steven Schveighoffer wrote:

 But again, solved with an enhancement that allows you to process the 
 data in your code. I'll file the enhancement request for you, as I think 
 it's a nice addition.


https://github.com/vibe-d/vibe.d/issues/2478

-Steve
Sep 18 2020
parent reply wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 12:58:29 UTC, Steven 
Schveighoffer wrote:

 On 9/18/20 8:39 AM, Steven Schveighoffer wrote:

 But again, solved with an enhancement that allows you to 
 process the data in your code. I'll file the enhancement 
 request for you, as I think it's a nice addition.


 https://github.com/vibe-d/vibe.d/issues/2478

 -Steve


Awesome! Thanks a ton :)
Sep 18 2020
parent reply ikod <igor.khasilev gmail.com> writes:
On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:

 On Friday, 18 September 2020 at 12:58:29 UTC, Steven 
 Schveighoffer wrote:

 On 9/18/20 8:39 AM, Steven Schveighoffer wrote:

 But again, solved with an enhancement that allows you to 
 process the data in your code. I'll file the enhancement 
 request for you, as I think it's a nice addition.


 https://github.com/vibe-d/vibe.d/issues/2478

 -Steve


 Awesome! Thanks a ton :)


My preferable way to handle such thing is: convert incoming data 
into input range of immutable(ubyte)[] and let user to consume 
this range (and handle it as it wish - transform, store in memory 
as is, or dump to disk)
Sep 19 2020
parent reply ikod <igor.khasilev gmail.com> writes:
On Saturday, 19 September 2020 at 11:11:21 UTC, ikod wrote:

 On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:

 On Friday, 18 September 2020 at 12:58:29 UTC, Steven 
 Schveighoffer wrote:

 On 9/18/20 8:39 AM, Steven Schveighoffer wrote:

 But again, solved with an enhancement that allows you to 
 process the data in your code. I'll file the enhancement 
 request for you, as I think it's a nice addition.


 https://github.com/vibe-d/vibe.d/issues/2478

 -Steve


 Awesome! Thanks a ton :)


 My preferable way to handle such thing is: convert incoming 
 data into input range of immutable(ubyte)[] and let user to 
 consume this range (and handle it as it wish - transform, store 
 in memory as is, or dump to disk)


sorry, this looks exactly like it described in proposal (and this 
is how requests works).
Sep 19 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/19/20 7:15 AM, ikod wrote:

 On Saturday, 19 September 2020 at 11:11:21 UTC, ikod wrote:

 On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:

 On Friday, 18 September 2020 at 12:58:29 UTC, Steven Schveighoffer 
 wrote:

 On 9/18/20 8:39 AM, Steven Schveighoffer wrote:

 But again, solved with an enhancement that allows you to process 
 the data in your code. I'll file the enhancement request for you, 
 as I think it's a nice addition.


 https://github.com/vibe-d/vibe.d/issues/2478


 Awesome! Thanks a ton :)


 My preferable way to handle such thing is: convert incoming data into 
 input range of immutable(ubyte)[] and let user to consume this range 
 (and handle it as it wish - transform, store in memory as is, or dump 
 to disk)


 
 sorry, this looks exactly like it described in proposal (and this is how 
 requests works).
 


You mean for each file, right? Yeah, that is the proposal, though it 
uses Vibe's io type (necessary I think).

-Steve
Sep 19 2020
parent ikod <igor.khasilev gmail.com> writes:
On Saturday, 19 September 2020 at 18:56:20 UTC, Steven 
Schveighoffer wrote:

 On 9/19/20 7:15 AM, ikod wrote:

 On Saturday, 19 September 2020 at 11:11:21 UTC, ikod wrote:

 On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:







 My preferable way to handle such thing is: convert incoming 
 data into input range of immutable(ubyte)[] and let user to 
 consume this range (and handle it as it wish - transform, 
 store in memory as is, or dump to disk)


 
 sorry, this looks exactly like it described in proposal (and 
 this is how requests works).
 


 You mean for each file, right? Yeah, that is the proposal,


If user uploads several files through single form then yes, for 
each file. Maybe handler should receive range of input ranges and 
handle each as necessary.


 though it uses Vibe's io type (necessary I think).

 -Steve
Sep 19 2020
prev sibling parent wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 12:39:43 UTC, Steven 
Schveighoffer wrote:

 On 9/17/20 8:07 PM, wjoe wrote:

 [...]


 See the code here: 
 https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311


 [...]


 No, not at the moment. Which is why I was saying, it could be 
 an enhancement request to vibe.


 [...]


 All the data is processed before the accessor to the form data 
 or the file data. It HAS to be this way, as the data is still 
 on the incoming network socket.


 [...]


 Yes


 [...]


 Again, enhancement request needed. The code currently is 
 hard-coded to write to disk.


 [...]


 If you had 1000 requests being processed simultaneously, and 
 each of those requests provided 10MB files, then you now need 
 potentially 10GB of RAM to hold those requests. This doesn't 
 scale when the application is unknown to vibe.

 But again, solved with an enhancement that allows you to 
 process the data in your code. I'll file the enhancement 
 request for you, as I think it's a nice addition.


 [...]


 Agreed. In my case, it was an actual copy, as the location of 
 the stored data was on a different filesystem than the 
 temporary files.


 [...]


 Yep, it's a waste in that case.


 [...]


 Agreed.


 [...]


 In D, I'm not sure what the other frameworks do. I believe 
 there are others if you search on code.dlang.org, you should be 
 able to find some.


 [...]


 web development sucks in general ;) Yet, we all still do it.

 -Steve


I was a little tired yesterday when I read the replies. After a 
couple hours of sleep and reading them again it makes a lot more 
sense to me.

Also your suggestion about a direct call to 
vibe.inet.webform.parseFormData with a specific handler for files 
and form data is a lot better than access via byte[].

Thanks for your help. Very much appreciated :)
Sep 18 2020
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:

 On Thursday, 17 September 2020 at 22:33:46 UTC, Steven 
 Schveighoffer wrote:

 On 9/17/20 6:13 PM, aberba wrote:

 On Thursday, 17 September 2020 at 21:57:37 UTC, Steven 
 Schveighoffer wrote:

 On 9/17/20 1:08 PM, wjoe wrote:

 [...]


 the `files` property actually does the processing only when 
 you call it.

 If you access the `bodyReader` property directly, you can 
 process that data yourself. You can even register a web 
 interface function with an `InputStream` parameter type, and 
 it will be bound to the body data.


 
 I'm not sure I understand how to do this and parser the files 
 in memory.


 So an HTTP request with form data will come in with the 
 headers parsed, but the data is still on the network stream.

 The first time you access `files`, it processes the stream 
 data, and splits it into form data and file data, saves the 
 files, and then gives you back the file dictionary so you can 
 use them.

 If instead, you access `bodyReader`, YOU get to process the 
 form data and file data.


 I've done this with my REST interface, though that's not 
 form data.

 That's not a great API, though. I would love to see vibe.d 
 allow a direct call to vibe.inet.webform.parseFormData with 
 a specific handler for files and form data.


 Can we file an issue for this? Because I'm very interested in 
 having this resolved


 You can always file an issue! 
 https://github.com/vibe-d/vibe.d/issues

 There may already be one in there.


 There's potential to results in out of memory condition. Its 
 a know issues. A complete parser (like multer in nodejs) 
 allowance you to limit file size as well for error handling.


 Meh, this is D :) we should be able to just process the data 
 and do whatever we want with it. What I would like to see is 
 vibe provide the parsing of form data, and just give me the 
 data as it comes (kind of like a SAX parser). Maybe just a 
 property in the HTTPServerRequest that I can set that says 
 "use this callback when you get Form File data".


 I've done this with my REST interface, though that's not 
 form data.


 
 Can you share your code for this?


 Heh, this is not form data, it's just file data, raw on the 
 stream. So I have a function like:

 ```
 class FileRestImpl
 {
      path(":cat/:id/:uuid/upload")
      getAuth
     void postUpload(HTTPServerResponse res, string _cat, int 
 _id, string _uuid, InputStream stream, Nullable!string md5sum, 
 NRMAuthInfo _authInfo)
     {
         ...
     }
 }
 ```

 You can see, I take an InputStream as a parameter -- the data 
 comes in there. I just read it and save it to a file (in the 
 correct location) anyway, verifying the md5sum is valid.

 -Steve


 Not a reply to this post in particular but to all the ones I've 
 read so far.

 If I understand correctly. Vibe parses the form data and writes 
 all files to disk. Where to ?
 Can I configure it ? I don't want libraries to just write data 
 to my file systems without me setting this up. Nowhere did I 
 find this behavior described in the docs.
 And if not, how is data processed with a 10mb file upload 
 followed by a few number fields ?
 It needs to read all of the file data to get to the other data 
 fields, doesn't it ?

 I'm sorry this is completely counter intuitive. I can 
 understand the memory/security risks and all but I have no 
 intention to hack, DOS or however else disrupt my private 
 server in my private network with garbage data. I just want to 
 get the data in a byte[].


That's what I was trying to answer. When Steve said meh, he 
probably didn't get what I said. Probably its because of my typos.

This sort of convenience and productivity benefit is part of why 
I use Node.Js in the job when I need to get things done....and 
not D yet. There are several pieces and bits you can't write 
yourself when working on projects.

In this case you want to get the file(s) in memory...in the form 
of bytes (or buffer) and probably set a file size limit. Its all 
doable through a library but such a library doesn't exist in D 
yet. At least not that I know of.

Its why I mentioned that multer[1] in Node.Js able to do 
that...hence the advantage. Its built for the express 
framework...meaning such library can be built to work with 
vibe.d. Not everything can be built into vibe.d..and I think 
that'll even make it bloated for other uses case. Its need an 
ecosystem of third-party libraries.

In the case of the vibe.d form, data and files are handled using 
this implementation[2] so its a reference to such a form parser 
implementation...with support for a storage parameter for either 
a MemoryStore or SessionStore. Multer does it pretty cleanly.

1. Multer: https://www.npmjs.com/package/multer
2. 
https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d



 Why does the lib not simply reject files that are unreasonably 
 (configurable) big ?
 Writing files to disk in order to then needing to copy them 
 somewhere else or to read them back into memory for further 
 processing sounds, above all else, incredibly inefficient.
 I might not even want to keep the file and drop it.


Not implemented yet.


 I guess it's no problem to parse the data myself, but then 
 what's the point in using a framework ?


Vibe.d still lacks many things I personally need... there's 
simply not enough ecosystem third-party libraries.


 Are there other frameworks besides vibe that can do what I want 
 ?


I don't think so. Vibe.d is the one with most features. But Does 
it need to be part of the framework itself?


 I'm sorry for the rant, developing this kind of software is a 
 pain in the drain and stresses me out to no end. It sucked hard 
 in the past with php and decades later with python, ruby, D, 
 you name it it still sucks ;)


Its sucks but the absence of libraries makes it suck more. Its 
only a matter of time...as long as more people try vibe.d and 
develop tools around it.

I love D but I also know its not ready for every task...unless 
you're willing to write some things yourself.
Sep 18 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 18 September 2020 at 22:02:07 UTC, aberba wrote:

 In this case you want to get the file(s) in memory...in the 
 form of bytes (or buffer) and probably set a file size limit. 
 Its all doable through a library but such a library doesn't 
 exist in D yet. At least not that I know of.


I actually added *exactly* this to cgi.d in... 2010 if I remember 
right. I even kinda documented it: 
http://dpldocs.info/experimental-docs/arsd.cgi.Cgi.UploadedFile.contentInMemory.html

The threshold where it goes to a file is right now 10 MB and not 
configurable, but I've been meaning to go back and clean up this 
api a little. The max file size it accepts is configurable 
<http://dpldocs.info/experimental-docs/arsd.cgi.GenericMain.html> 
but the threshold where it goes from memory to file is not.

Though I should note that if vibe is putting it in a temporary 
file it probably realistically stays in memory anyway.... this 
whole thing might be about nothing since the OS is pretty good 
about optimizing temp files.

Just I have bigger things to take care of right now (or should I 
say smaller things?!)
Sep 18 2020
parent wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 22:21:52 UTC, Adam D. Ruppe wrote:

 On Friday, 18 September 2020 at 22:02:07 UTC, aberba wrote:

 [...]


 I actually added *exactly* this to cgi.d in... 2010 if I 
 remember right. I even kinda documented it: 
 http://dpldocs.info/experimental-docs/arsd.cgi.Cgi.UploadedFile.contentInMemory.html

 The threshold where it goes to a file is right now 10 MB and 
 not configurable, but I've been meaning to go back and clean up 
 this api a little. The max file size it accepts is configurable 
 <http://dpldocs.info/experimental-docs/arsd.cgi.GenericMain.html> but the
threshold where it goes from memory to file is not.


This looks promising. How could I forget about arsd. You have 
something for anything :)


 Though I should note that if vibe is putting it in a temporary 
 file it probably realistically stays in memory anyway.... this 
 whole thing might be about nothing since the OS is pretty good 
 about optimizing temp files.


But if I wanted that sort of "convenience" I would still be using 
PHP ;)


 Just I have bigger things to take care of right now (or should 
 I say smaller things?!)


:)
Sep 19 2020
prev sibling parent reply wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 22:02:07 UTC, aberba wrote:

 [...]


 That's what I was trying to answer. When Steve said meh, he 
 probably didn't get what I said. Probably its because of my 
 typos.

 This sort of convenience and productivity benefit is part of 
 why I use Node.Js in the job when I need to get things 
 done....and not D yet. There are several pieces and bits you 
 can't write yourself when working on projects.

 In this case you want to get the file(s) in memory...in the 
 form of bytes (or buffer) and probably set a file size limit. 
 Its all doable through a library but such a library doesn't 
 exist in D yet. At least not that I know of.


Yes it would be convenient and thanks for the links but since I'm 
not familiar with Node.js at all parsing the input myself will be 
better than getting into Node.



 Vibe.d still lacks many things I personally need... there's 
 simply not enough ecosystem third-party libraries.


My first impression of vibe.d is that the author(s) made a 
presumptions about the one way things should be done(tm) and if 
that's not your use case too bad for you.
That's where most of my displeasure of using vibe.d comes from 
and that those aren't described in the documentation - not so 
much because of the things that aren't implemented (yet).

Handling file uploads is one example, another is command line 
arguments.
The presumption here is there is vibe and there can only be vibe. 
It takes them from Runtime.args. Duh?
This didn't make sense to me until I saw example where the 
initialization of vibe was done in a module constructor.
Looks cool on a cursory look but is incredibly impractical if you 
don't want to do it that way. And that's the gist - vibe.d is 
incredibly impractical if your use case doesn't exactly match the 
author(s) assumptions of the one way to do things(tm).
And there are issues with that, too. E.g. in the example the 
server will be started before command line args are completely 
processed. That means if there are wrong or missing or extra args 
the application aborts and leaks OS resources.
The way I would have implemented it is to take args from 
Runtime.args by default but allow passing an args[] to a vibe 
args-handler. I could then parse whatever args I'm interested in 
in my favorite way and pass the rest to vibe.

But you can handle those with vibe getOption or some such - why 
don't you want to do comandline args processing with vibe you ask 
?
Because 1. there's std.getopt which is awesome, and
2. If I don't depend on vibe to parse them and I have a build 
configuration with version="NO_SERVER" I don't even need to link 
against vibe and its dependencies.

By using vibe I feel like I need to bend myself like a snake and 
jump through the hoops of vibe's one way to do it. You save a lot 
of time here and there and then you lose half a day because of 
some stupid road block like the above.
Sep 19 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/19/20 6:59 AM, wjoe wrote:

 Handling file uploads is one example, another is command line arguments.
 The presumption here is there is vibe and there can only be vibe. It 
 takes them from Runtime.args. Duh?
 This didn't make sense to me until I saw example where the 
 initialization of vibe was done in a module constructor.


This used to be the expected way to set up vibe (using module 
constructors). And vibe would provide its own main function.

I *think* the idea was to allow registration of different handlers in 
their respective modules.

Nowadays, you just run the setup wherever you want (I do everything in 
main).


 By using vibe I feel like I need to bend myself like a snake and jump 
 through the hoops of vibe's one way to do it. You save a lot of time 
 here and there and then you lose half a day because of some stupid road 
 block like the above.


I used Kai's book, and yeah, you have to do things the vibe way. But 
most web frameworks are that way I think.

I recommend getting the book if you plan on doing a lot with vibe.d

Where vibe really shines are the diet templates and the web interface 
stuff. To not have to handle anything from forms, or query parameters, 
and just write normal D functions is really great. You can even do a lot 
of authentication and stuff using UDAs. It helps you write consistent 
web applications.

And I LOVE diet templates. So much less verbose than actual HTML. I have 
a hard time writing actual HTML now.

When you want to do stuff manually, I think it's not as well supported.

-Steve
Sep 19 2020
next sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 19 September 2020 at 19:27:40 UTC, Steven 
Schveighoffer wrote:

 I used Kai's book, and yeah, you have to do things the vibe 
 way. But most web frameworks are that way I think.


Do you have a reference to this book (web link, ISBN)?
Sep 19 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/19/20 3:36 PM, IGotD- wrote:

 On Saturday, 19 September 2020 at 19:27:40 UTC, Steven Schveighoffer wrote:

 I used Kai's book, and yeah, you have to do things the vibe way. But 
 most web frameworks are that way I think.


 
 Do you have a reference to this book (web link, ISBN)?
 
 


Sure:

https://www.packtpub.com/product/d-web-development/9781785288890

It might be a bit dated, but most of the concepts are the same.

-Steve
Sep 19 2020
prev sibling parent wjoe <invalid example.com> writes:
On Saturday, 19 September 2020 at 19:27:40 UTC, Steven 
Schveighoffer wrote:

 [...]
 This used to be the expected way to set up vibe (using module 
 constructors). And vibe would provide its own main function.

 I *think* the idea was to allow registration of different 
 handlers in their respective modules.


A reasonable decision if you assume that's the only thing you 
want to do.
Vibe and dub share this philosophy. The author(s) assumed those 
are the only use cases and nothing else flies.
Like for e.g. you can build a D application that uses a C library 
with D glue code and compile the files together in 1 go with GCC, 
maybe with LDC, too, but I haven't tried.
something like: gcc main.d dglue.d clib.c clib.h
But this doesn't work with dub, or at least it didn't work the 
last time I tried a year or so ago.
I'm not a fan of the "The one true way" philosophy.


 [...]
 I used Kai's book, and yeah, you have to do things the vibe 
 way. But most web frameworks are that way I think.

 I recommend getting the book if you plan on doing a lot with 
 vibe.d


I forgot that I got it some time ago, never really got around to 
take a look however. Thanks for the reminder :)


 Where vibe really shines are the diet templates and the web 
 interface stuff. To not have to handle anything from forms, or 
 query parameters, and just write normal D functions is really 
 great. You can even do a lot of authentication and stuff using 
 UDAs. It helps you write consistent web applications.

 And I LOVE diet templates. So much less verbose than actual 
 HTML. I have a hard time writing actual HTML now.


Those are the only reason why I didn't throw it out, yet. Maybe 
it will make up for some of the initial frustration :)


 When you want to do stuff manually, I think it's not as well 
 supported.

 -Steve
Sep 19 2020
prev sibling parent reply mw <mingwu gmail.com> writes:
On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:


 Are there other frameworks besides vibe that can do what I want?


Just FYI, there is also:

https://code.dlang.org/packages/hunt-framework


I never used myself, you need to investigate.
Sep 18 2020
next sibling parent wjoe <invalid example.com> writes:
On Friday, 18 September 2020 at 22:31:09 UTC, mw wrote:

 On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:


 Are there other frameworks besides vibe that can do what I 
 want?


 Just FYI, there is also:

 https://code.dlang.org/packages/hunt-framework


 I never used myself, you need to investigate.


Thanks for the link, I will have a look.
Sep 19 2020
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 18 September 2020 at 22:31:09 UTC, mw wrote:

 On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:


 Are there other frameworks besides vibe that can do what I 
 want?


 Just FYI, there is also:

 https://code.dlang.org/packages/hunt-framework


 I never used myself, you need to investigate.


Yeah, I'm aware of hunt too. Its like laravel in php whilst 
vibe.d feels more like express in nodejs. Infact I believe Sonke 
wrote vibe.d after using express or similar framework by design.


And yes, almost all frameworks work in a certain way. Arsd cgi.d 
might be what you want if you want to it your way as its more 
low-level interface-wise.

I personally (and many others in the industry... judging by 
popularity of express (node.js) and the plentiful third-party 
libraries,..do prefer the router.get() design. Also having 
everything abstracted in a convenient and consistent way...is 
very desirable. vibe.d's web interface API is something I've 
always praised and thanks to how powerful D is compare to say 
JavaScript. Diet templates is also an example.

However that power is not tapped in enough (yet) to favour using 
D conveniently over node (js). And web developers are interested 
in getting things done (at least my kind of web devs) rather than 
getting it our way...or squeezing every bit of efficiency out of 
it. Part of why v8 is fast by default (for us).

Unike express (which is a thing layer with interface for 
third-parties to hook in and extend with libs), vibe.d became a 
monolith with everything included... making it harder to maintain 
and extend in other ways. Plus too much hard-coding by default 
currently. Unfortunately Sonke doesn't work on it like he used 
to...and its quite an huge accomplishment...the work he's done 
for D.


I wish vibe.d could be positioned and sponsored officially or by 
community cus its the killer app.

Staying lean and being extensible will open up for more 
innovation around it. Eg. a form handler for files library that 
works like how Ikod suggested...a customizable stream.


Unless you're doing usual routing and database things, using 
vibe.d for a full stack projects can lead to a dead end unless 
you're positioned to write your own stuff to supplement. Of 
course its only a matter of time before this change for the good.

Personally I use vibe.d for basic side projects. Looking to use 
it more going forward. But that's how I see it.
Sep 19 2020
next sibling parent aberba <karabutaworld gmail.com> writes:
On Saturday, 19 September 2020 at 20:17:06 UTC, aberba wrote:

 On Friday, 18 September 2020 at 22:31:09 UTC, mw wrote:

 On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:


 Are there other frameworks besides vibe that can do what I 
 want?







 Personally I use vibe.d for basic side projects. Looking to use 
 it more going forward. But that's how I see it.


This is my personal collection of D web development packages. Let 
me know if I'm missing something.

https://gist.github.com/aberba/dcaf9102b35205080ad99a2af2c21142
Sep 19 2020
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 19 September 2020 at 20:17:06 UTC, aberba wrote:

 Arsd cgi.d might be what you want if you want to it your way as 
 its more low-level interface-wise.


Eh, it depends. My lib lets you do as much or as little as you 
want.

About ten years ago, I wrote a framework on top that 
automatically generates javascript/json interfaces as well as 
html form and output UI just given an ordinary D class. Had full 
interop with PHP too... all with no UDAs since they didn't exist 
yet!

I never really documented how to use it. Wrote a few examples on 
the forum but nothing formal. I kinda regret that; I've *still* 
never seen anyone else, anywhere, automate as much as it did.

old demo I posted in May 2011: 
http://arsdnet.net/cgi-bin/apidemo/javascript

I can't find the source to that anymore but it would be something 
like:

---
import arsd.web;

class CoolApi : WebApi {
    enum Color { red, green, etc }
    struct Person { int id; string first; string last; }

    Element getABox(Color c) {
          auto div = Element.make("div");
          div.style.color = to!string(c);
          return div;
    }

    int addSomeNumbers(int a, int b) { return a + b; }

    Person[] getPeople(int startingId) { return [Person(...)]; }
}
mixin FancyMain!CoolApi;
---

That's it - all the html and javascript are all auto-generated.


Nowadays I'm merging all those ideas into cgi.d and actually 
writing about it in my blog. There's a lot in my old 
implementation I didn't like but now that D's reflection is 
better than it was, I have techniques to make it even better.

My new dwidder website is the only public example of the new code 
so far 
https://github.com/adamdruppe/dupdates/blob/master/main.d#L157

Still a bunch more I'm slowly bringing together. D has unique 
strengths for web that javascript, ruby, and python can just 
*never* do and we should be pushing the edges to go as cool as we 
can.
Sep 19 2020
next sibling parent reply wjoe <invalid example.com> writes:
On Sunday, 20 September 2020 at 00:36:30 UTC, Adam D. Ruppe wrote:

 [...]
 That's it - all the html and javascript are all auto-generated.


Amazing :)
Would even be more awesome if it provided a function which could 
be called from a custom main on top of the FancyMain.
I find e.g. custom parsing of command line arguments incredibly 
useful.
Sep 19 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 20 September 2020 at 01:51:22 UTC, wjoe wrote:

 Would even be more awesome if it provided a function which 
 could be called from a custom main on top of the FancyMain.
 I find e.g. custom parsing of command line arguments incredibly 
 useful.


Yea, the version 2.0 stuff inside cgi.d does that. You can call 
`cgiMainImpl` from your own main thing (that's all the mixin does 
anyway) and disaggregate it as much as you want.

I was improving that this week when the baby decided to come, so 
the rest will be coming a little later. But on my copy right now 
you can even break up cgiMainImpl because its implementation is:

---
void requestHandler(Cgi cgi) {
    // if you call cgi.dispatcher in here, it does basically
    // what the old FancyMain would do, going to a class.
}

void main(string[] args) {
         alias CustomCgi = Cgi; // just normal class w/o more 
custom
         if(tryAddonServers(args))
                 return;

         if(trySimulatedRequest!(requestHandler, CustomCgi)(args))
                 return;

         RequestServer rs;
         rs.listeningPort = 3000; // if you want to change default
         rs.configureFromCommandLineArguments(args);
         rs.handleRequests!(requestHandler, CustomCgi)(); // this 
may never return
}
---


But I haven't pushed this live yet. Probably will write about it 
in the blog next week or the week after depending on how 
everything else irl goes. Still not 100% happy with it, but a big 
goal with my version 2.0 implementation here is to make more 
pick-and-choose customization possible while still using the 
automation in the places you want that.
Sep 19 2020
prev sibling parent wjoe <invalid example.com> writes:
On Sunday, 20 September 2020 at 00:36:30 UTC, Adam D. Ruppe wrote:

 [...]


I browsed in your arsd docs a bit and I'll have a closer look at 
the CGI module a bit later.
Your http2 module piqued my interest as it could come in handy 
some time later :)

Looks like your modules cover everything I need and requiring 
only 2 or 3 modules that cover everything I would use from vibe 
beats fighting with dub and vibe's complexity - and I can use a 
simple makefile :) That alone will save a ton of time.
Sep 20 2020
prev sibling parent wjoe <invalid example.com> writes:
On Saturday, 19 September 2020 at 20:17:06 UTC, aberba wrote:

 I personally (and many others in the industry... judging by 
 popularity of express (node.js) and the plentiful third-party 
 libraries,..do prefer the router.get() design. Also having 
 everything abstracted in a convenient and consistent way...is 
 very desirable. vibe.d's web interface API is something I've 
 always praised and thanks to how powerful D is compare to say 
 JavaScript. Diet templates is also an example.


I'm not a professional web developer and I don't intend to become 
one.
My intention is to live a long and mentally healthy life and web 
development is a serious threat to that ;)


 However that power is not tapped in enough (yet) to favour 
 using D conveniently over node (js). And web developers are 
 interested in getting things done (at least my kind of web 
 devs) rather than getting it our way...or squeezing every bit 
 of efficiency out of it. Part of why v8 is fast by default (for 
 us).


But joking aside, I'm going to use D over Javascript any day of 
the week no questions asked.
And despite my constant bickering, vibe is a huge accomplishment 
and improvement over what I used in the past. So kudos.
Sep 19 2020
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Thursday, 17 September 2020 at 16:32:55 UTC, WebFreak001 wrote:

 On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:

 I found this [1] but unfortunately the post this refers to is 
 a dead link and the content, unfortunately, didn't tell me 
 anything that I didn't already find in the docs.

 What I can get from the form is the form fields with content, 
 the field name for the file upload and the file name. But the 
 file name is useless to me because I need the file content.

 Where is it stored ?

 [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/


 hi, you can access HTTPServerRequest.files which contains the 
 uploaded file.

 Example in real code: 
 https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159

 Documentation: https://vibed.org/api/vibe.inet.webform/FilePart

 Note: the file is only downloaded from the client / stored on 
 disk once you access the files or the form property, though 
 this isn't documented.

 I don't believe the temp file behavior is customizable as it is 
 hardcoded here to write to a temporary file (which is called on 
 form parsing): 
 https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311

 However if you really wanted to (but I'd advise against it) you 
 could parse the form data from the HTTPServerRequest.bodyReader 
 yourself



Yeah I think what he wants is a way to write the file into an 
in-memory buffer. This is especially necessary it some 
environments where you can write to disk at all.


How that's done, even in node.js is to use a middleware for 
parsing library to parse the headers in such way. I often use 
multer (which is also based on busybody) to do that in nodejs.

Unfortunately I haven't used D to that extent to need such a 
thing... I in never hit that problem.

I wonder if using a psuedo file handler will work for the 
in-memory buffer thing.
Sep 17 2020