digitalmars.D.learn - vibe.d maxRequestSize
- Chris (14/14) Sep 14 2016 The vibe.d server rejects `XMLHttpRequest`s that are too long (in
- Steven Schveighoffer (5/18) Sep 14 2016 Hm.. I have adjusted this in my project, and it works (set to 50M).
- Chris (4/8) Sep 15 2016 It doesn't seem to make any difference in my case. I wonder what
- Steven Schveighoffer (4/15) Sep 15 2016 Seems to be in use:
- Chris (9/29) Sep 15 2016 Hm, maybe the browsers are limiting the the length of the query
- sarn (6/6) Sep 15 2016 I hope this isn't too obvious, but I have to ask because it's
- Chris (4/10) Sep 16 2016 No. Atm I'm running the server locally on my machine (for
- Chris (13/33) Sep 19 2016 Finally, I could find where it happens. If I use "get" in a HTML
- Steven Schveighoffer (5/37) Sep 19 2016 Hm... you don't get a full stack trace? Hard to tell what max_bytes
- Chris (24/29) Sep 19 2016 I didn't want to litter the thread, here it is:
- Steven Schveighoffer (10/59) Sep 19 2016 Here is the culprit:
The vibe.d server rejects `XMLHttpRequest`s that are too long (in the eyes of the server). In the docs it says "maxRequestSize ulong Maximum number of transferred bytes per request after which the connection is closed with [sic!]" However, when you go to http://vibed.org/api/vibe.http.server/HTTPServerSettings.maxRequestSize it says "Maximum number of transferred bytes per request after which the connection is closed with an error; not supported yet" "not supported yet" and I think it is not supported yet. Can I still change the server settings so that it accepts longer query strings? Atm, the server rejects strings that are > ~2,500 characters, which is not much.
Sep 14 2016
On 9/14/16 9:58 AM, Chris wrote:The vibe.d server rejects `XMLHttpRequest`s that are too long (in the eyes of the server). In the docs it says "maxRequestSize ulong Maximum number of transferred bytes per request after which the connection is closed with [sic!]" However, when you go to http://vibed.org/api/vibe.http.server/HTTPServerSettings.maxRequestSize it says "Maximum number of transferred bytes per request after which the connection is closed with an error; not supported yet" "not supported yet" and I think it is not supported yet. Can I still change the server settings so that it accepts longer query strings? Atm, the server rejects strings that are > ~2,500 characters, which is not much.Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve
Sep 14 2016
On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote:Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -SteveIt doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 15 2016
On 9/15/16 9:11 AM, Chris wrote:On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote:Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -SteveHm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -SteveIt doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 15 2016
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote:On 9/15/16 9:11 AM, Chris wrote:Hm, maybe the browsers are limiting the the length of the query string themselves, although that shouldn't be an issue with POST. I can send very long strings, if I enable HTTPServerOption.parseFormBody, but then I can no longer parse the URI at the same time (as far as I know). The browser always sends `Content-Length: 0` in the request header, which doesn't affect vibe.d (short strings work).On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote:Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -SteveHm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -SteveIt doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 15 2016
I hope this isn't too obvious, but I have to ask because it's such a common gotcha: Are you reverse proxying through a server like nginx by any chance? There are default request size limits there. (For nginx specifically, it's this one: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size)
Sep 15 2016
Chris <wendlec tcd.ie> On Friday, 16 September 2016 at 00:35:25 UTC, sarn wrote:I hope this isn't too obvious, but I have to ask because it's such a common gotcha: Are you reverse proxying through a server like nginx by any chance? There are default request size limits there. (For nginx specifically, it's this one: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size)No. Atm I'm running the server locally on my machine (for development), so I access vibe.d directly. But later it will run on Apache, and query string size will inevitably be an issue.
Sep 16 2016
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote:On 9/15/16 9:11 AM, Chris wrote:Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/strea /operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote:Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -SteveHm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -SteveIt doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 19 2016
On 9/19/16 11:59 AM, Chris wrote:On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote:Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -SteveOn 9/15/16 9:11 AM, Chris wrote:Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote:Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29It doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 19 2016
On Monday, 19 September 2016 at 16:55:05 UTC, Steven Schveighoffer wrote:Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -SteveI didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/strea /operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos std/exception.d:405 pure safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos std/exception.d:363 pure safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.All cAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1659 bool vibe.http.server.handleRequest(vibe.core.stream.Stream, vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo, ref vibe.http.server.HTTPServerSettings, ref bool) [0xb5c5d0] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1552 void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo) [0xb5be4e] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1433 void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings).doListen(vibe.http.se ver.HTTPListenInfo, bool, bool).__lambda4(vibe.core.net.TCPConnection) [0xb5b814] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/drivers libevent2_tcp.d:610 void vibe.core.drivers.libevent2_tcp.ClientTask.execute() [0xbe8715] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:488 void vibe.core.core.makeTaskFuncInfo!(void delegate()).makeTaskFuncInfo(ref void delegate()).callDelegate(vibe.core.core.TaskFuncInfo*) [0xafea75] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:1119 void vibe.core.core.CoreTask.run() [0xb8dd79] ??:? void core.thread.Fiber.run() [0xc70ea5] ??:? fiber_entryPoint [0xc70c27] ??:? [0xffffffff]
Sep 19 2016
On 9/19/16 1:34 PM, Chris wrote:On Monday, 19 September 2016 at 16:55:05 UTC, Steven Schveighoffer wrote:Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -SteveHm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -SteveI didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:405 pure safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:363 pure safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core.stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827]
Sep 19 2016
On Monday, 19 September 2016 at 17:54:05 UTC, Steven Schveighoffer wrote:On 9/19/16 1:34 PM, Chris wrote:Thanks a million! I'll look into it tomorrow.[...]Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Sep 19 2016
On Monday, 19 September 2016 at 18:13:12 UTC, Chris wrote:On Monday, 19 September 2016 at 17:54:05 UTC, Steven Schveighoffer wrote:FYI, I had to solve the issue by sending a form via a JS XMLHTTPRequest. maxRequestSize didn't help.On 9/19/16 1:34 PM, Chris wrote:Thanks a million! I'll look into it tomorrow.[...]Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Sep 22 2016