digitalmars.D.learn - Good way to send/receive UDP packets?
- Dukc (12/12) Jul 18 2020 I have a project where I need to take and send UDP packets over
- H. S. Teoh (6/17) Jul 18 2020 If you already have the raw packets, there is no need for any library,
- IGotD- (3/15) Jul 18 2020 D has socket wrapper interfaces just as many other languages.
- Adam D. Ruppe (4/6) Jul 18 2020 I wrote an example using phobos on my blog a while ago that might
- notna (11/23) Jul 19 2020 Someone once wrote about a UDP library, which was used to sync
- wjoe (22/25) Jul 21 2020 In the best case scenario, and orders of magnitude more
- notna (6/32) Jul 21 2020 well, I guess all your remarks are true... and irrelevant at the
- wjoe (34/41) Jul 22 2020 ... and continues with:
- Dukc (10/13) Jul 22 2020 But doesn't the UDP header include the sender IP address? So
- Dukc (6/6) Jul 19 2020 Thank you everybody - Especially for the links to the blogs. This
I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently. I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why. Other advice about projects like this is also welcome, should anyone wish to share it.
Jul 18 2020
"H. S. Teoh" <hsteoh quickfur.ath.cx> On Sat, Jul 18, 2020 at 04:00:09PM +0000, Dukc via Digitalmars-d-learn wrote:I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently. I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why.If you already have the raw packets, there is no need for any library, just call the OS's C API directly (such as core.sys.posix.sys.socket). For UDP you don't even need to set up anything, just create a socket and fire the packets away. --T
Jul 18 2020
On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently. I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why. Other advice about projects like this is also welcome, should anyone wish to share it.D has socket wrapper interfaces just as many other languages. https://dlang.org/phobos/std_socket.html
Jul 18 2020
On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:I have a project where I need to take and send UDP packets over the Internet. Only raw UDPI wrote an example using phobos on my blog a while ago that might help you get started: http://dpldocs.info/this-week-in-d/Blog.Posted_2019_11_11.html#communication-by-datagram
Jul 18 2020
On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently. I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why. Other advice about projects like this is also welcome, should anyone wish to share it.Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)... I couldn't find this info now, neither in the forum not something meaningful in code.dlang.org... Maybe the author is still around or someone else has this post at hand and could share it?! Beside this, there is: - https://dlang.org/blog/2016/10/21/project-highlight-libasync/ Hope this helps...
Jul 19 2020
On Sunday, 19 July 2020 at 09:48:24 UTC, notna wrote:Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)...In the best case scenario, and orders of magnitude more unreliable otherwise. Choosing UDP over TCP because speed is like choosing a hammer over a screwdriver to drive in a screw because it works and is faster. But it's still the wrong tool for the job. UDP is a protocol for broadcasting messages which means it's connection less, unreliable (as in no guarantees for delivery of datagrams or that they'll be delivered only once, order i.e. datagrams sent in order A B C D can be delivered like e.g B C A D), and data integrity. It's insecure (as in everyone who listens can receive it). Once you need any of these features/guarantees you'll lose performance just as you would by using TCP, plus you pay the cost for re-inventing the wheel, bugs, testing, maintenance, support and all. UDP is the protocol of choice when you want to broadcast, the data you send isn't important or the TCP overhead is bigger than the transmitted message. Like broadcasting radio/podcasts, weather updates for your status bar or some such or DNS queries. If you need reliable transmission you need to use a reliable protocol.
Jul 21 2020
On Tuesday, 21 July 2020 at 13:05:21 UTC, wjoe wrote:On Sunday, 19 July 2020 at 09:48:24 UTC, notna wrote:well, I guess all your remarks are true... and irrelevant at the same time. please go back and read his first post.... starts with "I have a project where I need to take and send UDP packets over the Internet"...Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)...In the best case scenario, and orders of magnitude more unreliable otherwise. Choosing UDP over TCP because speed is like choosing a hammer over a screwdriver to drive in a screw because it works and is faster. But it's still the wrong tool for the job. UDP is a protocol for broadcasting messages which means it's connection less, unreliable (as in no guarantees for delivery of datagrams or that they'll be delivered only once, order i.e. datagrams sent in order A B C D can be delivered like e.g B C A D), and data integrity. It's insecure (as in everyone who listens can receive it). Once you need any of these features/guarantees you'll lose performance just as you would by using TCP, plus you pay the cost for re-inventing the wheel, bugs, testing, maintenance, support and all. UDP is the protocol of choice when you want to broadcast, the data you send isn't important or the TCP overhead is bigger than the transmitted message. Like broadcasting radio/podcasts, weather updates for your status bar or some such or DNS queries. If you need reliable transmission you need to use a reliable protocol.
Jul 21 2020
On Tuesday, 21 July 2020 at 18:35:34 UTC, notna wrote:well, I guess all your remarks are true... and irrelevant at the same time. please go back and read his first post.... starts with "I have a project where I need to take and send UDP packets over the Internet"...... and continues with: On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:[...] And only communication with a single address, no need to communicate with multiple clients concurrently.Let me elaborate on why what I wrote is both, on topic and relevant at the same time: It's a fundamental characteristic of UDP that you can't communicate with a single client only but you always are communicating with everyone who listens concurrently. The combination of the Internet and theoretically only 65535 available ports means that (mis)communication with other clients is likely to happen. OP's client may receive datagrams that are broadcast on that port by other programs (port forwarding). Likewise, other programs, which have a socket bound to the same port, may receive OP's data. Examples: - You behaving correctly yourself doesn't make others behave correctly now or retroactively. - Choosing a port which isn't in use right now isn't good enough because a few minutes later there may be another program using it, too, and for the same reason. - Checking a magic byte at offset 0 followed by the size isn't good enough. Let's say I decide my magic is a word because I want to be able to determine byte order - I choose 0x015a. When I broadcast a packet, e.g. [[0x015a], [0x0, 0x5a, 0xff]], but because I sent this from a little endian machine, the datagram will be delivered like so [0x5a, 0x01, 0x0, 0x5a, 0xff]. When OP's client receives this message it goes like so: Read 2 bytes: [0x5a, 0x1] 0x5a, oh that's me, 0x01 aha length of 1 byte - except it's not even a valid packet in the context of their program... Using UDP in a good way is far more complicated than it first appears to be.
Jul 22 2020
On Wednesday, 22 July 2020 at 13:17:11 UTC, wjoe wrote:- Choosing a port which isn't in use right now isn't good enough because a few minutes later there may be another program using it, too, and for the same reason.But doesn't the UDP header include the sender IP address? So together with the magic number, shouldn't it be good enough? I know it definitely is not going to hold against jamming or a man in the middle, but it isn't supposed to, at this stage. It's used only for simulations that have the criticality of a casual Tetris server. I do acknowledge that the needs may rise later on. And if so, I understand that I'm much better off switching the protocol than trying to hardening the UDP.
Jul 22 2020
On Wednesday, 22 July 2020 at 15:26:23 UTC, Dukc wrote:On Wednesday, 22 July 2020 at 13:17:11 UTC, wjoe wrote:No, the UDP header includes the source and destination ports only. For transmission over the internet the Internet Protocol (IP) is used, which contains, among other things, the source and destination addresses. The anatomy looks like this: [ -------- IP - Datagram ------ ] [IP-Header][UDP-Header][UDP-Data] [ - UDP - Datagram - ] But keep in mind that the destination address can be a broadcast address, like e.g. 255.255.255.255, which you would use to announce your server to every PC in the network. If you send a UDP datagram to a single address, however, it will still be delivered to every program on that PC which receives UDP datagrams from that port. Also if you send UDP datagrams to multiple specific addresses, you need to send the same packet multiple times losing the major benefit of UDP - broadcast. And packets with a broadcast address sent over the internet are dropped, as that would affect every connected PC. If you are behind a router and send over the internet, your router will modify the IP-header, namely the sender address and replace that with your public address. When receiving packets, the IP header contains the destination address of your public IP (the router), which it will translate to the local address according to the port forwarding setup. That process is called network address translation (NAT) and is not only relevant for UDP.- Choosing a port which isn't in use right now isn't good enough because a few minutes later there may be another program using it, too, and for the same reason.But doesn't the UDP header include the sender IP address? So together with the magic number, shouldn't it be good enough? I know it definitely is not going to hold against jamming or a man in the middle, but it isn't supposed to, at this stage. It's used only for simulations that have the criticality of a casual Tetris server. I do acknowledge that the needs may rise later on. And if so, I understand that I'm much better off switching the protocol than trying to hardening the UDP.
Jul 22 2020
On Wednesday, 22 July 2020 at 16:14:24 UTC, wjoe wrote:When receiving packets, the IP header contains the destination address of your public IP (the router), which it will translate to the local address according to the port forwarding setup.Pardon me, I meant to say according to the current routing table rules. Port Forwarding forwards all packets that match the defined criteria to the specified PC in the network. A port forwarding rule isn't required for NAT to work. Sender and receiver both need to send a packet to each other. This will cause the router to add a rule and expect an answer on that port from the destination address. When this packet arrives before the rule has been made it will be dropped (unless a port forwarding rule exists), so several packets may need to be sent for a successful punch-through.
Jul 22 2020
On Wednesday, 22 July 2020 at 16:14:24 UTC, wjoe wrote:If you send a UDP datagram to a single address, however, it will still be delivered to every program on that PC which receives UDP datagrams from that port.Normally binding two sockets to the same port is not allowed.
Jul 23 2020
On Thursday, 23 July 2020 at 13:29:47 UTC, Kagamin wrote:On Wednesday, 22 July 2020 at 16:14:24 UTC, wjoe wrote:But it's possible when bound with the socket option SO_REUSEPORT (at least that's the name of the flag on linux since 3.9).If you send a UDP datagram to a single address, however, it will still be delivered to every program on that PC which receives UDP datagrams from that port.Normally binding two sockets to the same port is not allowed.
Jul 27 2020
On Monday, 27 July 2020 at 09:41:44 UTC, wjoe wrote:But it's possible when bound with the socket option SO_REUSEPORT (at least that's the name of the flag on linux since 3.9).The docs say it can't be used to hijack an address.This option must be set on each socket (including the first socket) prior to calling bind(2) on the socket.
Jul 28 2020
On Tuesday, 28 July 2020 at 15:01:08 UTC, Kagamin wrote:On Monday, 27 July 2020 at 09:41:44 UTC, wjoe wrote:Nowhere did I claim that you could hijack a bound port but that it's possible to reuse a port. Nothing prevents a UDP library from setting that option automatically or by default. Nothing prevents any program to bind to a port with that option set. Normal behavior doesn't matter - what matters is what the library you're using and other programs are doing. All of this doesn't change the fact that every program that has a socket bound to the same port on the same address at the same time will get the datagrams deliverd. One way to handle this scenario is to bind your socket with the reuse option unset. That would be first come first served. Problem not solved - you still need to consider the fact that another program or server on a different PC/address in the network can assume that their corresponding client is up and running and wants to receive on that very port. Due to the nature of UDP communication, the server needn't care if this assumption is true and send datagrams to it which you will receive instead.But it's possible when bound with the socket option SO_REUSEPORT (at least that's the name of the flag on linux since 3.9).The docs say it can't be used to hijack an address.This option must be set on each socket (including the first socket) prior to calling bind(2) on the socket.
Jul 28 2020
Thank you everybody - Especially for the links to the blogs. This is just the kind of stuff I seek (didn't give a close look yet, though). I think I'm going to try std.socket first, since it's in the standard library. If it feels like it could be easier, I'll consider Libasync.
Jul 19 2020