While working on tuntcp, one of the first things I did was set up a TUN interface, a virtual network device operating at the Network Layer/Layer 3 of the network stack. I used the following commands:

sudo ip tuntap add dev tun0 mode tun user $USER
sudo ip addr add 192.0.2.1 peer 192.0.2.2 dev tun0
sudo ip link set dev tun0 up

I ran these commands with only a vague understanding of what's going on - it creates a TUN device called tun0, assigns it two IP addresses for some reason and sets the interface to up.

However, I had questions, which I kept ignoring as I was making progress in terms of code. When I got stuck on another part of the project and was unable to code, I decided figure out what these commands really do. This post is a collection of notes I made along the way.

Why are there two addresses for a TUN interface?

The first thing that confused me was the mention of two addresses. I initially thought that it might not be significant, however any packets I created used the peer address as the source address, 192.0.2.2, which didn't make sense.

I started with the man page for ip-address:

peer ADDRESS
        the address of the remote endpoint for pointopoint inter‐
        faces.
...

Well, this just threw more terms I don't understand at me, particularly pointopoint interfaces. So I focused on understanding that first, with the help of an answer on StackOverflow.

A point-to-point interface connects two hosts directly. The only hosts on this interface are 192.0.2.1 and 192.0.2.2 in our example.

Due to this, this interface does not have a MAC address and anything related to Layer 2, like ARP or the Address Resolution Protocol. This is in contrast to an Ethernet device, like the eth0 interface, which is not a point to point interface and has a MAC address.

Next came the remote endpoint part of the explanation. Point-to-point interfaces were still not clicking at this point, so I was confused on why the remote address was even required.

This confusion was cleared by a beautiful answer on Stack Exchange. I didn't even try paraphrasing this because it's explained so well:

In ancient times, when people used to connect a modem device to the telephone line and dial the phone of an internet provider to establish connection to the internet, the pppd daemon used to be responsible to establish a point-to-point tunnel to the server located at the other end of the call. On those tunnels, the local address was the address assigned to the network interface being created in kernel network stack and the remote address was the local address of the other computer answering the phone call and, also, the address set as the default gateway in the local side.

For TUN virtual devices, your user-space program will act as the remote computer at the other side of the tunnel. Therefore, in order to have your program injecting IP packets to the kernel network stack, the program is supposed to generate packets with source address set to the tunnel remote address (192.168.69.1) and receive packets whose destination is set to the same address.

After reading this, it all clicked into place.

This confusion comes from the output of the TUN interface settings.

$ ip addr show tun0
8: tun0: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 500
    link/none
    inet 192.0.2.1 peer 192.0.2.2/32 scope global tun0
       valid_lft forever preferred_lft forever

This situation appears when the link is set to up, but is not being actively used by a program, indicated by the NO_CARRIER flag. This seems similar to a laptop charger being connected to a socket, but the charger isn't plugged into a laptop yet.

tuntcp contains code that connects to this interface. So once the program is run, the connection is established, and the interface's output changes:

$ ip addr show tun0
8: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/none
    inet 192.0.2.1 peer 192.0.2.2/32 scope global tun0
        valid_lft forever preferred_lft forever
    inet6 fe80::2db0:db32:1898:51a5/64 scope link stable-privacy proto kernel_ll
        valid_lft forever preferred_lft forever

NO_CARRIER disappears, LOWER_UP appears and the state changes to UP.

Why does pinging work differently for the two addresses?

Another useful resource I found was a post titled Tun/Tap interface tutorial. It goes in-depth about how TUN interfaces work and are set up.

The part that confused me was the section about pinging the two addresses (10.0.0.1 and 10.0.0.2 in the article), and the difference in output. The interface address (10.0.0.1) responded to pings, whereas the peer address (10.0.0.2) did not, and I didn't fully understand the explanation provided.

I also wanted to know where the ping packets for the interface address were going, which wasn't specified in the post. So I decided to capture ICMP packets at all interfaces to find out.

First, I ran openvpn in the background to keep the TUN interface up, which I discovered from the tutorial post.

$ sudo openvpn --dev tun0
2023-10-22 18:21:54 OpenVPN 2.6.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2023-10-22 18:21:54 library versions: OpenSSL 3.0.8 7 Feb 2023, LZO 2.10
2023-10-22 18:21:54 TUN/TAP device tun0 opened
2023-10-22 18:21:54 Could not determine IPv4/IPv6 protocol. Using AF_INET
2023-10-22 18:21:54 UDPv4 link local (bound): [AF_INET][undef]:1194
2023-10-22 18:21:54 UDPv4 link remote: [AF_UNSPEC]
^Z
[1]+  Stopped                 sudo openvpn --dev tun0
$ bg
[1]+ sudo openvpn --dev tun0 &

This command is also used to create the interface, but in this case works as the program connecting to the interface. Then, I started capturing packets with tcpdump:

$ sudo tcpdump -ni any 'icmp'
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

I ran ping -c 1 192.0.2.1 from another shell session, which returned the following packets (emphasis mine):

18:22:35.053952 lo    In  IP 192.0.2.1 > 192.0.2.1: ICMP echo request, id 28435, seq 1, length 64
18:22:35.053964 lo    In  IP 192.0.2.1 > 192.0.2.1: ICMP echo reply, id 28435, seq 1, length 64

The packets came from the loopback interface (lo) and the source and destination addresses are the same, interesting.

Running ping -c 1 192.0.2.2 returned only one packet (emphasis mine):

18:22:37.419702 tun0  Out IP 192.0.2.1 > 192.0.2.2: ICMP echo request, id 6389, seq 1, length 64

The source address is 192.0.2.1, and the packet is captured at the tun0 interface. Also, no echo reply packet after the request.

Why does each address return different packets? I think it may have something to do with the routes created when the interface is set up.

Checking the routing table

$ ip route show table all | grep tun0
192.0.2.2 dev tun0 proto kernel scope link src 192.0.2.1
local 192.0.2.1 dev tun0 table local proto kernel scope host src 192.0.2.1

Once again, lots of words and little to no idea of what they mean. A Recurser pointed to this guide, which helped make sense of the terminology.

Let's start with the route for 192.0.2.2, the peer address:

192.0.2.2          the route is for the address 192.0.2.2
dev tun0           reachable from interface tun0
proto kernel       route added by the kernel when setting up the interface
scope link         valid only on the mentioned interface, tun0
src 192.0.2.1      preferred source address when sending to this destination

The src part was a little confusing at first, however it does match the ping output for 192.0.2.2 mentioned earlier. The source address there was 192.0.2.1, and I think its because the peer address is reachable only via the local address, being a point-to-point link.

The reason there is no echo reply from 192.0.2.2 is because the connected program has no implementation for ICMP or the TCP/IP stack in general. It doesn't use the kernel's network stack and requires its own, which is where tuntcp steps in.

Next comes the route for 192.0.2.1, which looks pretty different:

local 192.0.2.1    address 192.0.2.1 is locally hosted on this machine
dev tun0           reachable from interface tun0
table local        part of the local routing table
proto kernel       route added by the kernel when setting up the interface
scope host         valid only on this machine
src 192.0.2.1      preferred source address when sending to this destination

This is specified as a local address, an address directly accessible to the machine. The route is added to a routing table called local. When pinging local addresses, the packets are sent in and out from the loopback interface, lo, as seen earlier.

Being a local address, the kernel's network stack manages the sending and receiving, which is why there was an echo reply when pinging to this address.

This is how local routes for other interfaces are written too, which I found out from this comment from the tutorial post.

More questions?

TUN interfaces are starting to make a lot more sense now, but it also raises one more question. Both the hosts are only accessible within this machine. I'd like to send TCP packets to other hosts on the Internet, which doesn't seem possible with this setup.

Turns out it is possible, with the help this of NAT or Network Address Translation, which I'll save for another post as this one has a TUN of information already.