Cdtjkyj

I'm working on NixOS, creating some functions in C to programmatically manipulate veth pairs and assign IPv6 addresses in order to create a program for setting up a network for containers on a single host.

To do so, I'm using libnftnl and libmnl to handle the fine details of constructing rtnetlink packets and setns() to set the network namespace of the calling thread to the correct network namespace before performing actions such as bringing an interface up.

The general flow of the program goes like:

- create veth pairs in the host network namespace

- move one end of each veth pair to the corresponding network namespace

- set all the veth endpoints in the host network namespace to UP

- call `setns()` into each other network namespace to set

  the loopback interface and vethpair end to UP

- add an IPv6 address to each of the veth ends in the host network namespace

- call `setns()` into each other network namespace to set the IPv6 address

  of the other veth endpoints

The Problem

When I run the program, IPv6 addresses in the host network namespace aren't always assigned to the interfaces. All the previous steps occur and exist and even the IPv6 addresses are assigned to all interfaces in other network namespaces but not always in the host network namespace.

This happens when I run the program, delete all the interfaces and rerun the program within about a minute of each other.

Some solutions that I've come up with are that if I make the program sleep for, say 5 seconds, before assigning addresses to interfaces in the host namespace, I can run the program, delete the interfaces and rerun the program nonstop and the addresses are all assigned. I can also wait a while after deleting the interfaces before rerunning the program and this will also cause all IPv6 addresses to be assigned correctly.

Other weird stuff

When running the program, there are times when some IPv6 addresses in the host network namespace are added and some aren't. The netlink ACK message returned for the packets also indicate that no error occurred so I can't tell that an address has failed to be assigned until I run ip address after the program exits.

Since this program is eventually going to be part of a larger container orchestration system, I'm looking to see if there's a way to either detect or prevent this kind of error.

I've thought about it potentially being duplicate address detection or the IP address being still stored somewhere in the kernel for a while after being delete but I receive no error from the netlink module.