Setup a network bridge for your LXC containers with lxc-net

After having played around a bit with LXC and discovered its main features, you may want to have a proper network setup for your containers.

There are multiple network setups possible and multiple ways to implement them. In this post, we are going to setup a bridge, using lxc-net. It requires very little configuration and should be enough for a simple LXC architecture.

More details about this bridge setup:

Containers will have an IPv4 within their own subnet
Containers will be able to access each other within this subnet
The host will be able to access the containers trough this subnet
Containers will have access to the internet thanks to the bridge interface

Note that I'm using Debian 9 for this tutorial. Also, if you're using LXD to manage your LXC containers, this isn't necessary as it does everything automatically.

Install lxc-net#

That's pretty easy, as lxc-net is a part of LXC, it's already installed.

lxc-net uses dnsmasq to manage DHCP and DNS.

To install it, run:

apt install dnsmasq-base

However, lxc-net spawns its own dnsmasq process, so if you install dnsmasq, it will run on its own and cause a conflict with lxc-net, for example :

systemctl restart lxc-net
systemctl status lxc-net

....
failed to create listening socket for 10.0.3.1: Address already in use
Failed to setup lxc-net.
...

We need the dnsmasq binary and not the service, so we only install dnsmasq-base.

If you're using a distribution without dnsmasq-base, stop and disable the service:

systemctl stop dnsmasq
systemctl disable dnsmasq

(This is basically the same thing as uninstalling dnsmasq on Debian)

I'm adding this in the post in case someone else encounters this problem, which I did because I installed dnsmasq in the first place.

That being said, back to the tutorial!

Configure the bridge#

The bridge interface will not be configured by default.

root@host ~ # cat /etc/lxc/default.conf
lxc.network.type = empty

Replace the line by this:

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

lxcbr0 is the name of our bridge on the host.

Now, tell LXC to use the bridge. Create /etc/default/lxc-net and put this in it:

USE_LXC_BRIDGE="true"

Restart lxc-net and make sure it's running:

systemctl restart lxc-net
systemctl status lxc-net

The lxcbr0 interface should be up:

root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0 inet 10.0.3.1/24 scope global lxcbr0\ valid_lft forever preferred_lft forever

Our newly created containers will now be getting an address within the 10.0.3.1/24 subnet.

Test your bridge#

Let's use our bridge!

lxc-create -t debian -n c1
lxc-create -t debian -n c2
lxc-create -t debian -n c3

lxc-start -n c1
lxc-start -n c2
lxc-start -n c3

root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.3.32 -
c2 RUNNING 0 - 10.0.3.200 -
c3 RUNNING 0 - 10.0.3.77 -

There you have it! Do some tests with your containers:

root@host ~ # ping -c2 10.0.3.32
PING 10.0.3.32 (10.0.3.32) 56(84) bytes of data.
64 bytes from 10.0.3.32: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 10.0.3.32: icmp_seq=2 ttl=64 time=0.087 ms

--- 10.0.3.32 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.043/0.065/0.087/0.022 ms

root@host ~ # ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200) 56(84) bytes of data.
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from 10.0.3.200: icmp_seq=2 ttl=64 time=0.067 ms

--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.067/0.069/0.071/0.002 ms

root@host ~ # lxc-attach -n c1

root@c1:~# ip -4 -o a
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
15: eth0 inet 10.0.3.32/24 brd 10.0.42.255 scope global eth0\ valid_lft forever preferred_lft forever

root@c1:~# ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200): 56 data bytes
64 bytes from 10.0.3.200: icmp_seq=0 ttl=64 time=0.077 ms
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.088 ms
--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.077/0.082/0.088/0.000 ms

root@c1:~# curl -I https://angristan.xyz/
HTTP/2 200
server: nginx
date = Thu, 15 Feb 2018 23:32:51 GMT
content-type: text/html; charset=utf-8
content-length: 17443
vary: Accept-Encoding
x-powered-by: Express
cache-control: public, max-age=0
etag: W/"4423-aJyAbkYavS/1P5xFFFgdnme4hCQ"
vary: Accept-Encoding
x-cache-status: EXPIRED
strict-transport-security: max-age=31536000; includeSubDomains; preload

Neat, right?

Use another subnet#

Here is the default subnet and DHCP range used by lxc-net:

LXC_ADDR="10.0.3.1" #Address of lxcbr0 on the host
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"

To use something else, define these options in /etc/default/lxc-net.

For example:

LXC_ADDR="10.0.42.42"
LXC_NETWORK="10.0.42.0/24"
LXC_DHCP_RANGE="10.0.42.100,10.0.42.200"

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3

Check the result:

root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0 inet 10.0.42.42/24 scope global lxcbr0\ valid_lft forever preferred_lft forever

root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.42.185 -
c2 RUNNING 0 - 10.0.42.153 -
c3 RUNNING 0 - 10.0.42.117 -

Use static IPs#

As you may have noticed, by default, new containers will get a random IP address within the subnet.

If you want to define your own IPs, it's simple.

Create /etc/lxc/dhcp.conf, and define you CTs as following:

dhcp-host=<ct-name>,<ip>

Example:

dhcp-host=c1,10.0.3.11
dhcp-host=c2,10.0.3.12
dhcp-host=c3,10.0.3.13

To tell lxc-net to load the configuration, add this line in /etc/default/lxc-net:

LXC_DHCP_CONFILE=/etc/lxc/dhcp.conf

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3

root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.3.11 -
c2 RUNNING 0 - 10.0.3.12 -
c3 RUNNING 0 - 10.0.3.13 -

Success! 🎉

Define a domain#

A quick tip: you can define a default domain for you containers using lxc-net.

Add this in /etc/defalt/lxc-net

LXC_DOMAIN="domain.tld"

For example:

LXC_DOMAIN="angristan.xyz"

If I create a new container, it now has a FQDN:

systemctl restart lxc-net
lxc-create -t debian -n ct
lxc-start -n ct
lxc-attach -n ct

root@ct:~# hostname -f
ct.angristan.xyz

Route a host port to a container#

With our bridge, containers can access to the internet, but cannot be accessed.

We can use the iptables NAT routing table to map a host's port to a container's port, with the following command:

iptables -t nat -A PREROUTING -i <host_nic> -p tcp --dport <host_port> -j DNAT --to-destination <ct_ip>:<ct_port>

To be more specific, we're mapping a port from the host's public interface to the container's IP. Obviously, if you want your container to be accessible from the internet, use the interface (host_nic) where you public IPv4 is mounted.

As an exemple, I want the web server of my c1 container to be publicly accessible.

host_nic: eth0
ct_ip: 10.0.3.11
host_port: 80
ct_port: 8080

We'll use:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.11:8080

You can now reach your container's port 8080 trough your host's 80 port (via its public IP).

IPv6#

Sadly, I have not succeeded in setting up IPv6 with lxc-net, with both my public and private IPv6 blocks. Even tough the feature has been added in 2015, there is not documentation whatsoever and I have not found any tutorial.
It seems another solution is to set up our own bridge manually.

Sources:

Comments

12Atom feed

Manuel Ignacio López Quintero Feb 19, 2018
Attractive tutorial, easy to read, well explained and very didactic.

I was struggling to give internet to my containers. And, above all, you clear me some basic net concepts.

Thank you very much!
1. StanislasAuthorFeb 19, 2018
  I'm very glad it helped you, thanks for you comment!
NekrofageFeb 21, 2018
Merci Angristan pour ton tuto qui a fonctionné du 1er coup !!! \o/
1. StanislasAuthorFeb 21, 2018
  C'est le but \o/
FredMay 22, 2018
Please How do I apply this in centos
AnonymousOct 21, 2018
thank you. you tutorial is very simple. thanks to you I saved a lot of time because I could not do it with the Debian lxc wiki.
MaxJan 12, 2019
I wasn't able to assign static ip to containers with your method.

I did this instead :

Set network configs in /config:

And set manual interface configuration in /etc/network/interfaces: (in the container ofc)
AnonymousMay 22, 2019
I appreciate it's not your fault, but what you've described is not a bridge. A bridge connects two networks at the MAC level. This is a router. It's very frustrating because I'm trying to find online documentation that tells you how to allow LXC containers to access the "real" network, with their own addressable IP addresses, and because the LXC maintainers do not know what a bridge is, 99% of the links are completely inappropriate. sigh
1. AnonymousOct 29, 2019
  LXC uses another package for networking -- bridge-utils/dnsmasq. Perhaps better docs could be found under those. They're not network admins -- they got the terms wrong.
2. AnonymousOct 31, 2019
  Seems like lxcbr0 is indeed a bridge in nature? Because with command "brctl show" you can check it and the interfaces it uses to connect containers. I am a new learner...
JasonOct 31, 2019
It really helps! Thanks!
Imran CApr 15, 2020
For Debian 10 "Buster" the config is slightly different, I used this to get it working https://gudok.xyz/lxcdeb/