Run 2 Nebula lighthouses on the same machine

By | July 3, 2024
0 Comment

Do you want to use your Nebula lighthouse for 2 different overlay networks? It is (I think) technically possible for one nebula node to serve as a lighthouse for 2 different overlays, but it’s way simpler to run 2 different instances of the nebula software, one for each overlay. We can do this on the same machine by making a few adjustments to our basic setup. I’m currently using a $4/month droplet at DigitalOcean to run my lighthouses.

I assume that you are familiar with setting up a basic overlay network with nebula, so I will only go over the changes I made to run 2 lighthouses on one machine.

Also, I’m not sure what would be considered The “right” way to do this. I’m simply showing you what has worked for me. I welcome your constructive (and instructive) comments and criticisms regarding this.

Here are the steps:

  • Modify the /etc/nebula directory structure.
  • Choose a listening port for the second (or third or fourth) lighthouse.
  • Update your lighthouse config (config.yml) to reflect the new file locations and port.
  • Update your startup scripts.
  • Make sure to use the proper port on any node connecting to the second lighthouse.

Modify the /etc/nebula directory structure

In out basic setup, we have our certs, key, and config in /etc/nebula. For a multiple lighthouse setup, create a subdirectory for each lighthouse (so something like /etc/nebula/lighthouse01 and /etc/nebula/lighthouse02 or something descriptive for your situation). Put the certs, key, and config for each lighthouse in the appropriate directory. Your files will look something like this:

ls -R /etc/nebula
/etc/nebula:
lighthouse01  lighthouse02

/etc/nebula/lighthouse01:
ca.crt  config.yml  lighthouse01.crt  lighthouse01.key

/etc/nebula/lighthouse02:
ca.crt  config.yml  lighthouse02.crt  lighthouse02.key

Choose a listening port for second (or third or fourth) lighthouse

Since the lighthouse must be accessible from the Internet at all times, it needs to listen on a predefined port. Each lighthouse needs a different port. The default port for Nebula is 4242, so the first lighthouse will use that. I’m using port 4243 for the second lighthouse. Maybe i would use 4244 for a third.

Update your lighthouse config (config.yml) to reflect the new file locations and port

Each lighthouse should have a config.yml in it’s /etc/nebula subdirectory. There are 2 sections in each that need your attention, the “pki” section and the “listen” section. They probably look something like this:

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/lighthouse.crt
  key: /etc/nebula/lighthouse.key

listen:
  host: 0.0.0.0
  port: 4242

Obviously, here, “lighthouse.crt” and “lighthouse.key” will be values appropriate to our situation. If we continue with the “lighthouse01” and “lighthouse02” naming and we use port 4242 for lighthouse01 and port 4243 for lighthouse02, we’ll modify our configs as follows:

Update the config.yml in /etc/nebula/lighthouse01 to reflect the new directory structure and the appropriate port:

pki:
  ca: /etc/nebula/lighthouse01/ca.crt
  cert: /etc/nebula/lighthouse01/lighthouse01.crt
  key: /etc/nebula/lighthouse01/lighthouse01.key

listen:
  host: 0.0.0.0
  port: 4242

And do the same for the config.yml in /etc/nebula/lighthouse02:

pki:
  ca: /etc/nebula/lighthouse02/ca.crt
  cert: /etc/nebula/lighthouse02/lighthouse02.crt
  key: /etc/nebula/lighthouse02/lighthouse02.key

listen:
  host: 0.0.0.0
  port: 4243

Update your startup scripts

I’m doing this on Linux with systemd.

In the basic setup, the systemd service file is /etc/systemd/system/nebula.service which has something like the following contents:

[Unit]
Description=Nebula overlay networking tool
Wants=basic.target network-online.target nss-lookup.target time-sync.target
After=basic.target network.target network-online.target
Before=sshd.service

[Service]
Type=notify
NotifyAccess=main
SyslogIdentifier=nebula
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nebula -config /etc/nebula/config.yml
Restart=always

[Install]
WantedBy=multi-user.target

If you already have that service enabled, stop it and disable it:

sudo service nebula stop
sudo systemctl disable nebula.service

Rename the nebula.service file to nebula-lighthouse01.service (or create it with the above contents if you don’t already have this running) and then copy it as nebula-lighthouse02.service:

cd /etc/systemd/system
sudo mv nebula.service nebula-lighthouse01.service
sudo cp nebula-lighthouse01.service nebula-lighthouse02.service

In each file you’ll need to edit the “SyslogIdentifier” and the “ExecStart” lines. So, in nebula-lighthouse01.service, they will look like this:

SyslogIdentifier=nebula-lighthouse01
ExecStart=/usr/local/bin/nebula -config /etc/nebula/lighthouse01/config.yml

And you’re probably already know that those lines in nebula-lighthouse02.service will look like this:

SyslogIdentifier=nebula-lighthouse02
ExecStart=/usr/local/bin/nebula -config /etc/nebula/lighthouse02/config.yml

Now you can start and enable both services:

sudo service nebula-lighthouse01 start
sudo service nebula-lighthouse02 start
sudo systemctl enable nebula-lighthouse01.service
sudo systemctl enable nebula-lighthouse02.service

A now we have the lighthouses sorted. Make sure your firewall is allowing all the necessary ports (4242 and 4243) to reach this machine.

Make sure to use the proper port on any node using the second lighthouse

Configure your nodes to connect to the lighthouses. You’ll need to adjust the static_host_map section and verify the lighthouse section in config.yml. Under static host map, make sure you have the proper port for each lighthouse. For this example we’ll say that lighthouse01 has nebula ip address 10.100.100.1 and lightouse02 had ip address 10.100.200.1. So, in the config,yml for any node that connects to lighthouse01 these sections should look like this:

static_host_map:
  "10.100.100.1": ["internet.address.of.your.lighthouse:4242"]
lighthouse:
  am_lighthouse: false
  interval: 60
  # under hosts we need the nebula ip of the lighthouse
  hosts:
    - "10.100.100.1"

And for any node that connects to lighthouse02, these sections should look like this:

static_host_map:
  "10.100.200.1": ["internet.address.of.your.lighthouse:4243"]
lighthouse:
  am_lighthouse: false
  interval: 60
  # under hosts we need the nebula ip of the lighthouse
  hosts:
    - "10.100.200.1"

And that’s it. Let me know how it went.