Running on OpenBSD - Multiple Gateway System (0x04)

Pantelis Roditis's picture

Here we are again with another episode of "Running on OpenBSD" that describes how we configured a system with multiple gateway interfaces, allowing equal distribution of connections among them. The system was asked by an Internet Cafe with multiple ADSL connections that needed a better solution than the one already in place.

We will describe the requirements and setup of the system we finally installed along with some notes on the ideas behind each decision. As always the system was OpenBSD and the provided ports.

Background

Our story starts in Greece where the price of leased lines is still prohibiting for small size businesses and the only means of getting a decent speed is by using ADSL connections. The Internet Cafe that we asked to help was using four ADSL connections, shared among 24 computers that needed to access the internet.

Each of those computers was logically segmented into groups of 6 computers. Each group was manually configured to use one of the four ADSL connections. However, this lead to a couple of obvious problems. A single group could utilize the entire internet connection while the rest of the ADSL connections was unused.

Furthermore, whenever one of the ADSL connections was having connectivity problems an entire group of computers was also unable to use the internet connection. This usually required to manually reconfigure the said systems to use one of the other available connections.

The internet usage was also another factor that needed to be considered. The users often times visited bandwidth consuming sites such as, YouTube, Google video and other similar services that utilised this form of streaming video and not only that, but as we all know most of the time were full of banners that delayed the overall loading time and caused pains to all the other systems that used the internet connection.

The requirements

The requirements were straight forward for us but there were some extra things that we needed to take into consideration before we moved into the implementation of a solution that we would suggest.

  1. All systems needed to have a single gateway.
  2. Advertisements and banners needed to be limited as much possible in order to minimize the network spent.
  3. There was a need for a web caching solution that would also allow caching of web pages along with famous video services.
  4. Many of the Internet Cafe users were playing on-line games (such as Battlefield, Lineage, World of Warcraft etc) and we needed to provide them with bandwidth since many games were experiencing connection problems whenever the internet usage was high.
  5. There is a desire to be able to view the current bandwidth utilisation on each network connection including an overall view of the remaining system resources.
  6. We had to utilise as much of the existing equipment as possible in order to keep the cost of the changes to a minimum.
  7. Finaly, all this would have to be implemented in a very limited time frame and during a period of "renovation" for the entire building.

The Solution

We suggested an OpenBSD 4.2 system with 5 total Ethernet interfaces. Four of them used to balance the bandwidth utilisation with the external world and one to keep the system connected with the internal network. The OpenBSD PF took care of the balancing of the requests through different interfaces each time with a very simple round-robin route-to syntax. PF also redirected transparently all requests for websites through a squid proxy server which was installed from the ports.

Squid's configuration was minimalistic apart from some parts that were required to allow caching of special content such as:

  • .exe files
  • .flv, .fla, flw and other Adobe Flash and Shockwave files.
  • .avi, .mov, .rm, .mpg and other known filetypes of video and audio files.

A small Perl script was used along with Squid in order to filter the remaining of the advertisements that were hiding inside URL's and weren't block by the hosts blacklist.

We used the default ISC Bind that ships with OpenBSD to cache and provide name resolution services for the local machines. Used a hosts blacklist to avoid spending our time with bandwidth wasting, advertising hosts.

For network usage and utilization statistics we took advantage of PF's internal functions again and with the help of pfstat we were able to produce nice looking graphs representing all the required parameters. We used three different generation policies, one every hour, day and month.

A comparison of the two network topologies makes the benefits in simplicity obvius.
Topology before Network Topology before and after Network Topology after

Final words

The simplicity of PF made it a clear winner. Within minutes our gateway was able to distribute the requests among the 4 interfaces. Our configuration included mostly bits from the PF FAQ.

For the Squid configuration part you can find more of the configuration examples at Caching YouTube Content.

The whole project was completed in a matter of a couple of hours at which point we were enjoying our pizza and fast internet.

I hope you enjoyed the reading, 'till next time "Stop slacking you lazy bum".

PS: The system is now in place for almost 6 months. There have been no problems whatsoever related with the system. Even the owners, which come from a Microsoft Windows background, are impressed by the fact they have one less system to worry about.

Average: 5 (2 votes)

Comments

Hi! This is great stuff.

Hi!

This is great stuff. I'm currently setting up an openbsd firewall with 2 external gateways (2 DSL lines) and I need some ideas on altq queueing like what you have. Could you post up your pf.conf or email it to me? (Feel free to mask the sensitive bits.) Thanks! :)
-T

Pantelis Roditis's picture

Hi, Unfortunately we didn't

Hi,

Unfortunately we didn't use altq for this setup. We only used route-to and some specific rules in order for the daemons, which run on the local gateway, to be able to use specific gateways. For instance we wanted only one of the four ADSL lines (the one with lower bandwidth) to be used for DNS and Web queries.

Nevertheless, if you still need the pf.conf for some hints feel free to let me know.

Pantelis
___
Dont prohibit what you cant prevent.

I'm just curious. Did you

I'm just curious. Did you use tables to hold the external gateways? I found I had to use:

pass in on { $int_if lo0 } route-to { ($ext_if1 ), ($ext_if2 ) } round-robin proto tcp from ($int_if:network) to any flags S/SA modulate state

Then I had a pfctl --t extgw1 -T replace 0.0.0.1 in my /etc/hostname.pppoe0 (since that's handled by the kernel)

The PPP was trickier. I needed /etc/ppp/ppp.linkup to do:
isp:
!bg /sbin/pfctl -t extgw2 -T replace HISADDR

Since both gateways are dynamic and can change if the DSL line drops, this saves lots of headaches.

The only problem now is.. if one of the lines goes down out of the two, 50% of the traffic goes into a blackhole. Do you have any scripting to detect when a line fails over and change the route-to statements accordingly?

Pantelis Roditis's picture

Hi, We did use tables but

Hi,

We did use tables but not for this purpose. Our gateways were static so we had no need for tables at that time.

Dont prohibit what you cant prevent.

Happy that you are back

Happy that you are back ... I really liked your "Running on OpenBSD" series. Really nice to see examples of commercial usage of OpenBSD.
Pantelis Roditis's picture

Hi Peter, Thanks for you

Hi Peter,
Thanks for you comment, it's good to be back and what's more to be welcomed back. I have quite a lot of episodes yet to complete, although tight schedule makes it very hard.

However, I'll do my best to keep them coming.

___
Don't prohibit what you can prevent.

Wow that is a great system.

Wow that is a great system. 6 months with no problems? i am lucky to keep mine going a day.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

Please solve the math problem above and type in the result. e.g. for 1+1, type 2.
The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • No HTML tags allowed

More information about formatting options