Running on OpenBSD - The Conclusion (Part 4)
In a world that three operating systems dominate (Windows, Linux, MacOS) and alternative sounds weird we gave a run at OpenBSD as our operating system from end-to-end. The following document is the fourth and final part of a four part paper that describes how we managed to setup our entire company using only OpenBSD and its provided ports and tools.
We hope you enjoy the reading.
Why we got into this adventure in the first place? When we formed Echothrust Solutions, we needed to setup our servers (web, mail, firewalls at first). We could use a hosting provider (and in matter of fact we did and still use) but we felt that we had to maintain control of the servers. We had to create the entire infrastructure from start, and we wanted to make it right in the first place. As we started to gather our detailed requirements it was obvious that we had a lot of servers to setup. Since I was already a fanatic OpenBSD user I proposed to go with it. Once we had decided that our servers would go with OpenBSD we added the workstations into the picture and verified that the system still meets the initial requirements.
Requirements and solutions
Even though it was decided before hand, that we will go with OpenBSD all the way, we had some prerequisites that had to be worked out.
- Security: Security is highly required. We keep customer contacts, projects etc, that need to be secured. The OS has to be resistant in attacks (worms, exploits, trojans) and must provide enough tools to help maintain a healthy system (such as firewalls, logging, etc).
- Minimal and Configurable: We needed the system to offer only the bare minimum and allow us to build on top of that. So a highly configurable and customizable OS and set of applications were required.
- Must support existing hardware: We didn't want to replace the systems, although, we were willing to update certain parts (memory, hard drives, graphics cards).
- Maintenance: We needed a system that required the minimum of maintenance over the time, be it updates, clean-ups, installations or administration.
- Costs and Secondary Costs: Costs associated with the OS. Secondary costs are considered things like software which we would have to purchase, including subscriptions and support. Furthermore, as secondary costs we consider possible hardware updates.
Keeping those in mind, the questions were mostly directed towards the OS rather than the applications. The Open Source tools, that we needed, were available for most of the operating systems.
Applying those to the four OS's (OpenBSD, Linux, Windows, MacOS) it was clear who the winner was.
- Windows: Licenses for server, workstation and support. Furthermore, we were not quite sure how well some of the hardware would handle the Windows Server.
- MacOS: Simply impossible, it would require us to purchase new set of machines which is not what we had in mind.
- Linux or Free/NetBSD: Linux could serve our purposes as well. However, certain tasks were unreasonably complicated compared to BSD equivalents. Furthermore, on the available Linux distributions everything seems to be overloaded with features which we would have to disable. The rest of the BSD's could serve our purposes as well, but we had no familiarity with them before and thus we chose to go with options that we were familiar with.
- OpenBSD: Secure by default. Minimal base footprint (something like 400mb). Easy to automate (specially for our installation purposes, we had to setup more than 30 systems in total). Update takes less than an hour including packages installed. Manual pages that actually HELP. Simple firewall with nice features (specially for workstations user specific rules).
Security and Maintenance
Security is build into OpenBSD since its very creation. The OpenBSD team strives to provide a secure operating system and the vulnerability numbers seem to agree. Furthermore, firewalling with Pf on all our servers and workstations, strict permissions and encryption gave us our peace of mind.
As far as maintenance goes, the collection of pkg_* utilities and OpenSSH provided an easy way to maintain and update packages whenever it was deemed necessary. The time spend was minimal, system updates are been made available every 6 months, packages installed and updated with one command in less than an hour (will be longer with network installations). This meant that we did not have to invest much time for updates every so often.
What you realise with OpenBSD and the packaging system is that it is so easy to upgrade your system, that you area actually looking forward to the day a new version is made available.
Issues and Difficulties
Although we enjoyed the whole deal with OpenBSD, we did face certain issues and difficulties.
- Hardware: OpenBSD supports a wide range of hardware but not all devices were able to work. The supported Webcams, USB multimedia, sound and video devices seem to be limited in numbers. We did not see this as a limitation of OpenBSD, all the times. If a vendor provided documentation for a device, the driver was there. Other drivers were simply the fruit of hard work from OpenBSD developers who spend time reverse engineer existing drivers in order to provide an open alternative. Although the last thing we wanted was to run blobs, the fact still remains that certain driver categories are not a priority on OpenBSD.
- Software: OpenBSD comes with more than 4000 packages and the number is always increasing. Not everything is there, as one might expect. We did came to a stop when we tried to find a VoIP client and when we managed to make one functional, we had hardware sound issues, this was one of the worse cases though. When we started installation, the available version was 3.9 and no OpenOffice packages were available. The emulation hack was unstable, so we had to use alternatives, and this is how we came to love LaTeX. LaTeX stayed with us even after OpenOffice became available, which now we only use for documents that we receive. Flash players are another issue, even though there are some ways around this (opera with flash plugin, linux emulation), we didn't feel such a way was appropriate.
- Compilations: Something that many OpenBSD users have come across is the fact that a very large portion of projects simply fail to compile under OpenBSD. The reasons vary from compiler incompatibility, application OS specific (eg. for Linux), badly coded application, to sane limits of OpenBSD. Again, depending on the application and how badly we needed it, we had to invest time on dealing with those issues which not always ended in success.
- Latest versions: Although, the ports are updated constantly (under -current), the speed that projects make new versions available, has an impact on OpenBSD. Often the available packages are a couple of versions behind the official releases and this makes the system look outdated. However, when a security update was issued for a project, the package maintainers provided updated packages fast enough.
- Performance: Great performance improvements have been observed throughout OpenBSD's development, but still this is a "grey" area. I can understand that all those security mechanisms will have an effect on the overall performance and although Linux, Free/NetBSD certainly perform better on certain areas, the difference is not that big to cause a noticeable annoyance. However, although ways to improve performance are available, by tuning system parameters, we were feeling more confident with the defaults provided with OpenBSD.
- Sacrifice for the team: All those new things and applications had to be discovered from a number of available packages (did we mention more than 4000?). This meant one of us had to sacrifice time and weekends in order to determine which tools meet our needs and how usable are they. Many of the tools were already familiar to us (like gaim, firefox, etc), others completely new.
- “Isolation”: I don't know if this has to do with all those security mechanisms or the fact that not much software is written with BSD in mind. But when you surf the net you rarely come across OpenBSD (even as a word). Very few packages mention that support OpenBSD (from those not available on packages). You can't help but feel a bit of isolation... No spam, no worms, no virus, nothing to keep you on your toes.
When we initially started with the idea, it was really the beginning of our existence as Echothrust Solutions. Like most startups, budget was limited and everything had to be specially tailored to meet the particular economic needs. The systems were already there, so acquiring new ones was out of the question. We needed to somehow make everything work as it was and make it work reliably, even if that involved a lot of voodoo rituals.
However, the direct costs are not always the scary ones. The costs of maintaining, updating and keeping your systems clean from virus, malware and exploits could make the most cost effective solution look dangerous.
Although Linux or another Open Source operating system could virtually offer similar benefits we argue to differ.
Throughout OpenBSD's development, the level of security and quality was clearly high on the priorities of the developers. Once you get familiar with the OpenBSD "concept" you get to appreciate the quality. We did not have to perform extensive audits for every package we used, since most of those are already checked at some degree by the port maintainers and this can be a huge time saver.
Another thing that made us love OpenBSD was stability. We didn't come across any system crashes, even when we tried to, the darn thing just stood there. At some point, the systems were only rebooted when it was time for upgrade and this says a lot about stability.
Our systems run smooth and safe and even though things were a bit “difficult” in the beginning now our users cant get their hands off their system.
The minimum downtime for security updates, patches and package distribution has proved our trust in OpenBSD is right.
I hope you enjoyed the reading. Stay tuned for more detailed updates on how we configured our servers and services, in detail, with OpenBSD.