Gentoo Universe News

<UU> Somedays, I think why can't we have computers which just work.
<UU> But then I remember that I am a Computer Scientist.
<UU> So, yeah, I guess I understand why.
<Nirbheek> :D
Quite related to GNOME, really.

Gentoo Prefix is still alive and going strong. In my opinion, Gentoo Prefix remains a strong point of Gentoo Linux and really establishes that Gentoo Linux is a metadistribution. In this post I want to focus on the numbers. The number of packages in the Gentoo Prefix tree, specifically. But first, a history lesson. It wasn’t until EAPI3 in Gentoo that “allowed” Gentoo Prefix variables into the main Gentoo Linux tree. That was in late 2011, but Gentoo Prefix existed much before then, all the way back to 2006 (at least). Before EAPI3, the prefix team made slight modifications to ebuilds and placed them in a repo and called it the tree of packages for Gentoo Prefix. This worked fine, but we had growing pains. The major issue was that we were getting too successful to manage the increased contributions from users. In other words, as the number of “forked” packages grew, the amount of maintenance time increased greatly – this is due to the fact that it is a chore to keep our forks synced. At least, a large chore for a small team. This is why we looked for help and adoption from the other pool of 200 Gentoo Developers, hence EAPI3 and beyond. Since supporting Gentoo Prefix is not a big use of overall developer time, this has gone over quite well in my opinion – yes, there are some pain points at times I do realize. Enough history, here are the numbers:

• Number of packages in Gentoo Linux: 15554 packages in 154 categories.
• Number of total* packages in Gentoo Prefix: 9483 packages in 154 categories.
• Number of KEYWORDED packages in Gentoo Prefix: About 3000 for the most popular arch
• Number of packages still NOT in the main Gentoo Linux tree: 369 packages

* The total packages in the tree also contains non-keyworded packages because that just makes life simple. Once packages started migrating to the main tree, I helped think of this “whitelist” concept. The short version of the whitelist is that if a package is listed in that text file, it gets included in the Gentoo Prefix tree as a direct copy of the version in the Gentoo Linux tree. The presense of the package in the old repo means that it is used instead. Eventually, this concept will go away and we will overlay the Gentoo Linux tree directly.

So why is it taking so long to migrate ALL packages to the Gentoo Linux tree? Well, that is where the rubber meets the road and we get into roadblocks. A roadblock for us could be a number of things, such as a disagreement with the Gentoo Linux maintainer, some patches existing that we don’t feel are a good fit for Gentoo Linux, or even us being lazy and not submitting stuff to upstream. We also don’t want to push invasive changes to Gentoo Linux for critical packages, like the toolchain for example.

It has long since been our agenda to not add anymore packages to the old repo and going forward only adding new stuff to Gentoo Linux directly. I hope we can make a dent in those remaining 369 in 2012!

If you do, maybe you want to consider proxy-maintaining it as it now on its way out. Upstream has a much newer version available, and we in the Proxy Maintainers team will be glad to steer you in the right direction when you need the help.

Just send us an email.

I keep getting e-mails asking me why app-text/xpdf is masked for removal from the portage tree. It's getting too much to reply individually, so let me sum up the situation here in a blog post.
# Andreas K. Hüttel <dilfridge@gentoo.org> (27 Jan 2012)
# Has developed into an unmaintainable mess, and everyone who
# knows about it is either retired or missing in action.
# Several minor bugs and one ugly security issues (#386271).
# Masked for removal because of lack of maintainer.
# Please try app-text/epdfview as light-weight replacement.
app-text/xpdfXpdf is a package with a long history, and in a way a strange remnant of bygone times. Since PDF rendering is a function that many different programs could use, some years ago the Poppler library was forked from the Xpdf codebase. By now, Poppler is a much more active project, and used by dozens of packages in the Gentoo portage tree, all the way from LibreOffice and PDFTeX to Calligra, GIMP, and e.g. Okular or Evince. Being the more active project is important in this case, because PDF files are frequently shared and distributed and PDF rendering is thus a security-relevant task.
The original Xpdf remained independent of Poppler, not using the library - with the effect that every now and then security bugs kept popping up. Some time ago, some Gentoo developers started modifying and patching Xpdf to use the Poppler library. What resulted was the complicated construct that right now noone here is willing to maintain anymore. (Otherwise some Gentoo developer would have contacted me in the meantime.) Implementing a version bump to a more recent Xpdf version is a non-trivial task because all the Gentoo-specific patches have to be reviewed and if necessary rewritten.
Thus, app-text/xpdf needs to go the way of the dinosaur. Two alternatives exist, but both do not seem realistic at the moment:
1) We could go back to the original, unpatched Xpdf from upstream. I'm not going to do it, and I doubt anyone else of the Gentoo devs will.
2) Rogério Brito has started maintaining a fork of Xpdf at Github, which uses the Poppler library. However, there is no released version yet, and as he told me myself, he's rather busy in real life right now...
In the meantime, please try one of the following packages:

• app-text/zathura (based on poppler & gtk+)
• app-text/apvlv (based on poppler & gtk+)
• app-text/epdfview (based on poppler & gtk+, ~unmaintained)
• app-text/mupdf
• app-text/gv (based on ghostscript)
• app-text/evince (based on poppler, Gnome application)
• kde-base/okular (based on poppler, KDE application)
• app-text/acroread (yes I know...)

Ironically, the first mail reply to the last-riting of xpdf was from one of our security team members, promising me a beer the next time we meet in person. Only afterwards the complaints started.

sed -i 's/gregkh@suse.de/gregkh@linuxfoundation.org/g' .addressbook

I started looking at ModSecurity when I wanted to implement a Uesr-Agent based antispam method which has proven time and time again working quite well to the point I started publishing the ruleset which takes care not only of working as an antispam method, as well as a way to avoid tons of bad crawlers from finding my email addresses and so on.

When I first proposed this kind of filtering I received quite a few complains, that the HTTP protocol didn’t define the User-Agent in such a way, but thanks first to EFF’s Panopticlick – demonstrating clearly that the “anonymised” requests are not as anonymous as their perpetrators would expect them to be – and most recently SpiderLabs’s work I am now fully certain that I took the right road.

I’ve spent a bit more work on the rules this week, to make them further resilient to fake the requests such as those coming from scriptkiddies’ tools such as the HOIC tool described in the SpiderLabs’s blog post linked above. One of the most interesting detection I came up with is for real Chrome requests: while it seems to me like Google itself does not leverage it, Chrome as of version 18 is still implementing their own proposed Shared Dictionary Compression for HTTP even though I don’t think it’ll ever be used in the real world. Being the only browser actually requesting such an encoding, I can easily assume a connection between the two — this was only disattended by Epiphany, which in its most recent versions declares to be Chrome… which means you then have a browser claiming to be another (Chrome), which in turn claims to be a third (Safari), which uses an engine (KHTML) claiming to be the same as another (Gecko), all the while declaring it’s all compatible with Mozilla/5.0.

One issue I found while doing this work had to do with Android. For both versions 2 and 3 (is somebody really hoping to use Android 1?), the (default, AOSP) browser sends a full-fledged HTTP request, which among other things include an Accept header. This is what every browser I ever tried does, to the point that ModSecurity’s own Core Rule Set assigns negative points to requests coming without one; in my ruleset it’s further tightened by checking whether the request is purportedly from a known browser, and if so rejecting it if it doesn’t include that header; this worked up to now — note that requests coming through a Proxy, making that explicit through a Via header, are not validated against these checks simply because many proxies are known to muck with the headers.

Anyway as I was saying this is disattended badly by Android 4 (up to 4.0.3, and CyanogenMod as well); it might have started as a way to minimise the bandwidth usage, but for whatever reason in this version, the AOSP browser does not send an Accept reader at all — actually it seems like it dropped most of the headers that it was sending before and that are not strictly necessary for the server to process the request. I could have sworn that Accept was mandatory for the HTTP protocol, but it seems that either I was totally mistaken, or it was only noted in some recommendation that never made it to the standard. The ruleset now exonerates Android 4 from that particular test, but I’m not really too happy about it.

But that’s definitely not the only thing that is out of place with Android. Indeed, if you take an HTC Android device, the browser you open is not the AOSP one, but it’s HTC’s own implementation. This version … does not fully declare itself as an Android device, using a browser compatible with Mobile Safari. Instead, what it reports itself as is a complete Safari, and not in the way that Chrome does it, but by pretending it’s Mac OS X 10.6.3 running on an Intel Mac. Honestly, that’s way crazy to do.

There are a few more things that I hope to be able to handle in my ruleset to make it even tighter, without adding substantial false positives. This means not only fewer spam comments, but also fewer crawlers finding our email addresses, and fewer risks associated with Denial of Service attacks, distributed or not.

If you would like to help with the ruleset, you can find it on Flattr where it’s depressingly stopping at only two clicks. If you would like to use the ruleset, you can find it on GitHub and you can use it for free, obviously.

As discussed in the last Gentoo Qt meeting, we moved our overlay from gitorious to git.overlays.gentoo.org. This is going to be the final move, I promise

Along with that, we decided to change the overlay from qting-edge to just qt. Layman list is alreay updated, so if you still have the old one, you should remove it and add the new one:

# layman -f # layman -d qting-edge # layman -a qt

Keep in mind that this overlay contains mostly live ebuilds of Qt (branches 4.7 and master), so make sure that you really need it before blindly adding it (the same applies for the kde overlay). Enjoy!

1. Roll call

johu, hwoarang, pesa, tampakrap, wired

2. Qt 4.8

* cairo fails to build, patched ebuild available in qting-edge, #380013

Cairo build issue is fixed in qting-edge overlay, will be moved together with Qt 4.8.0 to tree.

* qt now defaults to the raster graphicssystem, we should remove raster USE flag, #398283

Wired created a eselect module to choose the Qt graphicsystem. Raster is default, other selectable are opengl, openvg and native. Raster use flag is not needed anymore, qt-gui depends on the new eselect module.

* do we really want to keep qpa USE flag?

qpa and c++0x will be masked in tree.

* are we going to fix #363939 for 4.8?

Wired fixed this bug in qt 4.8.0. Qt 4.8 will be moved to tree on next weekend. Dilfridge prepares kde-base/kstyles-4.7.4 to be rebuild together with Qt 4.8.0 to prevent crashes in KDE apps with Oxygen style.

3. Minor arches and Qt >= 4.7

Upstream supports official amd64, arm and x86, but other arches also considered in configure script. Keep stable keywords for minor arches in Qt 4.6. Wait for minor arches arm, ppc, ppc64 in current stabilization in Qt 4.7.4. Drop sparc keywords in Qt 4.8.0.

4. Overlay migration to git.overlays.gentoo.org

Tampakrap will set up overlay on git.overlays.gentoo.org on next weekend. The new overlay will be renamed to qt instead of qting-edge.

5. Open bugs

* #398885 qdoc3 broken on arm

We will ask the reporter if it works when he builds manually by providing him a configure command to make sure he tries the proper build.

* #394533 Libreoffice crashes in qt on exit

Can’t be reproduced with Libreoffice 3.5.0.1, seems to be resolved by upstream.

* #392433 desktop file name issues

Will be fixed in Qt 4.8.0, so that qt-gui and qt-assistant no longer pass absolute paths to make_desktop_entry().

* #388551 qt-gui[gtkstyle] should depend on gnome-base/libgnomeui-2

We will add a elog message in qt-gui[gtkstyle] saying that for things to work you either need libgnomeui or that variable set properly in your env.

* #382559 qt_mkspecs_dir() returns bad spec directory

The bug will be marked as RESOLVED WORKSFORME, because we can’t reproduce it. Additionally we change the eclass not to use LIBDIR in favor of get_libdir() after Qt 4.8 hits the portage tree.

* #359391 qt4-build.eclass should check for —buildpkgonly before downgrade sanity check

Resolution will be RESOLVED WONTFIX. Sanity check is there for a reason. It’s not a matter of source or binary downgrade.

Since one hour ago, Qt-4.8.0 is in Gentoo portage tree. New major release so lots of new (or broken) stuff. The most important feature in this release is the integration of a new eselect module. This module will allow you to set your default graphics engine without the need to recompile Qt (x11-libs/qt-gui to be precise) from scratch. So, provided you have qt-gui-4.8.0 installed, you should be able to use the eselect module as follows:

hwoarang@mystical ~$ eselect qtgraphicssystem list Available Qt Graphics Systems: [1] native [2] opengl [3] raster *

(note: if you have x11-libs/qt-openvg installed, one more option should be available)

Simply select your graphics system of preference, and then logout and login again.

hwoarang@mystical ~$ eselect qtgraphicssystem set 2 Setting opengl as your active Qt Graphics System... done Please logout for changes to take effect.

Thanks to Alex(wired) for the eselect module implementation.
Enjoy ;-)

 

 

A small notification to tell you that the SELinux policies that were pushed to the main tree 30 days (or more) ago have now been stabilized (none of them introduced problems, although some of them have other bugs still open which are either fixed in ~arch or will be fixed in the hardened-dev overlay soon). I’ll be working on pushing an additional set of changes to hardened-dev overlay today as it includes fixes for openrc that are quite important, and might even push this to the tree faster than usual.

The reference policy is also working on a new release, so the moment it is released we will be picking that up as well (give or take a month, since my availability will be a bit less the next month).

In this podcast, create a best off cd with soundconverter and gnomebaker. The new Gentoo LiveDVD with persistance. The Northeast Linux Fest Saturday March 17, 2012, Worcester MA. Samsung ML3312 and Linux plus an Interview with Milan Kazarka.

Links

Northeast Linux Fest | Saturday March 17, 2012.
http://www.northeastlinuxfest.org/

Gnome Shell Extensions
https://extensions.gnome.org/

Gentoo 12.0 LiveDVD
http://www.gentoo.org/news/20120102-livedvd.xml

Samsung ML-3312ND
http://www.samsung.com/us/support/downloads/ML-3312ND/XAA
http://gpo.zugaina.org/net-print/samsung-unified-linux-driver

Interview with Milan Kazarka
http://www.gentoo.org/news/20120119-milan-interview-announcement.xml

Download

You might remember that a couple of years ago I ranted about my choice of a Dell laptop — I have not found the time until now to write a full retraction of that post, but you might have guessed that I’m not that bothered by the laptop anymore.

Indeed, after a few rough months, the laptop is working quite nicely nowadays; not only the issues with PME I reported were solved a version of the kernel in or two, but also nowadays gentoo-sources have a (patched) experimental driver for the touchpad that lets me disable it exactly like I wish to. After a firmware upgrade (which is unfortunately only available for Windows, but it’s a small price to pay), both the contactful and the contactless smartcard reader interfaces work fine, the SD card reader works nicely with modern kernel, and so does the soundcard (both speakers and microphones). Even the HSDPA modem (that I bought last year, separately, and was quite easy to set up!) works fine on Linux, even though I haven’t found a way to set up the GPS, or to read/send SMS, not that I care about the latter.

Indeed I haven’t run Windows in there for quite a bit, especially since last time I tried to repartition it I couldn’t get grub2 and Windows 7 to play well together, so I just let it “rotting” for the moment, and I’m now honestly considering whether I want to keep Windows 7 in there – it has a few uses for me at customers’, other than updating BIOS and various devices’ firmware – or just install an SSD and be done with it. Third option would be to find an HDD-in-Optical-Bay adapter and get an SSD for Linux and a (pluggable) HDD for Windows 7.

Anyway, after all this I’m pretty happy with Dell, to the point that I both started suggesting it for my customers, and got a few more things from them (namely a Vostro 3750 laptop to use for Windows development, and an U2711 monitor). Why did I change my mind so completely? Mostly because I have seen how other vendors seem to make it more and more inconvenient to use them for anything but looking at facebook.

Take HP: I had to downgrade a laptop for a customer last week, from Vista to XP. It was not the first time I did that, and not the first time I had to do so to an HP laptop.. but this time it got even worse than usual. Let’s ignore the fact that HP pretends that a ton of their “softpaq” packages only work on Vista (while they contain the XP drivers as well); at the end of the day, the BIOS is enforcing some stupid policy on the HDA-based soundcard… I was able to get it running by using the devcon.exe command from Microsoft and making it reset the PCI ID of the soundcard at each Windows startup, which makes it work nicely.

Or take Gigabyte, which usually has a decent support for Linux: yesterday I built a computer for a friend of mine, with a Gigabyte GA-970A-UD3 motherboard; he’s running Windows 7 there, but as usual I wanted to write down the list of components and settings with lshw, so I plugged in my usual SysRescueCD thumbdrive and … it didn’t boot. The same goes for the CD-Rom version; FreeDOS and Windows 7 boot cleanly, so my first guess is that there is something wrong, or at least different, in the way Syslinux boots. Contrarily to the kind of replies I received on twitter, I don’t think that Gigabyte is “not supporting Linux” given that they do list Linux support on their website for this board, more likely there is something funky with SysLinux.

But today’s hall of shame entry is quite enraging: Packard Bell (which has been bought by Acer a few years back) has a netbook line that is called “dot”; an acquaintance of mine received a “dot S” device that is actually a DOT_SE3/W-100IT, which comes with 1GB of RAM, and he asked me if I could get more RAM on it. Sure usually I can — in this case the maximum available is 2GB. He brought the device to me and I tried to find how to open it…

There are no instructions, it’s hard to find anything; DuckDuckGo does not find anything useful, while Google’s “did you mean?” feature made it impossible to find something related to SE3, with many more sources for SE2 and simple S instructions. It goes without saying that neither is anywhere near similar to this one. At the end of the day it seems like the only way you have to access the backside panel under which the memory is, is to disassemble almost the whole motherboard. Not going to.

You probably remember my previous notes about Wordpress, FTP and the problem with security. At the end after a (boring) set up session I was able to get vsftpd provide FTPS service, which should be usable both by Wordpress and by Dreamweaver, so that my friend the webmaster can upload through it directly.

This is important because as it happens I have another prospective customer who’s going to run Wordpress, and FTPS now start to look more interesting than SSH, as it doesn’t require me to give shell access to the server either.

Unfortunately I’m a bit worried (maybe more than I should be) for the use of standard passwords rather than certificates or keypairs for authentication. Which meant I went tried to think of other alternatives.. of which there are mostly two: Google Authenticator and YubiKey .

The latter I knew by name already because I proxy-maintain the required software for Brant, and I know it’s outdated already and would require a new maintainer who can deal with those packages – I already posted about hardware-related maintenance for what it’s worth – so it was my first choice: while it meant I had to spend some money, it would have solved my problem and improved Gentoo, even if just for a tiny bit. The price for YubiKey devices is also low enough that, if I felt like providing more FTPS access to customers, I could simply bill it to them without many complaints.

So I went on the manufacturer’s (Yubico’s) website and tried to buy two of them (one for me to test and set up, and one to give my friend to access the server); despite publishing the prices in dollars, they sell through Sweden and UK, which means they are part of EU’s VAT area, and me being a registered business within EU, I should receive a reverse-charge invoice by stating my own VAT ID… never had much of a problem with it, as many of my suppliers are sparse through Europe, I registered for the “foreign-enabled” registry right when I opened business — don’t ask me why Italian (and Spanish as far as I can tell) business owners are not enabled by default to have intra-union suppliers.

Now trouble starts: since, as I just noted, not all VAT IDs are valid to use for intra-union trade, there has to be a way to ensure you’re dealing with an acceptable party. This is implemented through VIES the VAT Information Exchange System which, for what concerns Italian businesses, only tells you a boolean result of valid/invalid (and not the full registration data that most other states seem to provide). I knew VIES from a previous business agreement, but I never cared much. Turns out though that most e-Shops I encountered validate the VAT ID after order completed ­— or in the case of Amazon it seems like they check their internal database as well as VIES.

Yubico instead validates the request through VIES at the time of registration:

VAT Number could not be validated with VIES at this time. This typically happens when the service is under maintenance. Please retry after some time. For urgent orders, please contact order@yubico.com

Considering that the VIES website has a long disclaimer (which I can’t quote here for reasons that will be clear in a moment) stating that they do not guarantee the availability of the service at any time, and only seem to guarantee the validity of the data to the extent that the law ask them to (which probably means “as long as the states’ own databases are correct”), relying on such a service for registration is .. bad.

The VIES website is indeed down since at least 11am today (over four hours ago as I write this); for a moment they also gave me an interesting page (which I forgot to save), telling me that there were too many requests’ failures from “my IP address” … listing an IP address in the 212/8 range — my actual IP address is in the 94/8 range.

What’s the end result here? I’ll probably waste some more time trying to get Google Authenticator; Yubico basically lost a customer and a (possible) contributor by trying and failing to be smarter and won’t have a dedicated maintainer in Gentoo in the near future. It’s sad, because it seems to be easily the most cost- and time-effective solution out there (Google Authenticator is free, but it requires a greater investment of time, and time is money as we all should know).

Okay, I love to rant, so what?

Just the other day I have complained about Rails’s suggestion for world-writable logs and solved it by making it use syslog and now I’m in front of another situation that makes me think that people still don’t know how to stop themselves from creating software that is pretty much insecure by design.

So what’s up? For a customer of mine I ended up having to install a full LAMP stack, rather than my usual LAPR. In particular, this is for a website that will have to run Wordpress. Thankfully, I have ModSecurity to help me out, especially since not even two hours after actually setting up the instance, Spiderlabs announced two more security issues including an extract of their commercial rules.

Anyway, the Wordpress instance will have to be managed/administered by a friend of mine, who has already had some trouble before with a different hoster, where the whole Wordpress instance was injected with tons of malware, so was quite keen on letting me harden the security as much as I could… the problem here is that it seems like there’s not much that I can!

The first problem is that I don’t have a clean way to convert the admin section to forced SSL: not only wp-login.php is outside of the admin subdirectory, but most of Wordpress seem to use fully qualified, absolute URIs rather than relative URLs — such as the ones I’m used with Rails, which in the case both of Typo and Radiant let me restrict the admin/ directory to SSL quite easily. Why is that so important to me? Because I would have used an admin URL outside of the website’s domain for SSL: I don’t own a certificate for the website’s domain, which is not mine, nor I want to add it to the list of aliases of my own box. Oh well for now they’ll live with the “invalid certificate” warning.

Next stop is updating the webapp itself; I was sure at that point that “updating the webapp” meant letting the web server write to the wordpress deployment directory… yes, but that’s just part of it. As it happens, plugins are updated via FTP, like my friend told me.. but not in the sense of “downloaded from an FTP website and written to the filesystem” but the other way around: you have to tell Wordpress how to access its own deployment via FTP. In a clear-text web form. Admittedly, it supports FTPS, but it’s still not very funny.

I’m unsure if it was a good idea on my part to accept hosting Wordpress: we’re talking about installing MySQL, PHP, vsftpd and enabling one more service on the box (vsftp) just to get a blogging platform. Comparatively, Rails look like a lightweight approach.

Well… I finally figured out that the ucb package isn’t installed on Solaris 11 by default (resource). Unfortunately, the Oracle docs are confusing to follow. Here is a cheatsheet for installing the ucb package on your shiny Solaris 11 install.

1. Figure out the IPS installer, read man pages, get frustrated at lack of detail, run to Google.
2. Find the package you want on http://pkg.oracle.com/, in this case compatibility/ucb
3. Add the publisher link to your config, by the way, this link is not documented that I can find so I had to guess and check. A publisher is a package list of sorts, I guess.
   # pkg set-publisher -G '*' -M '*' -g http://pkg.oracle.com/solaris/release solaris
4. Install the package, # pkg install compatibility/ucb

# pkg install compatibility/ucb
Packages to install: 1
Create boot environment: No
Create backup boot environment: No

DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 80/80 0.4/0.4

PHASE ACTIONS
Install Phase 166/166

PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2

1. Behold, that you now have the compatibility libs for software that may need to use them

Whew…now, you might wonder what is so hard about that. Well, traversing Oracle docs is the hard part.

Here are the docs that I had open in my browser, they may or may not help and I fully expect the links to break in the future because Oracle is good at that.

• Copying and Creating Oracle Solaris 11 Package Repositories
• Oracle Solaris 11 Package Repository
• Oracle Solaris 11 Package Management with IPS
• Image Packaging System Man Pages

Seems the patch I committed for the fix was corrupted.  So, I am rebuilding and releasing kernels for 3.2 , 3.1 and 3.0.

Thanks for wired for pointing this out.  I will be removing the ones from yesterday.

The following kernels now contain the fix:

gentoo-sources-3.2.1-r2

gentoo-sources-3.1.10-r1

gentoo-sources-3.0.17-r2

 

In my previous installment I ranted about. among other things, the way Rails suggests you to keep a world-writeable log file for the production environment. As I said at the end, I planned on looking at the syslogger gem and that was actually quite helpful.

The idea goes like this: by using syslogger you can tell Rails that the logs have to go through the syslog; in my case that means it goes to metalog, which then filters on the webapp names and pushes it to /var/log/rails, taking care of rotating the log as needed (either due to size or time — the former is quite useful to avoid that rogue bots cause a DoS, which happened to me when I was inexperienced with these technologies!). Of course, this only works on Unix, but that’s what I care about anyway.

Beside the placement of the logs, using metalog for me also means I can filter important messages and show them in the important messages’ log rather than being just limited to a hidden log file within the app’s own tree, and also means that I can mix in the messages of all the running applications, rather than having each report to a different file. If I were to use syslog-ng instead, I could easily make it send the logs via network to another box and aggregate all of them there… but I really don’t see the point (yet) for that, and the features that metalog comes with tramp easily the network support.

So how do you achieve this? It’s actually pretty easy. Obviously it starts with installing dev-ruby/syslogger (in Gentoo, through Portage, everywhere else, via gem); then you can configure this very easily on both Rails 2.3 and 3.x series (I have one server running Rails 2.3, the other 3.1… I have yet to set up Typo 6.x, but I’ll probably do that at some point in the near future, although unlikely before FOSDEM).

The trick is all in config/environments/production.rb, where you have to tell Rails to use a custom Logger; there is already an example, commented-out like that refers to the other gem, SyslogLogger, but you should change it to something like this

config.logger = Syslogger.new("yourappname")

This way you can distinguish each application’s messages in the log. Then in the metalog.conf file you can have:

Rails apps : program_regex = "^(typo|radiant|yourappname)" logdir = "/var/log/rails" maxfiles = 5 break = 1

so that everything is then readable as /var/log/rails/current.

I’m not sure how much it impacts performance; I’d be surprised if it decreased them, as metalog also buffers the disk writes, but you never know until you check for sure; in general I still prefer if the (multiple) Rails processes send everything to metalog for my own convenience.

Interestingly, if you have a webapp that does not deal with on-disk files directly, but just with a database, by using syslogger you’re basically limiting the writing to the cache directories only, which is probably a positive note.

Okay, so I decided to start yet. another. new. blog. It’s called “working with teenagers”. I’m reproducing … at least, in some fashion. I wonder if my parents are proud of me. Late at night, they can stay up and say, “this is about as close to grandkids as we’ll get! Pass me some Wheat Thins.” Seems reasonable.

Really, though, since I’m going to school to, you know, do this full-time, I thought it’d be cool to archive my old posts about working with them, and just post stuff to it whenever I feel like it. Like tonight, I just added another one, and I figured, “I should probably go to bed. And also write a blog post and my other blog!” And then my mind went blank after that.

In addition to the archives of stuff on here that you’ve already taken the time to memorize, I’ve added two new posts over there since then. You’ll notice that I’ve refrained from shamelessly using my blog to do some cross-posting mojo to do some self-promotion … at least until tonight. To make it seem like this blog post has actual content, I’ll throw in something slightly more interesting.

I found out recently that I really enjoy bowling. Me and my cousin have been going for a few weekends in a row. We’ve mastered the art of playing 4 games in a row for $10. That’s not bad, considering it’s late Saturday nights. Good times. I’m actually getting better at (since it’s impossible to be worse). The hardest part is getting people to ignore that I’m using an 8 pound ball because I’ll throw out my wrists if I use anything heavier.

Where was I going with all of this. I remember I was playing Skyrim tonight (level 60, yo!), and I was fighting a dragon and trying to eat cheesy nacho goodness at the same time. I kept having to pause my game so I could eat, and I thought to myself, “I can’t pause a nacho.” Words to live by.

In other entertainment, I present to you, the best picture on the internet:

It’s totally legit. They have their own domain and everything: http://thebestpictureontheinternet.com/

I think it’s time to go to bed.

I just released gentoo-sources-3.2.1-r1 for Linux Local Privilege Escalation via SUID /proc/pid/mem .

I plan on creating releases for additional kernels with this patch through the day.

See the link for more info on the privilege escalation.

The following kernel versions contain the patch:

gentoo-sources-3.2.1-r1

gentoo-sources-3.1.10

gentoo-sources-3.0.17-r1

 

At Sportradar, we have several products where everything is hosted on our servers, but our customers embed the them into their websites. The result is that we concurrently handle the accumulated traffic of all our customers. On a typical Saturday this is a five-digit number of requests per second. In order to handle all this traffic, and more importantly, making it easy to scale up to meet future traffic demands as we sell more products, we have spent quite a bit of time on researching what kind of service infrastructure works best with as little hardware as possible.

The stack I will describe here is not the same as we are using. It is a simplification. Linux provides a lot of buttons to push and knobs to turn that affect performance. But these settings are typically very tied to the workload and very difficult to generalise. We have achieved an understanding, mostly by trial and error, about what works for us, but the same settings will probably not be useful to anyone else.

This article is only concerned about how requests move from your users to the web servers serving content. It does not deal with how to scale the web application itself. I will also not go into much detail about how to configure each of the services mentioned.

Principles

I have to say I am a big fan of the Unix philosophy of using small, specialised services. It is the primary reason I like to use web servers like nginx, which only handles one single task, and why I think using PHP FPM instead of Apache/Mod_PHP is a good idea. Just like with programming, keeping stuff compartmentalised makes debugging easier, it leads to single failing nodes affecting only single services, and it is a whole lot easier to scale where necessary.

All of the machine in this setup are virtualised using Kernel Virtual Machine (KVM), and managed by Ganeti. The cool thing about using Ganeti is that it supports syncing disks to a secondary hypervisor using Distributed Replicated Block Device (DRBD). If any of these nodes fail, they can just be booted on the secondary hypervisor and pick up where the failing node left off. Note that if your application is very CPU bound, I would not use virtualisation. You lose quite a bit of CPU and I/O performance when virtualising.

The stack

Let me start of by presenting the stack. Then I’ll go through each level and give some more thorough explanations later.

1. Gateway
2. SSL termination/proxy
3. HTTP Accelerator
4. Web server/FCGI

Granted, using this many systems require its cost of system administration. But since the nodes individually are so simple, running software upgrades is rather trivial as there are no conflicting dependencies. Using virtual machines also make dist-upgrades trivial. We simply do not ever do it. Instead we fire up a new virtual machine with the newest OS version, configure it, deploy software and do some simple testing, and then just let it be a drop-in replacement of the old node.

The gateway

The purpose of the gateway is to handle routing between Internet and the application-specific subnets. I like using a load balancer like Linux Virtual Server (LVS) for this, because it allows me to scale the layer following horizontally. LVS can basically handle any amount of traffic you throw at it on a single node so there is no need to think about how to add more nodes into this layer. If it really became necessary to do so, and adding more hardware to the existing two nodes would not be possible, DNS round-robin could be a way to achieve a form of load balancing.

Even though I do not find load balancing necessary in this layer, I would still remand redundancy. Not only can nodes fail, but every now and then, I would like to be able to take the load gateway out of production to perform maintenance on it. Redundancy on this level is achieved by using Linux-HA. The simple explanation of what this software suite does is this: If the active node dies, the stand-by node takes over its IP, sends an ARP announcement, and, if configured correctly, resumes the work of the failed node.

SSL termination/proxy

So you may ask “Why do we need dedicated nodes to terminate SSL?”. Firstly it is because both web applications and SSL terminations are typically CPU-bound so you do not want these two parts fighting over resources. Secondly, Varnish, the next service in the stack, does not speak SSL.

This layer need to be scalable horizontally due to the CPU cycles required to terminate SSL. Especially if you allow ciphers using one-time Diffie-Hellman. I always make sure that I have enough nodes on this layer to handle at least a single node failure.

These days I use nginx for this layer, but any kind of light-weight, high-performance web server will do the job. The one thing worth mentioning about using nginx is that it does not (yet) support HTTP 1.1. So no keep-alive connections and no chunked response towards the backend. But since the backend is Varnish, this is not that big of an issue.

HTTP Accellerator

And now for the stack’s super hero: Varnish. It is an HTTP cache server that can handle pretty much any amount of traffic. During my stress testing I have seen Varnish handle thousands of connections on a single CPU core. Therefor I would not worry about scaling this bit horizontally unless you have to cache a huge amount of data.

Another reason for only having a single active node in this layer is that there is a chance for the same page being cached at different times with different contents. If the user continuously hit ‘refresh’ they would end flipping between the two different cached versions making your site look silly.

The redundancy setup is identical to that of the gateway layer.

Web server

In my sketch above, I just added a bunch of Nginx/PHP FPM servers behind the Varnish. This is how the setup would look like in its simplest form, assuming that you do not require cookies, user logins or anything else that require this layer to simulate some form of state.

The important bit is that this layer is easy to scale horizontally. All you need to do is add another server to the director configuration of Varnish. Varnish support several different form of directors, even directors that will help you maintain state. Going into details about this, however, is in itself worthy of an article.

Some final remarks

This setup is a bit simpler than I would put into production, but it contains the essential details. All of the services mentioned are quite trivial to configure and there should be lots of resources online about each of them.