For some reason, I seem to end up writing software for very esoteric use-cases. Today, though, I think I’ve outdone myself: I sat down and wrote a Ruby library to get and set process resource limits – those things that nobody ever thinks about except when they run out of file descriptors.
I didn’t even have a direct need for it. Recently I was grovelling through the EventMachine codebase, looking at the filehandle limit code, and noticed that the pure-ruby implementation didn’t manipulate filehandle limits. I considered adding it, then realised that there wasn’t a library available to do it. Since I haven’t berked around with FFI for a while, I decided to write rlimit. Now to find the time to write that patch for EventMachine…
Since I doubt there are many people who have a burning need to manipulate rlimits in Ruby, this gem will no doubt sit quiet and undisturbed in the dark, dusty corners of rubygems.org. However, for the three people on earth who find this useful: you’re welcome.
If you’ve noticed your chrome/chromium on Linux having problems since you upgraded to somewhere around version 35/36, you’re not alone. Thankfully, it’s relatively easy to workaround. It will hit people who keep their browser open for a long time, or who have lots of tabs (or if you’re like me, and do both).
To tell if you’re suffering from this particular problem, crack open your
~/.xsession-errors file (or wherever your system logs stdout/stderr from
programs running under X), and look for lines that look like this:
[22161:22185:0830/124533:ERROR:shared_memory_posix.cc(231)] Creating shared memory in /dev/shm/.org.chromium.Chromium.gFTQSy failed: Too many open files
[22161:22185:0830/124601:ERROR:host_shared_bitmap_manager.cc(122)] Cannot create shared memory buffer
If you see those errors, congratulations! The rest of this blog post will be of use to you.
There’s probably a myriad of bugs open about this problem, but the one I found was #367037: Shared memory-related tab crash. It turns out there’s a file handle leak in the chromium codebase somewhere, relating to shared memory handling. There’s no fix available, but the workaround is quite simple: increase the number of files that processes are allowed to have open.
System-wide, you can do this by creating a file
/etc/security/limits.d/local-nofile.conf, containing this line:
* - nofile 65535
You could also edit
/etc/security/limits.conf to contain the same line, if
you were so inclined. Note that this will only take effect next time you
login, or perhaps even only when you restart X (or, at worst, your entire
This doesn’t help you if you’ve got Chromium already open and you’d like to
stop it from crashing Right Now (perhaps restarting your machine would be a
terrible hardship, causing you to lose your hard-won uptime record), then
you can use a magical tool called
prlimit syscall is available if you’re running a Linux 2.6.36 or later
kernel, and running at least glibc 2.13. You’ll have a
line program if you’ve got util-linux 2.21 or later. If not, you can use
the example source code in the
prlimit(2) manpage, changing
RLIMIT_NOFILE, and then running it like this:
prlimit <PID> 65535 65535
<PID> argument is taken from the first number in the log messages from
.xsession-errors – in the example above, it’s
And now, you can go back to using your tabs as ersatz bookmarks, like I do.
$ sudo apt-get install -y leiningen [...] $ lein new scratch [...] $ cd scratch $ lein repl Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.pom from repository central at http://repo1.maven.org/maven2 Transferring 5K from central Downloading: org/sonatype/oss/oss-parent/5/oss-parent-5.pom from repository central at http://repo1.maven.org/maven2 Transferring 4K from central Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.jar from repository central at http://repo1.maven.org/maven2 Transferring 3311K from central [...]
lein downloads some random JARs from a website over
HTTP1, with, as far as far I can tell, no verification that
what I’m asking for is what I’m getting (has nobody ever heard of
Man-in-the-Middle attacks in Maven land?). It downloads a
.sha1 file to
(presumably) do integrity checking, but that’s no safety net – if I can
serve you a dodgy
.jar, I can serve you an equally-dodgy
.sha1 file, too
(also, SHA256 is where all the cool kids are at these days). Finally,
jarsigner tells me that there’s no signature on the
.jar itself, either.
It gets better, though. The
repo1.maven.org site is served by the
fastly.net2 pseudo-CDN3, which adds another
set of points in the chain which can be subverted to hijack and spoof
traffic. More routers, more DNS zones, and more servers.
I’ve seen Debian take a kicking more than once because packages aren’t individually signed, or because packages aren’t served over HTTPS. But at least Debian’s packages can be verified by chaining to a signature made by a well-known, widely-distributed key, signed by two Debian Developers with very well-connected keys.
This repository, on the other hand… oy gevalt. There are OpenPGP (GPG)
signatures available for each package (tack
.asc onto the end of the
.jar URL), but no attempt was made to download the signatures for the
.jar I downloaded. Even if the signature was downloaded and checked,
there’s no way for me (or anyone) to trust the signature – the signature
was made by a key that’s signed by one other key, which itself has no
signatures. If I were an attacker, it wouldn’t be hard for me to replace
that key chain with one of my own devising.
Even ignoring everyone living behind a government- or company-run intercepting proxy, and everyone using public wifi, it’s pretty well common knowledge by now (thanks to Edward Snowden) that playing silly-buggers with Internet traffic isn’t hard to do, and there’s no shortage of evidence that it is, in fact, done on a routine basis by all manner of people. Serving up executable code to a large number of people, in that threat environment, with no way for them to have any reasonable assurance that code is trustworthy, is very disappointing.
Please, for the good of the Internet, improve your act, Maven. Putting
HTTPS on your distribution would be a bare minimum. There are attacks on
SSL, sure, but they’re a lot harder to pull off than sitting on public wifi
hijacking TCP connections. Far better would be to start mandating
signatures, requiring signature checks to pass, and having all signatures
chain to a well-known, widely-trusted, and properly secured trust root.
Signing all keys that are allowed to upload to
maven.org with a “maven.org
distribution root” key (itself kept in hardware and only used offline), and
then verifying that all signatures chain to that key, wouldn’t be insanely
difficult, and would greatly improve the security of the software supply
chain. Sure, it wouldn’t be perfect, but don’t make the perfect the enemy
of the good. Cost-effective improvements are possible here.
Yes, security is hard. But you don’t get to ignore it just because of that, when you’re creating an attractive nuisance for anyone who wants to own up a whole passel of machines by slipping some dodgy code into a widely-used package.
To add insult to injury, it appears to ignore my
http_proxyenvironment variable, and the
repo1.maven.orgserver returns plain-text error responses with
Content-Type: text/xml. But at this point, that’s just icing on the shit cake. ↩
At one point in the past, my then-employer (a hosting provider) blocked Fastly’s caching servers from their network because they took down a customer site with a massive number of requests to a single resource, and the incoming request traffic was indistinguishable from a botnet-sourced DDoS attack. The requests were coming from IP space registered to a number of different ISPs, with no distinguishing rDNS (
184-106-82-243.static.cloud-ips.comdoesn’t help me to distinguish between “I’m a professionally-run distributed proxy” and “I’m a pwned box here to hammer your site into the ground”). ↩
Pretty much all of the new breed of so-called CDNs aren’t actually pro-actively distributing content, they’re just proxies. That isn’t a bad thing, per se, but I rather dislike the far-too-common practice of installing varnish (and perhaps mod_pagespeed, if they’re providing “advanced” capabilities) on a couple of AWS instances, and hanging out your shingle as a CDN. I prefer a bit of truth in my advertising. ↩
Gitolite is a popular way to manage collections of git repositories entirely from the command line – it’s configured using configuration stored in a git repo, which is nicely self-referential. Providing per-branch access control and a wide range of addons, it’s quite a valuable system.
In recent versions (3.6), it added support for configuring per-repository
git hooks from within the
gitolite-admin repo itself – something which previously required directly
jiggering around with the repo metadata on the filesystem. It allows you to
“chain” multiple hooks together, too, which is a nice touch. You can, for
example, define hooks for “validate style guidelines”, “submit patch to code
review” and “push to the CI server”. Then for each repo you can pick which
of those hooks to execute. It’s neat.
There’s one glaring problem, though – you can only use these chained,
per-repo hooks on the
update hook is special, and gitolite wants to make sure you
never, ever forget it. You can hook into the update processing chain by
using something called a “virtual
ref”; they’re stored in a separate
configuration directory, use a different syntax in the config file, and if
you’re trying to learn what they do, you’ll spend a fair bit of time on
them. The documentation describes VREFs as “a mechanism to add additional
constraints to a push”. The association between that and the update hook
is one you get to make for yourself.
The interesting thing is that there’s no need for this gratuitous difference
in configuration methods between the different hooks. I wrote a very small
that makes the
update hook configurable in exactly the same way as the other
server-side hooks, with no loss of existing functionality.
The reason I’m posting it here is that I tried to submit it to the primary gitolite developer, and was told “I’m not touching the update hook […] I’m not discussing this […] take it or leave it”. So instead, I’m publicising this patch for anyone who wants to locally patch their gitolite installation to have a consistent per-repo hook UI. Share and enjoy!
I just made an account on yet another web service. On the suggestion of my
password manager, I attempted to use the password “
W:9[$X*F”. It was
rejected because “Password must contain at least one non-alphabet character,
one lowercase letter, one uppercase letter”. OK, how about “
Yep, that’s fine.
Anyone want to guess which of those two passwords is going to fall victim to a brute-force attack first? Go on, don’t be shy, take a wild shot in the dark!
After dealing with the client side of the DNSSEC puzzle last week, I thought it behooved me to also go about getting DNSSEC going on the domains I run DNS for. Like the resolver configuration, the server side work is straightforward enough once you know how, but boy howdy are there some landmines to be aware of.
One thing that made my job a little less ordinary is that I use and love tinydns. It’s an amazingly small and simple authoritative DNS server, strong in the Unix tradition of “do one thing and do it well”. Unfortunately, DNSSEC is anything but “small and simple” and so tinydns doesn’t support DNSSEC out of the box. However, Peter Conrad has produced a patch for tinydns to do DNSSEC, and that does the trick very nicely.
A brief aside about tinydns and DNSSEC, if I may… Poor key security is probably the single biggest compromise vector for crypto. So you want to keep your keys secure. A great way to keep keys secure is to not put them on machines that run public-facing network services (like DNS servers). So, you want to keep your keys away from your public DNS servers. A really great way of doing that would be to have all of your DNS records somewhere out of the way, and when they change regenerate the zone file, re-sign it, and push it out to all your DNS servers. That happens to be exactly how tinydns works. I happen to think that tinydns fits very nicely into a DNSSEC-enabled world. Anyway, back to the story.
Once I’d patched the tinydns source and built updated packages, it was time to start DNSSEC-enabling zones. This breaks down into a few simple steps:
Generate a key for each zone. This will produce a private key (which, as the name suggests, you should keep to yourself), a public key in a
DNSKEYDNS record, and a
DSDNS record. More on those in a minute.
One thing to be wary of, if you’re like me and don’t want or need separate “Key Signing” and “Zone Signing” keys. You must generate a “Key Signing” key – this is a key with a “flags” value of 257. Doing this wrong will result in all sorts of odd-ball problems. I wanted to just sign zones, so I generated a “Zone Signing” key, which has a “flags” value of 256. Big mistake.
DSrecord is a hash of everything in the
DNSKEYrecord, so don’t just think you can change the
257and everything will still work. It won’t.
Add the key records to the zone data. For tinydns, this is just a matter of copying the zone records from the generated key into the zone file itself, and adding an extra pseudo record (it’s all covered in the tinydnssec howto).
Publish the zone data. Reload your BIND config, run
tinydns-datathen rsync, or do whatever it is PowerDNS people do (kick the database until replication starts working again?).
Test everything. I found the Verisign Labs DNSSEC Debugger to be very helpful. You want ticks everywhere except for where it’s looking for DS records for your zone in the higher-level zone. If there are any other freak-outs, you’ll want to fix those – because broken DNSSEC will take your domain off the Internet in no time.
Tell the world about your DNSSEC keys. This is simply a matter of giving your
DSrecord to your domain registrar, for them to add it to the zone data for your domain’s parent. Wherever you’d normally go to edit the nameservers or contact details for your domain, you probably want to do to the same place and look for something about “DS” or “Domain Signer” records. Copy and paste the details from the
DSrecord in your zone into there, submit, and wait a minute or two for the records to get published.
Test again. Before you pat yourself on the back, make sure you’ve got a full board of green ticks in the DNSSEC Debugger. if anything’s wrong, you want to rollback immediately, because broken DNSSEC means that anyone using a DNSSEC-enabled resolver just lost the ability to see your domain.
That’s it! There’s a lot of complicated crypto going on behind the scenes, and DNSSEC seems to revel in the number of acronyms and concepts that it introduces, but the actual execution of DNSSEC-enabling your domains is quite straightforward.
DNSSEC is one of those things that have been on the “I should poke at that sometime” list, but today I decided to actually get in and play around with it. Turns out it’s pretty trivially easy to setup, but I got bitten by a few “gotchas” during testing, that I thought worthwhile to document. Since I like to explain things, a lot of this article isn’t “how to” so much as “how does”.
Configuring The Resolver
First off, you need a resolver1 that you can trust, and which
implements DNSSEC validation. In theory, you could use your ISP’s
resolvers (if they’re DNSSEC-enabled), or even Google’s public resolvers
188.8.131.52), but it’s really not recommended – how much do
you trust those, really? More importantly, how much do you trust the
network path between them and you? Frankly, running a local on-host caching
resolver is trivial, consumes next-to-no resources, and is highly
recommended. It’s what I did.
unbound, finally casting free the venerable
dnscache, which has
served me well for many years. I chose unbound because it fits the “do one
small thing, and do it well” philosophy that I cherish (unlike BIND), and
because it’s what all the cool kids appear to be using these days.
Installing it was as simple as
apt-get install unbound – it’s available
in pretty much every distro, I believe.
Out of the box, unbound in Debian wheezy comes configured for DNSSEC
validation. Other methods of installation might not be so amenable, so just
make sure your
unbound.conf liiks like this, so that DNSSEC is enabled:
server: auto-trust-anchor-file: "/var/lib/unbound/root.key" val-log-level: 2
val-log-level line isn’t strictly required, but I found it useful to
have unbound log info on validation failures to syslog. Season to taste.
You also have to configure your system to actually use your new, shiny
local resolver. Editing
/etc/resolv.conf would, in an ideal world, work
Just Fine, but between Magical Unhelpful Pixies (I’m looking at you,
NetworkManager) and DHCP, it often isn’t that easy. Ultimately, though, you
should configure your system so that the only resolver in
/etc/resolv.conf is always
127.0.0.1. Anything else is a security hole
waiting to happen. If you’re using DHCP and a recent version of ISC
supersede domain-name-servers 127.0.0.1; to
dhclient.conf should do the needful.
Weighing the (Trust) Anchor
Whatever local resolver you choose, you need to tell it about “trust anchors”. This is a key (or keys) which are trusted to sign all the other data that the resolver will find. Like the rest of the DNS, DNSSEC is hierarchical, and you need to know who to “start with” before you can start.
To resolve a name using DNS, you ask the “root servers” for the answer to a
question (say, “what is the address of
www.example.com?”). The root
server doesn’t know the answer, but it knows who to ask, and it tells you
where to go (typically, “go ask the DNS servers for
.com, which are X, Y,
and Z”). Then you ask those servers, and they probably don’t know, but
they know who to ask, and they tell you to go ask those servers. This
continues until you finally get a server which says, “oh, yes, I know the
answer to that!” and hands over the information you’re after.
In DNSSEC, you get digital signatures that prove that the answers you get
from each DNS server you ask are legit, and not forged. So when you ask the
root servers, “what is the address of
www.example.com?”, they answer, “I
don’t know, but go ask DNS servers at addresses X, Y, and Z – and by the
way, you can trust this answer is legit because here’s this signature you
can check”. To verify a signature is legit, you need to know that the key
used to make the signature is trustworthy, and it wasn’t just any ol’ key
that made it – similar to how an ink scribble on a page doesn’t mean
anything unless you know what the signature you’re looking at should look
DNSSEC has two ways to make sure you know who to trust. Firstly, those “trust
anchors” I mentioned three paragraphs ago. These are the keys that the root
servers use to sign the responses they send out. You tell unbound (or your
local DNSSEC-enabled resolver of choice), “if you see a signature provided
by this trust anchor key, you can trust it to be legit”. The second way to
trust is key is to have someone you already trust say, “oh, you can trust
this key, too”. This is done by providing more data in the answer that
you get from the root servers. So the full answer you get from the root
servers when you ask, “what is the address of
www.example.com?” looks like
I don’t know, but you should ask the DNS servers at addresses X, Y, and Z. You can trust their answers because they’ll sign their responses with key K. Finally, you can trust that I’m legit because of this signature I’m providing you, generated from key T.
Assuming that T is the trust-anchor key you’ve already told unbound to
trust, it can then trust all of the information it got from that first
question. When it then goes to DNS server X and asks, “Hey, do you know
the address of
www.example.com?”, that server can send back its own
Gee, shucks, I don’t know, but DNS servers at P and Q can help you. You can trust them because they’ll sign responses with key S. And just to show you I’m on the level, check out this nifty signature I made, signed with key K.
Since a server we trust told us to trust key K, and then we got a response signed by key K that told us to trust key S, when we then go and ask server P and get a response signed by S, we’re still demonstrably secure. Neat setup, right?
Anyway, now you know all about trust anchors. How do you set it up? For unbound, it’s capable of “priming” the trust anchor itself. It does this by retrieving a copy of the trust anchor from [a well-known HTTPS URL]((https://data.iana.org/root-anchors/root-anchors.xml), which provides a reasonable measure of security – the trust anchor file should be safe as long as that HTTPS server is trustworthy. For the properly paranoid, you should determine an alternate means of verifying that you have the correct root key (a web search for “DNSSEC root key attestation” would probably be a good start).
It’s always handy to know that something’s working as you expect. For something that’s important for security, that goes double. So, how do you test that your DNSSEC-capable resolver is, indeed, working as designed?
Well, firstly, you can test that the resolver, by itself, is working. This is simple: do a query for a zone known to have broken DNSSEC. Thankfully, some nice people (Comcast, specifically) have already set this up. You can test a resolver for DNSSEC-correctness with this command:
dig @127.0.0.1 www.dnssec-failed.org
dig gives you back an IP address, something is broken. The
correct, “my DNSSEC works!” response should be a
That’s all well and good, but having a resolver that works isn’t much good if nothing’s actually using it. Your system configuration has probably got some other resolvers configured into it (such as those provided by a DHCP server), and you’ll want to make sure those aren’t getting used2. The trivial test is to visit http://www.dnssec-failed.org in your browser. If that comes back with a page, then something’s still wrong.
A more user-friendly test of your DNSSEC configuration is provided by http://dnssectest.sidnlabs.nl/test.php, which will display a page either way, and give you a tick or a cross depending on whether you’re DNSSEC-enabled or not.
One thing to be wary of when you’re doing browser tests is caching. I spent far too long trying to work out why “unbound wasn’t working”, when in actual fact it was working fine, it’s just that both my browser and local caching proxy (squid) were caching DNS requests3. So, even after I’d correctly setup unbound, visiting the test sites was providing me with erroneous results.
Fixing up the browser was simple enough. In Chromium, visiting
chrome://net-internals#dns will show you what’s being cached, what
resolv.conf settings are being used, and allow you to clear the browser’s
Squid was similarly simple, although it took me a while to figure out that it was causing me grief (software that “just works”, being so rare, tends to slip your mind, and so I often forget that squid is actually doing me good). Restarting squid has the appropriate result (clears the DNS cache), but in the end I decided that having another layer of caching wasn’t going to help significantly when I’ve got an on-host DNS cache already, in unbound. So, I just turned the knobs for DNS caching in squid right down:
negative_dns_ttl 1 second positive_dns_ttl 1 second
And voila! All was well.
DNSSEC is cool. It’s easy to get going with, and over time it will provide more and more practical benefits (I am a DANE fanboi). Get started with it now, and you’ll be ahead of the curve.
A “resolver” is a program which takes a name, does all the DNS queries required to get the data associated with that name, and return it to whoever wanted to know. This can be a library which links to a client application, or it can be a separate program which receives DNS requests itself.
There are usually both types of resolvers in use for a typical Linux system to resolve names. On every machine, in every program, there is the so-called “stub resolver”, which is part of
libc, and does nothing except create DNS requests and send them to the addresses listed in
resolv.conf. Also present, although not always running on every machine (typically your ISP or network admin has some running for everyone to share) are one or more “recursive resolvers”, which are separate programs which take DNS requests, do all of the queries themselves, and return the result to the stub resolver for presentation to the client program which made the request in the first place.
You will see this material again. ↩
You can configure dhclient to use your local resolver instead of the DHCP-provided ones by simply setting:
supersede domain-name-servers 127.0.0.1;
It also didn’t help that I hadn’t correctly knobbled
dhclientthe first time around, and so even when I thought I was only resolving against my local unbound server, it was still actually using the DHCP-provided resolvers… ↩
Since first posting my idea for an SSL co-op a couple of weeks ago, I’ve gotten some positive feedback from people, and further thinking and research has convinced me that it is feasible to at least attempt it.
As a result, I’d like to announce the public unveiling of The SSL Co-op. It is intended to be a commercial, not-for-profit1 organisation that issues widely-trusted certificates to members, for their use or for resale. Eventually, I’d like the co-op to be a root CA in its own right, with its certificate trusted by all the browsers and other X.509-using applications out there, but that isn’t something that’s achievable immediately.
At this stage, the co-op hasn’t been formed, and I’m looking for expressions of interest from individuals and organisations who would be interested in becoming members. If you fit that description, I’d really appreciate it if you could fill out a short survey so I can get a better idea of what sort of scale the co-op will be operating at initially.
This is the first step towards an interesting future, where there is more choice of provider for online identity verification. Exciting times.
Despite a lot of misunderstanding to the contrary, “commercial, not-for-profit” is not a contradiction. “Commercial” means “doing things for money”, and “not-for-profit” means “not returning a dividend to investors”. In the case of the SSL co-op, it will be providing services to members on a cost-recovery basis, and any excess funds left over from that will be re-invested in the co-op to improve the services provided to members. ↩
Ten days ago, I blogged about (finally) generating a GPG key transition statement. As the title of this post suggests, I have received zero signatures. Have other people had any success with transition statements? Perhaps it’s time to hit up Debian developers I know one-by-one…
To the IRCz!
I’m probably the last person on the planet to get around to doing this, but I figure it’s probably about time my venerable old 1024D key was put out to pasture.
My transition statement (signed, as usual, by both the old and new keys) is available. If you signed my old key, and still trust that I am who I think I am, I’d appreciate it if you could sign my new key and fling the signatures at me or the keyserver networks.