YO PYGMENTS 1.0 I’M REALLY HAPPY FOR YOU, AND IMMA LET YOU FINISH, BUT PYGMENTS 1.1 HAS ONE OF THE BEST RELEASES OF ALL TIME
The past couple of weeks, if not months, we’ve been seeing some processes clog up on Bitbucket. Long story short, eventually, some apache2 worker processes would start using a lot of CPU as well as memory, and sit there for a long time. Hours, in fact. This effectively caused those workers to stall, not letting new ones spawn, and over time, that would make the site unreachable and/or slow.
Today, I added some “forensic logging”, which let us trace what URLs those processes were choking on. After we found those, and could reproduce the crazy resource usage, we traced it back to a Pygments highlight. Specifically for the Scala lexer. Apparently that lexer didn’t work all too well.
Upgrading to 1.1 fixed the issue. Things are running much smoother now.
Moral of the story? Keep your software upgraded. It’s not only new features that are being released, also bugfixes.
Thanks for always being so open with what was going wrong, and for fixing things so quickly.
Ali
1 Oct 09 at 12:23 pm
What Ali said +1.
Paul
1 Oct 09 at 1:20 pm
I’d like to hear more about the “forensic logging” you did to determine this. :)
Rob Hudson
1 Oct 09 at 5:51 pm
@rob
It was just a basic middleware that logged the pid (os.getpid) as well as parent pid of the process, so when it showed up in ‘top’, we could track it down to an individual URL. It quickly became evident that requests to anything ending in .scala made the request choke :-)
jespern
1 Oct 09 at 6:11 pm
We’ve had *exactly* the same problem in code.openbravo.com, it was driving me crazy. It turns out I was also using pygments 1.0, so thank you for sharing this!
jpabloae
3 Oct 09 at 10:39 am