Slashdotted; Post-mortem

About a day and a half ago, Reddit user stderr posted a link to the PHP documentation showing that GOTO will be a part of the language as of version 5.3. Somewhere in the reddit comments for this post, a link was pasted to an entry on this blog where I announced the feature being added years earlier. This naturally brought out the usual flame wars that circle around something like GOTO and drove some new traffic to my blog. No big deal, my server is pretty low-traffic, it can handle a few extra hits.

Within a few hours, burghler had browsed through other entries on my blog, finding what was at the time, the most recent entry about my friend's experience burying her mother. Just like the first reddit post, which had made the front page, this one also had a somewhat incendiary title.

Okay, more than a little incendiary, but I'll get to that in a moment...

The lesson

You would think, given that I'm the Architect for Yahoo WebSearch Front-end Engineering, that I would know something about configuring a server to not fall over under load. And in fact, I spend a good portion of my time on making sure that unexpected traffic spikes aren't enough to make a server get overloaded and trigger a chain-reaction of front-ends falling over. I really have no excuse for not preparing my server for what happened.

When I woke up Wed morning, I found that my server just didn't seem to respond to SSH or HTTP attempts. A reboot request didn't help, and the colo folks insisted that the server was simply running slow, but it was running. So I left my ssh connection attempt running and eventually I did get a login prompt. Several pained minutes later I managed to get an iptables rule in place to block off the flood of traffic (quicker than trying to stop the webserver). Suddenly, the CPU load was gone! Turns out I had my MaxClients setting much too high, and after a certain degree of concurrency, enough web-server children had spawned off to use up the available memory, which triggered disk swap, and made the CPU load 10x worse. Again, I know this effect exists, I really should have set up this server better.

A few tweaks to the config later and I got my server running smoothly. I also took the time to re-run my access log statistics. Of the aproximately 3 years of stats I've got, a full 2% of my hits were logged yesterday. Impressive reddit.... you win this round...

Bright squares indicate heavy traffic, dark squares indicate low traffic

Prior to yesterday's traffic

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Current stats

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

My flickr views...

Clarifications

Despite the title of the reddit entry, I have nothing against Christians. Further, I count myself as one. So if you really want to take a piss at people who are calling Christians evil, don't aim at me. Thanks.

Moreover, I don't think those people were evil, or even horrible (though I did use that word in the heat of the moment). A loved one had just died, and everyone was in pain. Death SUCKS, and it was a bad situation no matter how you slice it. I don't hate those people for trying to erase my friend though, I pity them. I pity the fact that they turned down the chance to mourn the passing of their loved one by crying on another loved one's shoulder. I pity the fact that they didn't get nearly the closure that my friend did. I pity them for willingly becoming victims of their own grief.

The rest is between them and God.

I have officially arrived

I was at the Search Monkey Developer Launch tonight to answer questions and give demos to the attendees. The event itself was a nice success, but that's not what I'm writing about. What I'm writing about is one particular attendee. Apologies to this guy if I embarrass him, but he had to know I'd say something.

After getting through the initial queue of developers asking about Search Monkey, I was approach by a tall young man:

<guy> Hey, you're Sara Golemon, right?
<me>  Yeah...
<guy> Nice to meet you, I really like your blog entries about how PHP works,
     and I bought your book!
<me>  Oh fantastic! Are you enjoying it?
<guy> Yeah, it's really heavy, and I haven't had a chance to use a lot,
     but I was experimenting with writing my own libssh2 wrapper...

By this point, I'm suitably blushing. I'm easily swayed by compliments thanks to my lack of self-esteem. Then he takes a turn...

<guy> I almost emailed this to you, it's a photo my girlfriend took of me without warning...
     Let me just find it in my camera
* guy shows photo of himself lying in a water-filled bathtub, naked, covering his... bits
     with a copy of my book

I... um... well... okay... I can see why you wouldn't email that out of the blue.... Now, I try to think of myself as a fairly progressive, hard-to-phase sort. I can tell a dirty joke with the best of 'em, but this just... left me giggly for the next hour. Like, um...wow... okay... Thanks for sharing...

Is this a sign? Is it a measure of notoriety when strangers show you naked photos of themselves at random tech gatherings?

Hey Facebook! What gives?

One of the ways in which I prepare for an interview is to research the person I'm going to be interviewing. The first step is to toss their name into one (or more) search engines and see what falls out. Not getting any useful results isn't a negative (many people work on internal stuff with no concern for fame fortune and glory, and they're very good at it), but if I happen to learn a little about them from a blog entry or a profile on a social networking site, then I can direct my questions in a way that more precisely fits the candidate.

So today, I sat down to read through the resumé of an upcoming candidate and punched his name into the best damned search engine ever. What's relatively unique about this candidate is that his name contains an accented character. The search results handled this accented character cleanly of course, and when I followed the first result for "José Smith" (Not his real name -- I'm respecting privacy here), I found myself looking at his Facebook profile. All well and great of course, but then I noticed the URL in the address bar: http://www.facebook.com/Jos_Smith/123456789.

Jos??? The only Jos I know is a director with a penchant for vampires.

Now, I get not putting the UTF8 or latin1 literal in the URL, browsers don't agree nearly enough on URL encoding to be able to expect it to work reliably. And putting the urlencoded version would just be silly since /Jos%C3%A9_Smith/ looks even uglier. I get that. But what if his name were entirely non-ascii? Perhaps a chinesse name? Would it be so hard to do a transliteration? I seem to recall Derick sharing a few ideas on the subject a couple years ago...

When good encodings go bad

In the past year, I've been doing some work with Unicode as part of the PHP6 upgrade. I've learned more than I wanted to know about all sorts of encodings from UTF-7 to koi8-r to good old iso-8859-1. I've picked apart the picayune differences between UCS-2 and UTF-16, and played the game of surrogate pairing and orphaning. Despite all that exposure however, I wasn't prepared when a question crossed my inbox about a lesser known encoding called AL32UTF8.

I'd never heard of this one before, so I went to my favorite search engine for some answers. Turns out it's something Oracle came up with and later got adopted as a proper standard with the name CESU-8. At first glance, CESU-8 looks identical to UTF-8 in the same way that UCS-2 looks a lot like UTF-16. In fact every codepoint from U+0000 to U+FFFF is encoded identically under both sets of rules: 16 bits, split up over one, two, or three bytes, with leftover bits framing the encoding protocol.

When you jump up above U+FFFF however, into the realm of CJK codepoints and the like (such as my personal nom du pointe: 𣚺) something funny starts to happen. In the UTF8 world, these codepoints are accomodated by adding one extra byte to the mix which allows for up to 22bits of data (All of unicode only requires 21). In the CESU-8 world however, the code point is split according to UTF-16 surrogacy rules making two separate unicode points (each in the range U+D800 - U+DFFF). These two unicode points are then encoded individually into UTF-8 sequences. This means that we've now promoted our variable length (4 max) multibyte encoding to a variable length (6 max) multibyte-multibyte encoding. Thank you Oracle. Thank you for adding complexity to encoding rules while increasing data storage requirements. What would the world do without you?

P.S. - Java is at fault too... its 'Modified UTF-8' uses nearly identical rules.

Doctor! Doctor! It hurts when I do this!

I came across a bug while working on runkit today. On testing a batch of changes I'd made I was suddenly* running into a buffer overrun error on any request which instantiates the Runkit_Sandbox class. A few valgrind and gdb rounds later and I'd traced the corruption to apc.enabled's ini_entry->value which was actually encountering a form of double-free.

Now, I have no illusions regarding the fact that runkit is a black sheep. The things it does are contrary to PHP's design and probably shouldn't be done, so my first assumption is that APC is not the one at fault and that runkit is just coincidentally stepping on APC's toes. Time to break out the caffeine and get dirty...

After a little more digging I notice this block of code in APC:

PHP_MINIT_FUNCTION(apc)
{
   ZEND_INIT_MODULE_GLOBALS(apc, php_apc_init_globals, php_apc_shutdown_globals);

   REGISTER_INI_ENTRIES();

   /* Disable APC in cli mode unless overridden by apc.enable_cli */
   if(!APCG(enable_cli) && !strcmp(sapi_module.name, "cli")) {
       zend_alter_ini_entry("apc.enabled", sizeof("apc.enabled"), "0", 1,
                       PHP_INI_SYSTEM, PHP_INI_STAGE_ACTIVATE);
   }

   if (APCG(enabled)) {
       apc_module_init(module_number TSRMLS_CC);
   }

   return SUCCESS;
}

Ignore for the moment the fact that it's claiming we're in the ACTIVATE stage (we're really in the STARTUP phase -- but that's not where the problem lies). The problem with this statement is that zend_alter_ini_entry() uses estrndup(), a call which assumes that thread storage has been spun up and that the per-request pointer list is ready for the new entry to be indexed. During MINIT, that's just not the case.

The "right" solution for this is to move the CLI mode override to RINIT and sure enough, this fixes the overrun/double-free issues just fine. The trouble is, it's also a wasteful proposition. Compound this by the fact that the error only presents itself when all of the following conditions are met:

• APC being used on command line
• apc.enable_cli switch not set
• Thread Safety turned on (requires --enable-maintainer-zts during bulid)

Should APC take a performance hit in order to satisfy such an edge case scenario? &#@% NO! This is one bug that can be casually swept under the rug. Move along now, nothing to see here.

P.S. - In case you were wondering, the only reason I didn't notice this before is that I'd only just recently added APC to my development build. The coincidence of this bug showing up with this last batch of changes was precisely that, a coincidence.

Please don't code like this...

As a component of my job, I'm overseeing a fledgling developer in another department as he sharpens his teeth on putting together a simple application for one of my units. The first draft of his code was....well....It's a good example of why some people associate PHP with insecure applications. After some lengthy back and forths we've cleaned up the security related problems and gotten the functionality working. In my recent code review however, I came across this gem:

SELECT userid FROM (
SELECT COMPETENCY.userid,
SUM(CASE WHEN COMPETENCY.cid = 1 THEN 1 ELSE 0 END) AS comp_01,
SUM(CASE WHEN COMPETENCY.cid = 2 THEN 1 ELSE 0 END) AS comp_02,
SUM(CASE WHEN COMPETENCY.cid = 3 THEN 1 ELSE 0 END) AS comp_03,
SUM(CASE WHEN COMPETENCY.cid = 4 THEN 1 ELSE 0 END) AS comp_04,
SUM(CASE WHEN COMPETENCY.cid = 5 THEN 1 ELSE 0 END) AS comp_05,
...You get the idea...
SUM(CASE WHEN COMPETENCY.cid = 35 THEN 1 ELSE 0 END) AS comp_35,
SUM(CASE WHEN COMPETENCY.cid = 36 THEN 1 ELSE 0 END) AS comp_36,
SUM(CASE WHEN COMPETENCY.cid = 37 THEN 1 ELSE 0 END) AS comp_37,
SUM(CASE WHEN COMPETENCY.cid = 38 THEN 1 ELSE 0 END) AS comp_38,
SUM(CASE WHEN COMPETENCY.cid = 39 THEN 1 ELSE 0 END) AS comp_39,
SUM(CASE WHEN COMPETENCY.cid = 40 THEN 1 ELSE 0 END) AS comp_40,
FROM COMPETENCY, COMPETENCY_CODE
WHERE COMPETENCY.cid = COMPETENCY_CODE.cid
GROUP BY COMPETENCY.userid) AS my_view
WHERE (comp_20 = 1 AND comp_19 = 1)

Now, I generally don't like to be TOO abrasive with new developers, don't want to scare them off or anything, but can someone buy this child a SQL book?