Archive for the ‘Coding’ Category

Your Applied Statistics Homework

Watch this talk by the inimitable John Rauser:

Next, meditate for awhile on Anscombe’s Quartet.

Think on these both next time you are reviewing your metrics dashboard. Remind yourself of them the next time you implement metrics in your code. Present them the next time you see a colleague being lead down the primrose path towards presenting only the mean.

Now go forth and sin against operational statistics no more.

Debug.waitForDebugger

Debugging occasional or hard-to-hit code paths can be a bitch. This is especially true when working in Android across ADB. I recently discovered some magic to ease the pain significantly: Debug#waitForDebugger()

This handy little static method suspends the current thread of execution, pending connection of a debugger via DDMS. This allows you to easily attach a debugger in hard to debug code or only in those cases when a bug condition is hit. Crazy helpful.

Also very helpful is the Debug#startMethodTracing() family of methods, which lets you programmatically start method tracing, rather than having to do it yourself through DDMS.

Django on EC2 Instance Not Publicly Accessible

I’m not that familiar with Django, so this might be common knowledge, but it took me some flailing to figure out. If you have a Django instance running on EC2 that you can’t access from your webbrowser, it’s possible that your instance is listening on the loopback interface rather than the public network interface.

As such, you probably want to run

python manage.py runserver 0.0.0.0:8000

instead of just

python manage.py runserver

since the latter defaults (at least on EC2 Ubuntu machines) to listening on localhost, which isn’t terribly helpful.

chmod causes git to see files as changed

So I was having some permissions issues with a project. So as is my wont, I began wielding “chmod -R 755” with reckless abandon. The net result was that git claimed that I’d changed every single file in my project. This is because git, by default, tracks filesystem permissions as well as contents. I didn’t want to commit my permissions changes to the project, so a little light Googling informed me that you can turn this functionality off via:

git config core.filemode false

Of course, you can apply –global flags, or apply it on a per-project basis, whichever you prefer.

Handy bit of git trivia to have.

EC2 Cannot Stat EBS Volume /dev/sdf

Could not stat /dev/sdf --- No such file or directory

The device apparently does not exist; did you specify it correctly?

If you’re getting this error when trying to do anything with an EBS volume attached to an EC2 instance, it might be because Debian/Ubuntu renamed the device. For me it was renamed from /dev/sdf/ to /dev/xvdf/. EBS actually warns you about this when you select an attach point, but I, of course, blissfully ignored that warning and then spent an hour wondering why the hell my EBS volume wasn’t working.

Interesting TLS Timing Attack

This is an interesting timing attack against pre-1.2 TLS. It exploits the fact that systems decrypting TLS packets take measurably longer to checksum 56+ bytes of data than it does to checksum <=55 bytes and that every 64 bytes after that takes an extra chunk of CPU. As a result, you can man-in-the-middle a TLS session and intentionally truncate and corrupt a packet after a boundary that corresponds with one of these transitions in the CPU step function, then time how long it takes the receiving system to respond with an error. It's not a fatal attack, to be sure. It takes over 8 million (2^23) TLS packets and perfect timings to reliably decrypt 16 bytes, but it's an interesting approach. Suggested mitigation: use TLS1.2 with an authenticated encryption scheme.

JavaScriptCore Value Encoding

My friend and coworker Myles Maxfield recently wrote up this explanation (posted here with his kind permission) of how JSCore encodes multiple types to a single JSValue primitive. I found it very interesting and I think it’s good under-the-hood information for anyone who works with JSCore or JavaScript running thereon.

This is a really neat piece of code, I think. I wrote up an explanation
of how values are encoded:

The JavaScriptCore interpreter has a primitive type (called JSValue,
inside WebKit/Source/JavaScriptCore/runtime/JSCJSValue.h) which can
represent the following types in JavaScript:
– Int32
– Double
– Cell (meaning: a pointer to another data structure)
– Boolean
– Null, undefined, empty, recently deleted (I’m grouping all these
together since they’re conceptually similar and have only one valid value
each, similar to the “Unit Type”)

The size of a JSValue, however, is 64-bits, no matter what type it is
holding. It is not a polymorphic type. Instead, it is implemented as a
union.

JavaScriptCore runs on 32-bit machines and 64-bit machines. The layout of
the type is slightly different for the two architectures. I’ll start with
32-bit machines.

The overall bit-layout of a JSValue on a 32 bit machine is as follows:

|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|
|----------------tag----------------|--------------payload--------------|

The format for everything that isn’t a double is obvious – assign each
type a unique tag, and put it’s 32-bit payload in bytes 5-8 (since ints
and pointers are both 32 bits). But then, how do we encode a double,
since a double alone takes up the whole 64-bits? JavaScriptCore is tricky
about this – it assigns the tags starting at 0xFFFFFFFF and counts down.
The reason for this is that a double encodes “QNaN” if bits 1-13 are set
(counting from the left). Therefore, if any of the tags are set, the bits
in the JSValue (when interpreted as a double) would encode a NaN. There
are also more NaN values available, so it’s possible to encode an
::actual:: double NaN. So, to tell what type a JSValue is, just see if
the tag matches, and if it doesn’t, the value is a double.

—————-

Now, for a 64-bit machine. The types are all the same size, except that
now pointers are 64 bits (ints are still 32 bits even on a 64-bit
machine). So, how can we tag the pointer, even though it takes up the
whole 64-bits? Well, JavaScriptCore makes an observation that in most
pointers, the top two bytes are usually 0 (Yes, it is true that addresses
are arbitrary due to virtual memory, but usually machines don’t get that
high when allocating addresses. To get up that high in the first place
would require this process claiming 281 terabytes of memory). We can,
then, recognize a pointer as having its top two bytes == 0. So then, how
do we distinguish a pointer whose top two bytes are 0 from a double whose
top two bytes are 0? Well, we can artificially increment the top two
bytes of the double. So, to encode a double, we’d first reinterpret the
bits as if they were an int64, then add 0x0001000000000000. To get back
the original double, just subtract that value and reinterpret the bits as
a double. This won’t overflow because the only values that would overflow
have the leftmost 16 bits set, but that’s a NaN. The same argument can be
used to show that the leftmost two bytes won’t ever end up being 0xFFFF
either.

|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|
|00000000000000000|--------------------pointer value--------------------|
| 0x0001 - 0xFFFE ------------double value------------------------------|

Well, what about the other types that a JSValue can represent? Our old
tagging mechanism (used on 32-bit processors) won’t work because our tag
has now been shortened to only only 16 bits, but in order to encode a
NaN, the first 13 have to be set. This means that we only have 3 bits to
play with, which isn’t enough (we might want to add more than 8 extra
types). But, never fear! Just like there are invalid double values, there
are also invalid pointer values! Namely, pointer values that would map to
the first page of memory (which is usually mapped with no permissions, so
these addresses are invalid). Therefore, we can use the rightmost few
bits as a tag. All the types except for Int don’t actually encode much
(don’t have much entropy), so JavaScriptCore uses combinations of 0x2,
0x4, and 0x8 to create the necessary tags. However, for ints, it actually
does use one of the NaN values (using a leftmost 0xFFFF tag) which
encodes int32s, so the rightmost 4 bytes can be the payload. This would
be faster than putting the int32 in the middle of the 64-bit value
(something like making bytes 3-6 specify the payload) because you
wouldn’t need extra shifting operations to recreate the int.

|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|xxxxxxxx|
|FFFFFFFFFFFFFFFFF|00000000000000000|-----------integer value-----------|
|00000000000000000|-------------------------------------------------TAG-| 
<- other type

Amazonian Geekcraft

One of the things I love about working at Amazon is the sheer density of brilliant, obsessive nerds with diverse skills and passions. This translates not only into an awesome work environment and Amazon’s ability to ship some seriously cool products, but also in some incredible products bashed out on the side.

Three examples:

1.) St. Optimus of Prime.

2.) Custom robotic target holder with randomized target presentation:

3.) No link for this one, but I was talking to one of my fellow Amazonians over beer the other day, and apparently his team just got finished building out metrics, monitoring, and alarming for their kegerator. Yes, in the true “metrics or GTFO” engineering attitude, they added scales to check beer weight left in the kegs and wired it up via an Arduino processor to send an email when the beer is low. Apparently the next step is to set up a real-time metrics dashboard to track beer consumption over time and to add a way for the metrics to be content-aware so they can alert people which beers are on tap and which are running low.

Android View#onDraw Not Getting Invoked

I recently ran into a problem with the onDraw method in some of my custom views not getting invoked properly. It turns out that by default Android doesn’t dispatch onDraw calls to views that are part of a view group. You can fix this for your view by calling setWillNotDraw(false); I really only needed my custom onDraw during start ups, so my code looked like:

//Enclosing Activity
@Override
protected void onResume() {
...
myView.setWillNotDraw(false);
...
}
 
//Custom View
protected void onDraw (Canvas canvas) {
setWillNotDraw(true);
}

JSON TypeError: Converting circular structure to JSON when Stringifying HTML elements

Turns out this is caused by calling Stringify directly on DOM elements. So, for instance, if you were to try to stringify the results of document.getElementsByTagName(“img”), this would choke with the error above. You could, though, iterate through the resulting array and stringify just the attributes you’re interested in, e.g. img.src.

This is because DOM elements contain several cyclical references to, e.g., their siblings in the DOM tree, their parents, etc.

I’m posting this here mostly because it wasn’t obvious to me when I ran into the error a few days ago. Many thanks to my friend Chris for pointing out the problem to me.

Return top

Magic Blue Smoke

House Rules:

1.) Carry out your own dead.
2.) No opium smoking in the elevators.
3.) In Competitions, during gunfire or while bombs are falling, players may take cover without penalty for ceasing play.
4.) A player whose stroke is affected by the simultaneous explosion of a bomb may play another ball from the same place.
4a.) Penalty one stroke.
5.) Pilsner should be in Roman type, and begin with a capital.
6.) Keep Calm and Kill It with Fire.
7.) Spammers will be fed to the Crabipede.