April 10, 2005

The _alloca() Shiver

If you are working as a tester on a project being written in C/C++, do the following.

1) Get the source code.
2) Do a search through the source for "_alloca(".

Did you find any results? If no, then relax and read the rest for future reference. If yes, then your work just got 10 times harder. Feel that shudder down your spine? That's the feeling of Impending Doom©.

_alloca() allocates space on the stack, and is one of an optimizing developer's most powerful tools, but can be a nightmare if not handled with care.

"Why?" I hear you ask. "It's allocated on the stack, so that memory is freed once the function is over." Well, let's look at it this way.

You have a function that calls _alloca(), gets a pointer to that memory, and calls another function. That function incorrectly writes past the end of the allocated memory. Guess what happened? Your stack got thrashed.

You think it's not going to happen? Evidently, you haven't worked with C++ before. This kind of stuff does happen. That's one reason why I love managed code...you have to tell the compiler you're going to blow off your own foot and you have to tell your system that it has permissions to let you blow your foot off before you can remove your lower limbs.

Essentially, any function that will be called using a pointer to your _alloca()-allocated space as an argument requires in-depth testing. Make damn sure that there is no chance in Hell that those functions could ever do a buffer-overrun...because if they can, well, your code is owned.

[Updated to fix grammatical error and missing words. Note to self: Never post at midnight.]

No comments: