Of BOOL and stdbool

The C99 standard has added to the C programming language a Boolean type, _Bool and the bool alias for it. How well does this type interoperate with the Windows SDK BOOL type? The answer is, not at all well, and here’s the complete story.

How can this happen?

As an example, consider the following two files.

File1

#include <stdbool.h>

bool
false_function(void)
{
    return false;
}

File 2

#include <stdio.h>
#include <windows.h>

BOOL false_function(void);

int
main(int argc, char *argv)
{
    if (false_function())
        printf("False function returned true\n");
}

Compiling the two files with the Microsoft C/C++ Compiler (Version 19) produces the following result when the executable program is run.

False function returned true

Why does this happen?

The reason this happens, is because most compilers (including Microsoft C, GCC, and Clang) store bool values into a single byte, which can be returned in the low part of the x86 accumulator, al. In contrast, the Microsft Windows SDK defines (in file WTypes.h) BOOL as follows.

typedef long BOOL;

This Boolean value is passed around as a long, so it is returned in the register eax.

Consequently, the false_function just sets al to zero.

_false_function PROC
; Line 5
        push    ebp
        mov     ebp, esp
; Line 6
        xor     al, al
; Line 7
        pop     ebp
        ret     0
_false_function ENDP
_TEXT   ENDS
END

However, the if statement in main tests the whole accumulator eax as the return value.

        call    _false_function
        test    eax, eax
        je      SHORT $LN1@main
; Line 10
        push    OFFSET $SG88306
        call    _printf
        add     esp, 4
$LN1@main:
; Line 11
        xor     eax, eax
        pop     ebp
        ret     0

Because eax is likely to have some bits set in its most significant part, the comparison yields a true result and the program prints that the false function returned true.

How did I found out about it?

I discovered the problem while debugging a much more complex issue. At one point I was single-stepping through a function that was returning a false result. However, when its caller tested the function’s return value, it behaved as if the result was true. I discovered what was going on by single-stepping through the disassembled instructions. There I saw that the one function was setting al and the other was testing eax.

How can one avoid this problem?

The problem would be avoided, if the second file included a header file with the function’s correct declaration, such as the following.

#include <stdbool.h>

bool false_function(void);

In addition, if the first file also included the header file with the function’s declaration, an incorrect declaration (such as the one with BOOL) would be detected. You can see this in the following error message.

t.c(7): error C2371: 'false_function': redefinition; different basic types
t.h(3): note: see declaration of 'false_function'

What else can I learn from this?

Sometimes, debugging software at the level of machine instructions can help you unravel otherwise inexplicable mysteries. I advise this in Item 37 of the Effective Debugging book: Know how to view assembly code and raw memory.

Comments   Toot! Tweet Share