In this tutorial I’m going to focus on the SLAB Memory Allocator used in the Linux operating system which eliminates memory fragmentation caused by allocations and deallocations. In an operating system this is usually built on top of the Virtual Memory system which maps 4KB blocks of memory. This means the goal of the allocator is to chop those 4KB blocks up into more manageable pieces.

The virtual memory system provides an API where you can map a 4K block of memory to a specific memory address. In my kernel it looks like this:

void map_page(u32 virtualAddr);

If you fail to call map_page and attempt to read/write from a memory location that isn’t mapped then you get a page fault. I plan to talk in more detail about the virtual memory system in a future tutorial.

Allocator API

First off let’s define the interface to our memory allocator. We don’t want to confuse it with malloc so I’ve gone with kalloc. I also like to avoid sprinkling unsigned everywhere so let’s add some typedefs.

typedef unsigned int u32;typedef unsigned short u16;void kalloc_init(u32 memStart, u32 memSize);void* kalloc(u32 size);void kfree(void* ptr);

kalloc_init allocates a range of memory and sets everything up. We would call this from our kernel setup routine after we’ve initialised the virtual memory system.

Slab Algorithm

The core idea of the SLAB allocator is that you split each 4K block (from now on a Slab) into an array of blocks of the same size. For example a 4K slab can hold exactly 4 x 1024 byte objects and 32 x 128 byte objects. To avoid wasting space the object size must be a power of 2.

When you request a block of memory of a specific size the allocator finds a slab with objects that are the next power of 2 of that size and hands you a free block from it. If there are no free blocks left then a new slab is mapped from virtual memory.

Free memory inside a slab is tracked via a linked list where the pointer to the next free element is stored inside the free memory location itself! This constrains us to a minimum allocation size of 32 bits so we can store the pointer.

Allocation within a Slab

This diagram shows what’s going on for a single Slab as you allocate and free memory. In this example you start with a single 4KB Slab containing 4 x 1KB objects which has been allocated at location 10000 in memory. An unaligned low memory location like this is a bad idea in the real world but this makes it easier to understand ;)

Initially the free list is pointing at the first block, which is pointing to the second block and so on until we hit the null pointer at the end.

The first kalloc allocates 1KB of memory and writes 0xF00D to it which updates the free list pointer to point at the second block of memory. The second writes 0xBEEF and points the free list at the third block.

When the memory is freed we simply write the free list location into the freed memory location and update the free list to point at our freed block.

Let’s go ahead and write some code for this!

struct SlabEntry{    SlabEntry* next;};struct Slab{    Slab* m_nextSlab;    SlabEntry* m_freeList;    u32 m_slabStart;    u16 m_size;    void Init(u32 slabStart, u16 size);    bool Alloc(u32 size, u32& newLoc);    bool Free(u32 location);};

First we need to initialise the memory for a single slab by setting up the free list to point to each block of free memory. Initially this will be every block of memory in the slab, like this:

void Slab::Init(u32 slabStart, u16 size){    m_nextSlab = nullptr;    m_slabStart = slabStart;    m_size = size;    map_page(slabStart);    memset((void*)slabStart, 0, 0x1000);    u32 numEntries = (0x1000 / size) - 1;    m_freelist = (SlabEntry*)slabStart;    SlabEntry* current = m_freelist;    for (u32 i = 1; i<numEntries; i++)    {        current->next = (SlabEntry*)(slabStart + (i * size));        current = current->next;    }}

This function takes a virtual memory location which we will ask to be managed by this slab and the size of each object we’ll be allocating. We ask the virtual memory paging system to map this page for us and make it writeable.

Next the alloc function pops a free object off the free list!

bool Slab::Alloc(u32 size, u32& newLoc){    if (m_size != size || m_freelist == nullptr)        return false;    newLoc = (u32)m_freelist;    m_freelist = m_freelist->next;    return true;}

The free function adds an object back on the free list, after checking that the memory location being freed is within the slab:

bool Slab::Free(u32 location){    if (location < m_slabStart || location >= m_slabStart + 0x1000)        return false;    SlabEntry* newEntry = (SlabEntry*)location;    newEntry->next = m_freelist;    m_freelist = newEntry;    return true;}

There is a whole bunch of extra checks you can do here which I’ve left out to make the code simpler.

Allocating Slab Metadata

So at this point we have a way of allocating memory for a single 4KB block of memory for a single allocation size. How do we build on this?

We want to support multiple slabs of memory! The simplest option is to have a linked list of them, like this:

Slab* g_slabList;

This brings up another problem. Where do we store the members inside each Slab struct? The answer is to store the Slab structs inside a special list of Slabs set aside just for this purpose - a ‘metadata’ slab.

Slab* g_slabMetaData;

Here is helper function alloc_slab_meta that will set one of these up. The trick here is to use the first object in the slab to store the fields of the metadata slab itself. It’s turtles all the way down!

static Slab* alloc_slab_meta(u32 slabStart){    Slab slabMetadata;    slabMetadata.Init(slabStart, sizeof(Slab), true);    u32 slabLoc;    bool didAlloc = slabMetadata.Alloc(sizeof(Slab), slabLoc);    assert(didAlloc && slabStart == slabLoc);    Slab* newSlabMeta = (Slab*)slabLoc;    *newSlabMeta = slabMetadata;    return newSlabMeta;}

Putting it all together

We need to know where our heap needs to start from in virtual memory:

u32 g_memStart;

We’re ready to write our kalloc function! First we loop through the existing slab lists to see if we can allocate memory from those:

void* kalloc(u32 size){    u32 newLoc;    Slab* slab = g_slabList;    for (; slab; slab = slab->m_nextSlab) {        if (slab->Alloc(size, newLoc)) {            return (void*)newLoc;        }    }

If that fails then we need to allocate a new slab, and hand over the first block of memory from there. The memory for slabs is held by the g_slabMetaData which we can now allocate from:

u32 slabLoc;bool didAlloc = g_slabMetaData->Alloc(sizeof(Slab), slabLoc);

If the metadata is full then we need to allocate a new metadata slab! Since this allocator never throws away allocated slabs we don’t need to keep a reference to previous metadata slabs.

if (!didAlloc) {    g_slabMetaData = alloc_slab_meta(g_memStart);    g_memStart += 0x1000;    g_slabMetaData->Alloc(sizeof(Slab), slabLoc);}

Now we have somewhere to put the metadata we can allocate a new slab and add it to the slab list!

Slab* newSlab = (Slab*)slabLoc;newSlab->Init(g_memStart, size, false);g_memStart += 0x1000;newSlab->m_nextSlab = g_slabList;g_slabList = newSlab;

Now we can allocate the block of memory inside the slab.

    newSlab->Alloc(size, newLoc);    return (void*)newLoc;}

I have intentionally kept the code here simple meaning there is a lot of room here for optimisation. For example you could maintain multiple slab linked lists allowing you to skip over slabs that are full or find a slab with the correct allocation size more quickly.

I’ve also missed out some asserts and other safety checks. I’ve included those in the full version below.

Freeing memory

To keep things simple we’re only going to free memory inside each slab - once a slab has been allocated it is alive forever! A more fancy version would throw away everything that’s been allocated.

The free function simply asks each slab if it wants to free the block. Again you could be much cleverer and have multiple lists so you can find the slab much more quickly.

void kfree(void* ptr){    if (!ptr)         return;    u32 loc = (u32)ptr;    for (Slab* slab = g_slabList; slab; slab = slab->m_nextSlab)        if (slab->Free(loc))            return;}

Final bits

That’s the core stuff! All we need now is an init function and a main function to test this thing. The init function allocates the first metadata block and sets up the globals.

void kalloc_init(u32 memStart, u32 memSize){    g_memStart = memStart;    g_slabList = nullptr;    g_slabMetaData = alloc_slab_meta(g_memStart);    g_memStart += 0x1000;}

You don’t need your very own operating system to test this! We can point this code at a block of memory, like this:

char g_memory[1024*1024];// 1 MB!int main(int argc, char** argv){    kalloc_init(&g_memory[0], sizeof(g_memory));}

You can download the whole file here and then compile and run it like this:

gcc kalloc.cpp --std=c++11 -m32 -o kalloc && ./kalloc

This also includes a couple of simple test routines. Note there are almost certainly bugs - please don’t use this in code you care about ;)

Conclusion

So that’s how the slab allocator works which concludes this tutorial!

In a future tutorial series I’m going to talk about setting up virtual memory and interrupts. You might also be interested in the previous tutorial series I wrote on how to write a bootloader!.

Going beyond 512 bytes

The BIOS only loads the first 512 bytes of the bootsector. If we want to write programs larger than 512 bytes (maybe you don’t want to and like a challenge?) we’re going to have to load more off the disk. To do this we’re going to use the int 0x13 interrupts which provide disk services.

The ah=0x2 int 0x13 command reads sectors from a drive to a target location, like this:

mov ah, 0x2    ;read sectorsmov al, 1      ;sectors to readmov ch, 0      ;cylinder idxmov dh, 0      ;head idxmov cl, 2      ;sector idxmov dl, [disk] ;disk idxmov bx, copy_target;target pointerint 0x13

The disk number is implicitly placed into dl by the BIOS on startup. Earlier on we stashed it into memory with mov [disk], dl.

So we can now load 512 bytes from the second sector into memory! Let’s make use of that and move the hello world printing code from last time beyond the first 512 bytes of disk. Like this:

times 510 - ($-$$) db 0dw 0xaa55copy_target:bits 32    hello: db "Hello more than 512 bytes world!!",0boot2:    mov esi,hello    mov ebx,0xb8000.loop:    lodsb    or al,al    jz halt    or eax,0x0F00    mov word [ebx], ax    add ebx,2    jmp .loophalt:    cli    hlttimes 1024 - ($-$$) db 0

The last line pads our bootloader to 1024 bytes so we’re not copying uninitialised bytes from disk. It’s probably easiest to download boot3.asm directly. You can compile and run it with nasm -f bin boot3.asm -o boot.bin && qemu-system-x86_64 -fda boot.bin getting this result:

Getting to C++

Next we’re going to load a C++ function that prints Hello world into memory and then call it from the bootloader. Here is the C++ function we want to compile:

extern "C" void kmain(){    const short color = 0x0F00;    const char* hello = "Hello cpp world!";    short* vga = (short*)0xb8000;    for (int i = 0; i<16;++i)        vga[i+80] = color | hello[i];}

I’ve intentionally kept the function as simple as possible - it does almost the same thing as the assembly code. The extern "C" prevents C++ from name mangling the function allowing us to call it from assembly.

Compiling and Linking

We now need to compile and link this function. To do this safely we need to create a cross compiler - this is safer than using your system’s C++ compiler since we can be certain about what instruction set and function call method the compiler uses. I also can’t for the life of me convince clang to use a linker script and stop adding loads of OSX specific stuff. Since we’re building a 32 bit operating system we also want the compiler to output 32 bit instructions - not 64 bit ones!

Compiling a cross compiler is an absolute pain - but luckily I’ve hacked a homebrew tap you can use:

brew tap zanders3/homebrew-gcc_cross_compilersbrew install --debug i386-elf-gcc

Interestingly upgrading Mac OSX can mess gcc and all sorts of things up so try these commands if things fall over.

xcode-select --installbrew reinstall gcc

This will install i386-elf-_g++ on your path which will compile and link stuff for us! Installing takes a while - you might want to grab a cuppa whilst it compiles.

Linker Script

We need to tell gcc how to link our cpp file and asm files together. We want the boot4.asm code positioned at offset 0x7c00 so that the 510th bytes equal 0xAA55 which makes it a valid bootsector. We then want all of the C++ code placed after that in the file.

ENTRY(boot)OUTPUT_FORMAT("binary")SECTIONS {    . = 0x7c00;    .text :    {        *(.boot)        *(.text)    }    .rodata :    {        *(.rodata)    }    .data :    {        *(.data)    }    .bss :    {        *(.bss)    }}

The ENTRY(boot) means the entry point of the program is the boot symbol. OUTPUT_FORMAT("binary") tells gcc to output raw assembly directly. It will otherwise output a binary file in ELF. The . = 0x7c00 tells it to start outputting code at that offset, similar to the org 0x7c00 command.

Calling C++ from Assembly

Next we need to modify our assembly so it can be linked with gcc. First we’re going to put everything in the .boot section so it’s placed first as well as define a global boot: symbol so the linker knows about the entry point.

section .bootbits 16global bootboot:

Next we’ll hack the disk reading function call to load more than one sector:

mov al, 6      ;sectors to read

Finally we set up a call stack for C++ to use and call the actual function. We reserve 16 KB for a kernel call stack.

mov esp,kernel_stack_topextern kmaincall kmainclihltsection .bssalign 4kernel_stack_bottom: equ $    resb 16384 ; 16 KBkernel_stack_top:

That’s it. We’re ready to compile and run this thing! Here are my versions of the linker script, cpp file and bootsector assembly!

Compiling and Running

You want to compile the assembly, then link it all together.

nasm -f elf32 boot4.asm -o boot4.oi386-elf-_g++ x86_64-elf-g++ -m64 kmain.cpp boot4.o -o kernel.bin -nostdlib -ffreestanding -std=c++11 -mno-red-zone -fno-exceptions -nostdlib -fno-rtti -Wall -Wextra -Werror -T linker.ld

We can run hexdump kernel.bin to check it did the right thing:

0000000 b8 01 24 cd 15 b8 03 00 cd 10 88 16 61 7c b4 020000010 b0 06 b5 00 b6 00 b1 02 8a 16 61 7c bb 00 7e cd0000020 13 fa 0f 01 16 5b 7c 0f 20 c0 66 83 c8 01 0f 220000030 c0 b8 10 00 8e d8 8e c0 8e e0 8e e8 8e d0 ea 220000040 7e 08 00 00 00 00 00 00 00 00 00 ff ff 00 00 000000050 9a cf 00 ff ff 00 00 00 92 cf 00 18 00 43 7c 000000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00*00001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa0000200 48 65 6c 6c 6f 20 6d 6f 72 65 20 74 68 61 6e 200000210 35 31 32 20 62 79 74 65 73 20 77 6f 72 6c 64 210000220 21 00 be 00 7e 00 00 bb 00 80 0b 00 ac 08 c0 740000230 0d 0d 00 0f 00 00 66 89 03 83 c3 02 eb ee bc f80000240 be 00 00 e8 02 00 00 00 fa f4 55 48 89 e5 48 830000250 ec 20 c7 45 f8 00 0f 00 00 48 c7 45 f0 ad 7e 000000260 00 48 c7 45 e8 00 80 0b 00 c7 45 fc 00 00 00 000000270 83 7d fc 0f 7f 34 8b 45 fc 48 63 d0 48 8b 45 f00000280 48 01 d0 0f b6 00 66 98 80 cc 0f 89 c2 8b 45 fc0000290 48 98 48 83 c0 50 48 8d 0c 00 48 8b 45 e8 48 0100002a0 c8 66 89 10 83 45 fc 01 eb c6 90 c9 c3 48 65 6c00002b0 6c 6f 20 63 70 70 20 77 6f 72 6c 64 21 00 00 0000002c0 14 00 00 00 00 00 00 00 01 7a 52 00 01 78 10 0100002d0 1b 0c 07 08 90 01 00 00 1c 00 00 00 1c 00 00 0000002e0 6a ff ff ff 63 00 00 00 00 41 0e 10 86 02 43 0d00002f0 06 02 5e c6 0c 07 08 00                        00002f8

Note the 55 aa there near offset 0x200? That means it’s a valid bootsector! Let’s try running it with qemu-system-x86_64 -fda kernel.bin and you should get this result.

Wrapping Up

That concludes this tutorial series! I hope this was useful to you.

There many more interesting low level things to explore such as setting up virtual memory, handling interrupts and writing your very own memory allocator! I hope to start exploring these in the future.

Entering 32-bit Mode

In our previous article the CPU was still in Real Mode - this means you can call BIOS functions via interrupts, use 16 bit instructions and address up to 1 megabyte of memory (unless you use segment addressing). To access more than 1MB of memory we’re going to enable the A20 line by calling the ‘A20-Gate activate’ function. To do this we’ll create a new file called boot2.asm and put this in there:

bits 16org 0x7c00boot:    mov ax, 0x2401    int 0x15 ; enable A20 bit

Whilst we’re here we’ll also set the VGA text mode to a known value to be safe. Who knows what the BIOS set it to!

mov ax, 0x3int 0x10 ; set vga text mode 3

Next we’ll enable 32 bit instructions and access to the full bit 32 registers by entering Protected Mode. To do this we need to set up a Global Descriptor Table which will define a 32 bit code segment, load it with the lgdt instruction then do a long jump to that code segment.

lgdt [gdt_pointer] ; load the gdt tablemov eax, cr0 or eax,0x1 ; set the protected mode bit on special CPU reg cr0mov cr0, eaxjmp CODE_SEG:boot2 ; long jump to the code segment

Global Descriptor Table

The GDT we’re going to set up involves 3 parts: a null segment, a code segment and a data segment. The structure of each GDT entry looks like this:

Here’s what the fields mean:

• base a 32 bit value describing where the segment begins
• limit a 20 bit value describing where the segment ends, can be multiplied by 4096 if granularity = 1
• present must be 1 for the entry to be valid
• ring level an int between 0-3 indicating the kernel Ring Level
• direction
  • 0 = segment grows up from base, 1 = segment grows down for a data segment
  • 0 = can only execute from ring level, 1 = prevent jumping to higher ring levels
• read/write if you can read/write to this segment
• accessed if the CPU has accessed this segment
• granularity 0 = limit is in 1 byte blocks, 1 = limit is multiples of 4KB blocks
• size 0 = 16 bit mode, 1 = 32 bit protected mode

We’ll define this directly in assembly:

gdt_start:    dq 0x0gdt_code:    dw 0xFFFF    dw 0x0    db 0x0    db 10011010b    db 11001111b    db 0x0gdt_data:    dw 0xFFFF    dw 0x0    db 0x0    db 10010010b    db 11001111b    db 0x0gdt_end:

To actually load this we also need a gdt pointer structure. This is a 16 bit field containing the GDT size followed by a 32 bit pointer to the structure itself. We’ll also define the CODE_SEG and DATA_SEG value which are offsets into the gdt for use later:

gdt_pointer:    dw gdt_end - gdt_start    dd gdt_startCODE_SEG equ gdt_code - gdt_startDATA_SEG equ gdt_data - gdt_start

At this point we have enough to get into 32-bit mode! Let’s tell nasm to output 32 bit now. We’ll also set the remaining segments to point at the data segment.

bits 32boot2:    mov ax, DATA_SEG    mov ds, ax    mov es, ax    mov fs, ax    mov gs, ax    mov ss, ax

Writing to the VGA Text Buffer

Finally let’s write ‘Hello world!’ to the screen from Protected Mode! We can’t call the BIOS any more but we can write to the VGA text buffer directly. This is memory mapped to location 0xb8000. Each character on screen has this layout:

The top byte defines the character colour in the buffer as an int value from 0-15 with 0 = black, 1 = blue and 15 = white. The bottom byte defines an ASCII code point. Using this information we can write some assembly that writes ‘Hello World’ in blue text:

    mov esi,hello    mov ebx,0xb8000.loop:    lodsb    or al,al    jz halt    or eax,0x0100    mov word [ebx], ax    add ebx,2    jmp .loophalt:    cli    hlthello: db "Hello world!",0

Let’s run the thing!

We finally have everything! Save the whole thing as a boot2.asm file (source available here) then run it with nasm -f bin boot2.asm && qemu-system-x86_64 -fda boot.bin. You should get something like this!

Next Steps

Amazing! In part 3 of this series we’ll look at loading a C++ function that we’ve compiled into memory and then call it from our bootloader. Almost there!

A presentation of this article series is also available.

To follow along you’re going to need the NASM assembler and QEMU to emulate a virtual machine for us. QEMU is great because you don’t have to worry about accidentally destroying your hardware with badly written OS code ;) You can install these on Windows Subsystem for Linux or Ubuntu with this command:

sudo apt-get install nasm qemu

On a mac you can use homebrew:

brew install nasm

On Windows 10 you’ll also want to install an X Server which allows QEMU to open a window from the linux subsystem.

A Hello World Bootloader

We’re going to write a floppy disk bootloader because it doesn’t require us to mess about with file systems which helps keep things simple as possible.

When you press the power button the computer loads the BIOS from some flash memory stored on the motherboard. The BIOS initializes and self tests the hardware then loads the first 512 bytes into memory from the media device (i.e. the cdrom or floppy disk). If the last two bytes equal 0xAA55 then the BIOS will jump to location 0x7C00 effectively transferring control to the bootloader.

At this point the CPU is running in 16 bit mode, meaning only the 16 bit registers are available. Also since the BIOS only loads the first 512 bytes this means our bootloader code has to stay below that limit, otherwise we’ll hit uninitialised memory!

Let’s get hello world printing to the screen. To do this we’re going to use the ‘Write Character in TTY mode’ BIOS Interrupt Call and the load string byte instruction lobsb which loads byte at address ds:si into al. Here goes:

bits 16 ; tell NASM this is 16 bit codeorg 0x7c00 ; tell NASM to start outputting stuff at offset 0x7c00boot:    mov si,hello ; point si register to hello label memory location    mov ah,0x0e ; 0x0e means 'Write Character in TTY mode'.loop:    lodsb    or al,al ; is al == 0 ?    jz halt  ; if (al == 0) jump to halt label    int 0x10 ; runs BIOS interrupt 0x10 - Video Services    jmp .loophalt:    cli ; clear interrupt flag    hlt ; halt executionhello: db "Hello world!",0times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroesdw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

If you save this file as boot1.asm (or download it here) we can now use nasm to compile it:

nasm -f bin boot1.asm -o boot1.bin

If we run hexdump boot1.bin we can see that NASM created some code, padded some zeros then set the final two bytes to the magic number.

0000000 be 10 7c b4 0e ac 08 c0 74 04 cd 10 eb f7 fa f40000010 48 65 6c 6c 6f 20 77 6f 72 6c 64 21 00 00 00 000000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00*00001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa0000200

We can now run this thing! You can tell QEMU to boot off a floppy disk using qemu-system-x86_64 -fda boot1.bin on Windows 10 you might need to stick DISPLAY=:0 in front to open the window from WSL. You should get something like this!

Next Steps

Next we can start investigating getting into Protected Mode in Part 2!

I’ve been messing about with emulators! The GB hardware is very well documented with lots of interesting blog posts on the topic so I’ve decided to have a go at writing a gameboy hardware emulator myself.

The current status of the project is that I’ve implemented most of the CPU opcodes, GPU emulation and memory bank controller emulation required to get tetris up and running. I also wrote a GPU debugger, memory inspector and disassembler using imgui that can be used to find emulator bugs. I’m also using my own GLWT library to create a window and make an OpenGL context.

So there is just about enough there for you to get through the menus, watch the demo mode and play a game of tetris! There are some crazy bugs though - I’ve not implemented the hardware used for random number generation so you only get square blocks falling from the top of the screen making for a less than entertaining game of tetris. In addition the sprite rendering has an off by one error somewhere (a faulty instruction somewhere?) meaning the blocks are shifted down by 1. Audio is also not implemented yet.

I’m currently focused on writing some cpu unit tests for the whole thing to find and fix those faulty instructions, but it is a huge task and is taking me a while. Overall it’s been a pretty fun project to hack around with so far.

I’ve thrown the whole source code of the thing up on github - take a look! https://github.com/zanders3/gb

So I’ve decided to write my own window and platform abstraction library which will grow over time as I develop my other personal projects. The whole thing is pretty bare bones at the moment so I wouldn’t recommend you actually use it right now. Take a look!

https://github.com/zanders3/glwt2

Usage

This main.c will get you started:

#include "glwt.h"void glwt_setup(){}void glwt_draw(float time){    glClearColor(1.0f, 1.0f, 1.0f, 1.0f);    glClear(GL_COLOR_BUFFER_BIT);}int main(int argc, char *argv[]){    return glwt_init("Hello, GLWT!", 800, 600, false);}

Compiling

Simply create a new empty XCode/Visual Studio project and add glwt.(cpp/mm) and glwt.h to the project.

On Mac you will need to go to the ‘Build Phases’ XCode project settings tab and add the Cocoa.framework and OpenGL.framework for the project to link. Also ensure the glwt.mm file is an mm file not a cpp file.

Input

Pressed keys are stored in the global bool glwt_keydown[255]; structure as ascii keycodes. e.g. glwt_keydown['a']. This will work for a-z and 0-9 keys. There are additional keys defined in the Keys enum at the top of the header file. This currently only works on windows - I’ll be adding Mac support later!

I’ve had a go at writing my own JSON parser. Why?

Unity Engine’s mono runtime doesn’t have JIT support on iOS and has a relatively simple garbage collection scheme. It is fiddly with existing parsers to get them up and running in Unity. DLLs in Unity are a pain, especially if they use JIT or the wrong .NET framework version and you also lose source access when debugging.

So my main goal here is to write the simplest JSON parser and writer, in as few lines as possible, whilst minimising memory usage. Speed and performance is less of a concern.

Installation

Simply copy and paste the JSON Parser and/or the JSON Writer

Documentation

A really simple C# JSON parser in ~300 lines

• Attempts to parse JSON files with minimal GC allocation
• Nice and simple "[1,2,3]".FromJson<List<int>>() API

• Classes and structs can be parsed too!

  1 2 3 4 5

  class Foo { public int Value; } "{\"Value\":10}".FromJson<Foo>()

• Anonymous JSON is parsed into Dictionary<string,object> and List<object>

  1 2	var test = "{\"Value\":10}".FromJson<object>(); int number = ((Dictionary<string,object>)test)["Value"];

• No JIT Emit support to support AOT compilation on iOS

• Attempts are made to NOT throw an exception if the JSON is corrupted or invalid: returns null instead.
• Only public fields and property setters on classes/structs will be written to

Limitations:

• No JIT Emit support to parse structures quickly
• Limited to parsing <2GB JSON files (due to int.MaxValue)
• Parsing of abstract classes or interfaces is NOT supported and will throw an exception.

Example Usage

This example will write a list of ints to a File and read it back again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

using System;
using System.IO;
using System.Collections.Generic;
using TinyJson;
public static class JsonTest
{
  public static void Main(string[] args)
  {
    //Write a file
    List<int> values = new List<int> { 1, 2, 3, 4, 5, 6 };
    string json = values.ToJson();
    File.WriteAllText("test.json", json);
    
    //Read it back
    string fileJson = File.ReadAllText("test.json");
    List<int> fileValues = fileJson.FromJson<List<int>>();
  }
}

Save this as JsonTest.cs then compile and run with mcs JsonTest.cs && mono JsonTest.exe

Hi there!

I’ve taken the time to update my website again. This time I’ve kept the theme the same (so hopefully you won’t notice that too much has changed) but I’ve moved away from an Angular JS based site to a static one built using Hexo. This should make adding new articles and portfolio bits much easier!

I’ve gone through all of my old portfolio stuff and cleaned it up and tidied everything up a bit. Have a look around!

At the time there weren’t many mandelbrot fractal examples so I decided to try and find a way to create an all RGB image of the mandelbrot set. The resulting image is a 4096x4096 48.1Mb in PNG format. A compressed thumbnail is below; go to allRGB.com if you want to see the full image.

The code I wrote to create the image is available here. It works by first calculating a color mapping representing every RGB color, which is then sorted by Hue.

Next we generate a grayscale mandelbrot image whilst generating a histogram of the range of values in this grayscale image. The histogram is then converted into a cumulative histogram which gives us an offset into the color map. This in effect means we have a bucket of colours to choose from for each grayscale value representing the mandelbrot image.

We can now generate the image at this point by looping through the grayscale mandelbrot image, looking up which histogram bucket the gray pixel is in, use up one of those RGB colours in the bucket and use that colour in the final image.

This simple algorithm does have a drawback of producing weird shapes due to the hue sorting so we randomise the colours contained within each histogram bucket to produce a more pleasing image when viewed from a distance!

I hope that all made sense! It certainly was a fun challenge to do; I can highly recommend giving this challenge a go!

One of the major problems with the DirectX API is that it only works on Windows. Microsoft no longer has the massive monopoly that it used to have and people expect games these days to run on every flavour of platform (Windows, Linux, OSX to name a few). So in practical terms this means you have to use OpenGL; an API that is supposed to work on every platform. What a brilliant idea! So.. how hard can it be to use OpenGL?

Drawing a Triangle to the Screen

Lets try something simple, like drawing a triangle to the screen. Sounds good!

First we need to open a window. This part is platform specific, and people usually resort to a libraries like GLFW, GLEW and GLUT to handle things like creating a window, loading the OpenGL functions and handling mouse and keyboard input. For this post I’ll be using my own library I’ve cooked up called GLWT, the OpenGL Window Toolkit! The toolkit is very much a work in progress but enough stuff is working for me to be able to write this post. It only works for Macs at the moment, but I plan to make it work on Windows and Linux eventually.

Next we need to think about which version of the OpenGL API to use. OpenGL has a long history and a lot of old applications use the old, immediate mode API. These are the functions that pop up everywhere whenever you type ‘OpenGL tutorial’ into Google. For example, this is the old way of drawing a triangle to a screen:

void Draw(){    glBegin(GL_TRIANGLE_STRIP);    glVertex2f(0.0f, 0.75f);    glVertex2f(-0.5f, 0.25f);    glVertex2f(0.5f, 0.25f);    glEnd();}

The code above, whilst simple, is innefficient since you have to copy the triangles from the CPU to GPU every single frame. You also get loads of function calls and User -> Kernel mode switches. You can’t do cool things with this setup like vertex and pixel/fragment shaders, or geometry shaders, or render to a texture, or add realtime shadows, etc. Ouch.

So what’s the solution? OpenGL Core Profile! This removes all of the old fixed pipeline calls and replaces them with buffer objects which you only update when you need to. It is a lot more complicated, but you can do way more cool stuff. Here’s how you draw a triangle to the screen:

struct Vertex{    float x, y, z;};GLuint vertexBuffer, indexBuffer, vertexLayout;bool Game::Setup(int argc, const char** argv){    Window::Open(800, 600, false, "Hello World!");    GL::ClearColor(0.0f, 0.0f, 0.0f, 1.0f);    Vertex verts[] =    {        { 0.0f, 0.75f, 0.0f },        { -0.5f, 0.25f, 0.0f },        { 0.5f, 0.25f, 0.0f }    };    unsigned short inds[] =    {        0, 1, 2    };    //Create the vertex buffer object, then set it as the current buffer, then copy the vertex data onto it.    GL::GenBuffers(1, &vertexBuffer);    GL::BindBuffer(GL_ARRAY_BUFFER, vertexBuffer);    GL::BufferData(GL_ARRAY_BUFFER, sizeof(verts), &verts, GL_STATIC_DRAW);    //Create the index buffer object, set it as the current index buffer, then copy index data to it.    GL::GenBuffers(1, &indexBuffer);    GL::BindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);    GL::BufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(inds), &inds, GL_STATIC_DRAW);    //Create the vertex layout    GL::GenVertexArrays(1, &vertexLayout);    GL::BindVertexArray(vertexLayout);    //The vertex layout has 3 floats for position    GL::VertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), BUFFER_OFFSET(0));    GL::EnableVertexAttribArray(0);    return true;}void Game::Draw(float deltaTime){    GL::Clear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);    GL::BindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);    GL::BindBuffer(GL_ARRAY_BUFFER, vertexBuffer);    GL::DrawRangeElements(GL_TRIANGLES, 0, 3, 3, GL_UNSIGNED_SHORT, NULL);}

As you can see this is much more complicated, however the code above is not enough for the triangle to appear on the screen. We need to write a vertex shader and a fragment shader first. Blimey. Lets see how to do that.

Shaders!

OpenGL shaders are written in a language called GLSL, the OpenGL Shading Language.

Vertex shaders are basically a function that gets executed by your graphics card for every vertex, and it is usually where you do things like multiply it by a world view projection matrix to get the 2D position on the screen. Here is an example which just returns the vertex position without doing any processing:

in vec3 inVertex;void main(){    gl_Position = vec4(inVertex, 1.0);}

Fragment shaders are a function that gives the colour for each pixel in the triangle being drawn. The example shown here gives the triangle a nice blue colour. Yum:

out vec4 FragColor;void main(){    FragColor = vec4(0.0, 0.0, 1.0, 1.0);}

Putting it all Together

So we now have some GLSL code that will put the triangles onto the screen in a nice blue colour. We now need to tell OpenGL to load all of that shader code, turn it into a single shader object and apply it to the triangle we will draw to the screen. This code will load and compile the vertex and fragment shader, then put them into a shader program object:

//Create the shader programshaderProgram = GL::CreateProgram();//Create the vertex and fragment shadervertexShader = GL::CreateShader(GL_VERTEX_SHADER);fragmentShader = GL::CreateShader(GL_FRAGMENT_SHADER);int vertexCodeLen = (int)strlen(vertexShaderCode);int fragmentCodeLen = (int)strlen(fragmentShaderCode);//Load the shader source code and compile itGL::ShaderSource(vertexShader, 1, &vertexShaderCode, &vertexCodeLen);GL::ShaderSource(fragmentShader, 1, &fragmentShaderCode, &fragmentCodeLen);GL::CompileShader(vertexShader);GL::CompileShader(fragmentShader);char log[255];int len;bool hadError = false;GL::GetShaderInfoLog(vertexShader, 255, &len, (char*)&log);if (len > 0) {    std::cout << "Vertex Compile error:" << std::endl << log << std::endl;    hadError = true;}GL::GetShaderInfoLog(fragmentShader, 255, &len, (char*)&log);if (len > 0) {    std::cout << "Fragment Compile error:" << std::endl << log << std::endl;    hadError = true;}if (hadError)    return false;GL::AttachShader(shaderProgram, fragmentShader);GL::AttachShader(shaderProgram, vertexShader);//Bind the vertex position to the inVertex variable in the vertex shaderGL::BindAttribLocation(shaderProgram, 0, "inVertex");//Link the shader programGL::LinkProgram(shaderProgram);GL::UseProgram(shaderProgram);

Putting this code into the Game::Setup() function will cause a nice blue triangle to draw on the screen.

So obviously this is a teeny tiny first step into the exciting world of graphics programming. Now that the low-level faffing is finished with we can get into more interesting stuff like lighting models, shadows and so on to get a more accurate image. I hope to cover this sort of stuff in future articles!

Since we used Unity you can play the full game here!

The theme was ‘Heartbeats’ so naturally as big fans (and for some of us ex-employees) of Bullfrog we made a Dungeon Keeper clone, since Dungeon Keeper has a heart in the middle. The game was built with the Unity Game Engine using C#. I find Unity one of the easiest game engines to use, and it is the perfect tool for prototyping and game jamming.

At first we decided how to best split up the work. It was decided that I would focus on the code, Leanne on the Art and Trowers would handle the game design and jump into the other two disciplines where needed.

Day 1

To start with I focused on getting a basic tile based map engine going, and we made a decision to go 3D. Leanne started cranking out some monsters whilst Trowers searched for sound effects and started throwing some particle effects together. Each level used a really simple text file to define each level layout, with each different character in the text file representing a tile in the map in the level. This allowed us to create a whole bunch of levels really quickly.

So I got that all up and running and then started to focus on the biggest, scariest task - AI Pathfinding! Whilst I knew about the A* pathfinding algorithm and how it worked, I had never attempted to write one from scratch during a game jam before. Luckily thanks to this fantastic explanation and some fixing of a few off by one errors I managed to get a basic pathfinding system working.

I then spent the next few hours setting up a basic AI system. None of the characters in the game were controlled directly and instead needed to decide for themselves where they needed to go. This was done using Dijkstra’s algorithm, a variant of A* to find the nearest target. For good characters their targets involved other creatures as a first priority, followed by the Dungeon Heart itself. For the evil creatures they simply targeted the nearest (in map steps) good creature.

The AI system also had a really simple animation system that made the characters jump and move between tiles to the time of the Dungeon’s Heartbeat.

There were 6 character types implemented in the game - 3 good and 3 evil. Naturally the good guys are sneaking into your dungeon and trying to smash up your Dungeon Heart whilst the evil player needs to defend the Dungeon by setting trapdoors (a last minute inclusion, but one of the best bits) and clicking on spawn points at the correct time. If you clicked the spawn point on the beat of the heartbeat three times in row then you would spawn the most powerful creature, two times for the second most powerful and once for the least. Mess up your timing and nothing would happen! The balancing and implementation of the spawn timing system was done by Trowers whilst I was sorting out the pathfinding and character AI system.

By the end of the first night we were in a pretty good position - we had a basic pathfinding system with good creatures pathfinding to the Dungeon Heart but not doing any damage.

Day 2

So with the solid start from the previous day things were looking pretty good the following morning! The final day was spent implementing the final gameplay design making the two different sides attack each other by introducing an attack system and hit points to each character. I also set up the victory and loss conditions, making the Dungeon Heart beat faster when you are about to lose, creating lots of tension in the final moments.

The rest of the day was spent getting multiple levels with multiple waves of enemies working and fixing lots of bugs that had popped up over time. Trowers also got the sound effects plugged in and threw some quick designs for multiple levels together as well as the coding for the spawn pads. Leanne had by this point finished all of the art so was making papercraft of the characters in the game.

The final few hours were spent manically bug fixing, as a few hours before the end a load of really strange bugs reared their heads giving us an exciting dash for the finish! At the final deadline naturally the Global Game Jam servers got overloaded but we were allowed to put our entries on a Dropbox instead which we were assured would count.

A few hours later we had the judging, done by the very scientific method of whooping and arm waving, and we were positioned in second place. We were all quite happy with that as the competition was fierce!

In summary the global game jam was great fun and opened my eyes to how much fun a game jam can be! It is a nice change of pace from what sometimes feels like the snails pace of normal game development during your day job, since in your job the software you write has to actually work and not have some really nasty bugs lurking in there! ;)

Historical Event Extraction

To generate a timeline of every event in history the processing system first removed all syntax and formatting from the text. Since the raw text contains wikimedia formatting which is an undocumented and messy format with no official documentation this was achieved through use of lots of regular expressions. Next the text was split into sentences with the Stanford Named Entity Recogniser which also pulls out named entities giving additional information useful for later.

At this point we could start looking at the sentences in detail. The final algorithm ran a regular expression over each sentence searching for four consecutive numbers, e.g. 1990. It would then attempt to parse the date in the location around the number looking for months and days.

This system is clearly not foolproof and was very easily confused by pretty much all of the Maths pages in Wikipedia. This issue was resolved by doing a statistical analysis of the dates extracted from the page. If a date fell outside of two standard deviations of all of the dates of the page it would be discarded. In addition any events set more than 50 years in the future (e.g. 2050) or with negative years would be discarded. This really helped to clean things up.

Finally all of these events were indexed into a Lucene index which could happily search over and retrieve the millions of events produced. A page rank algorithm based upon each wikipedia page ranking was applied to each event to bring the important events to the top and duplicate detection and removal was applied based upon the spot signatures algorithm.

Processing the entire text of the english wikipedia was quite a challenge, since the file was around 44 GB uncompressed. To make the time to process this sane a threaded pipeline was written that splits the work over multiple threads, with queues between them. This allowed the whole wiki to be processed in around 12 hours on a reasonably powerful 12 core server my university left lying around ;)

The Website

The next challenge was the website needed to display the events in a web browser in an understandable way. I ended up using the Simile Timeline Web Widget to visualise the events which I then glued to the lucene index using a bit of JQuery and Javascript. This resulted in a nice and reasonably easy to use interface.

So in conclusion it was a fun final year project for my degree which pushed my interests far beyond the graphics and game design which I had being doing in my free time up until that point. In case you missed the link above I highly recommend checking out the demo which uses the simple english wikipedia. If you are still interested in how the whole thing works, you can read my project report.

I arrived at the studio during the final few weeks of Kinect Sports bug testing before release, so I was pulled in to assist with some of the servers that would support the final game when launched. After that I was put onto the pre production tools team helping to prepare Kinect Sports: Season Two for production. This involved fixing bugs and improving the art content, localisation and performance metrics reporting pipelines working mainly in C++ and C#.

After this the game moved into production and I was moved onto the core engine team working in C++ where I wrote a new GPU and CPU profiler for the internal game engine. I was then put on the multiplayer team where I worked directly with Artists and Designers on the ‘Challenge Play’ gameplay mode. This mode glued all of the various game modes together into a new mode which allowed you to challenge your Xbox Live friends and local profiles on the same Xbox asynchronously.

Implementing this feature involved modifying the game code for each game to support async challenges and implementing the user interface using a combination of C++ and Actionscript since Scaleform was used for the UI. Each game mode was implemented differently and in some cases by a team working from a remote office so this presented a challenge at times. There was also an online server component that delivered notifications of a challenge from an Xbox Live friend directly from the main menu screen.

The competition theme was ‘Alone’ - a tough one for sure! I ended up trying to create a spooky first person castle exploring game where you had to escape from a Castle. This was my first time using Unity game engine so it was quite a learning curve. I decided to focus mainly on graphics and art over gameplay quality for this jam.

To start with I set about creating a basic tileset for the Castle, the plan being that I would snap all of the different castle assets together in different ways to form a much bigger and more complicated building. I created a castle wall, floor, ceiling, carpet, door, light, portcullis and some crates to throw around.

Next I got all of these lovely art assets imported into Unity and I started placing and combining them into a large Castle scene. This seemed to work quite well when combined with some point lights and some linear fog in the distance to make the Castle all dark and spooky. I wish I had known about Unity’s Lightbaking tools at the time as this would have made the graphics look a lot better! Creating all of these assets and getting them imported correctly took me most of the first day.

The final day was spent getting some basic gameplay working. This involved setting up a basic first person camera and a mouse drag system that would let you drag objects in the world realistically to allow the player to look and move around the world.

I also set up the door levers and the portcullis animations along with some tutorial text and a simple trigger to reset the level at the end of the game.

In summary I think I hit the target I aimed for which was to create a nice looking game within such a short timeframe, however due to a massively over sensitive first person camera the controls left a lot to be desired! I plan to focus much more on gameplay in future game jams.

The game involves you delivering trucks from your factory to the other side of the map by building roads. You must deliver the trucks before your competitors do, so to get a monopoly on the truck delivery market you sabotage your opponents roads with nails, dynamite and bombs. It is often described as a chess game with bombs. There is also a infinite runner mode where you need to keep your truck on the map as it scrolls down the screen. It sure was a weird game design.

I was completely responsible for the graphics and wrote all of the content pipelines, shaders and special effects from scratch. I also helped out with some of the in game UI. The shaders were written in HLSL and the graphics pipeline used the DirectX 9.0 SDK provided by XNA.

The renderer batched draw calls into buckets and used static mesh instancing to draw the tiles quickly. It had support for multiple geometry passes and the final game had some sweet effects implemented such as realtime reflections on the water, realtime soft shadows, local lighting, model recolouring, smoke particles, grass billboards, lasers, shield distortion, fog of war desaturation, bloom, tone mapping and a fullscreen explosion blur. I learned a lot about graphics programming in the process.

Looking back I wish I had known more of the tricks to speed things up such as combining meshes dynamically to reduce the total number of draw calls and state switches. I did know some of the tricks such as doing a Z-buffer pre-pass to allow hidden fragments to be discarded. All in all it was a fun project to work on.

The design is pretty awesome - you place peas who are convinced they are ninjas on a dinner plate assualt course of your own design. The ninja peas will climb to the top of your towers and jump. The longer they jump and the higher they fall (without squishing themselves) the bigger the score. Score the most points to win!

I implemented the design in C++ using the DirectX 9.0 and Win32 APIs for Window and Mouse Input as well as the Box2D library to do physics collisions and response and the irrKlang library to play the sound effects and music. The sounds effect were recorded and processed by myself using Audacity and the music was from incompetech - a fantastic source of gameplay music.