webshop-design/Developer-documentation.md

Overview of the task

The task was to implement a security-critical webshop that can be used to sell and buy animated images stored in a custom format. The webshop has to support CAFF (CrySyS Animated File Format). The system consists of a web service and a mobile or web client.

Overview of the implementation

During the design phase, we decided to implement a web client.

Our implementation consists of the following modules:

• CAFF previewer / caff-previewer: A program written in C that is used to extract a preview (a single frame) from a CAFF file. The extracted preview is saved in .tga (TARGA) format, which is a simple bitmap image format.
• CAFF previewer wrapper / caff-previewer-wrapper: A wrapper written in Python/Flask that provides a simple HTTP interface between CAFF previewer and the web client. The wrapper also converts the extracted preview to PNG using ImageMagick.
• Web application / webshop: A web client written in Python/Flask. This client implements the user functions.

Note: each module is hosted in its own repository.

CAFF previewer

This module is responsible for extracting previews from CAFF files. The module can be built using CMake, detailed instructions are available in the module's README file.

Error handling

Functions that perform a complex task can fail in multiple ways. In order to be able to pinpoint the exact issue, these functions always have to return with a status code. The type of the status code is uint8_t.

Successful execution is indicated by the status code 0x00. Any outputs of the function must be returned using pointers given in the parameters. The status code is almost always used as an exit code for the program, so users can easily identify the exact issue.

Each error type must have a status code that is unique in the whole program. The status codes are divided into two parts:

• First 4 bits: identifies a certain module of the previewer
• Last 4 bits: identifies the exact problem

Memory allocation

The module provides its own memory allocation library called MagicMemory.

The library can be used to create MagicMemory contexts. All memory allocation calls must be performed through this library. If a function allocates memory, it has to take a MagicMemory context (that has the type mm_ctx) as its first parameter and perform the memory allocation inside that context.

Before the program exits, all of the contexts must be freed up.

Data validation

There are two different kinds of functions that perform data validation:

• Validator: A validator only validates data according to its format specification.
• Parser: A parser validates data and then extracts some fields from the data.

It is not unusual for a parser function to call a validator function to validate its whole input before extracting the fields.

Functions

This section gives an overview of the important source files and functions.

caff_tools.c

This file contains functions that are used to validate and parse CAFF files.

parse_caff_header is used to parse the header of a CAFF file.

validate_caff_credits is used to validate the credits section of a CAFF file.

validate_caff_animation is used to validate a single CAFF animation section.

validate_caff_file is used to validate the whole CAFF file.

parse_caff_get_first_ciff is used to parse a CAFF file and extract the first CIFF image from it.

ciff_tools.c

This file contains functions that are used to validate and parse CIFF images.

validate_ciff is used to validate a CIFF image.

parse_ciff_get_pixel_data is used to parse a CIFF image and to find the location of pixel data.

magic_memory.c

This file contains our own MagicMemory library that provides memory allocation functions.

magic_memory_begin is used to create a MagicMemory context that can be used with other MagicMemory functions later.

magic_malloc is the same as the standard malloc, but the pointer is also added to a context. Contexts are used to keep track of pointers so that they won't be lost. A context is basically a linked list of pointers.

main.c

main is the entrypoint of the previewer module.

perform_extraction takes care of the preview extraction process: it calls parser and validator functions and other helper functions. In theory, the size of a CAFF file can be really huge. In the case of our previewer, only files smaller than 512 megabytes are processed.

utils.c

read_file_to_mem is used to read CAFF files into memory.

libtarga.c

This file contains the LibTarga library by Alex Mohr.

CAFF previewer wrapper

This module makes communication between the previewer and the web client possible.

Endpoint

The module provides only one endpoint:

POST /preview

Request: The endpoint expects a single CAFF file to be sent. The content of the CAFF file is expected to be senr in a raw binary format without any extra encapsulation. The content type must be set to application/octet-stream.

Response: If the preview extraction is successful, the response contains a single PNG image. The content type is set to image/png.

Integrity checking

CAFF previewer wrapper provides two headers in the response that can be used to check the integrity of transferred files. These are the following:

• X-request-checksum contains the md5 checksum of the received file.
• X-response-checksum contains the md5 checksum of the converted file.

These checksums can be used to detect errors during file transfer.

Preview process

This section gives an overview of the preview creation process.

Converting CAFF to TGA

After the wrapper receives a CAFF file, it uses the CAFF previewer written in C to extract a preview. The output of the previewer is a TGA file.

Converting TGA to PNG

The wrapper uses ImageMagick to convert the TGA file (output of the previewer) to PNG. This step is done because PNG is a more suitable image format for the web application than TGA.

Web application

This module provides the web UI for the users.

Trying out our webshop

Since our webshop is a modern web application, it has quite a few dependencies. To make trying it out easier, we deployed the web application to our own secure Kubernetes cluster. It is available at unstablevortex.kmlabz.com.

The application connects to every dependency (Minio, CAFF previewer wrapper, SMTP and PostgreSQL server) over TLS and the configuration environment variables are stored in Vault.