Thesis Abstracts 2003

Research and Graduate Studies Electrical and Computer Engineering

Syntax Testing of the Entrust Public Key Infrastructure for Security Vulnerabilities in the X.509 Certificate

By: Maj Yves Turcotte

Supervisors: Dr. G.S. Knight (RMC) Dr. T. Dean (Queen's University)

Abstract

A Public Key Infrastructure (PKI) makes use of a set of data structures and protocols to provide secure communication between a sender and receiver with the properties of authenticity, integrity, non-repudiation and confidentiality. The infrastructure relies on Public Key Cryptography to provide digital signature and encryption of communications. Public Key Cryptography involves the use of at least one key pair, where one key is kept private while the other is made public, with the property that, in the key pair, the information encrypted by one key can only be decrypted by the other. The public key is bound to a specific user identity in what is called a digital certificate such as the X.509 certificate.

The Department of National Defence has identified a requirement for an enterprise wide PKI to manage its day-to-day business and has installed the Entrust PKI product. In the DND context, the PKI is required to provide a high level of security as vulnerabilities could have disastrous implications. Protocol implementations are generally tested extensively for conformance, but not necessarily for security vulnerabilities.

PROTOS, a research group from Oulu University in Finland, proposed a functional method for assessing protocol implementation security, based on the protocol syntax, by generating multiple, engineered mutations of protocol data units (PDU). The security of a protocol implementation can be assessed based on its ability to correctly handle or reject the malformed PDUs and not to reveal security vulnerabilities. Security vulnerabilities are often revealed in the form of stack or memory overflow, which cause the implementation or entire Operating System to cease its operation in an unexpected manner, which could enable an attacker to gain privilege or interfere with its functionality.

This thesis builds on PROTOS's approach to protocol testing by using a more generalized methodology which can be readily adapted to various protocols using the BER encoding rules. The model and tools developed as part of this thesis were used to test the Entrust PKI implementation of the X.509 standard for security vulnerabilities. The testing framework makes use of the X.509 certificate syntax to generate potentially harmful X.509 certificates with the aim of revealing security vulnerabilities in the Entrust implementation of the X.509 standard