Supporting Encrypted Content-Encoding in HttpClient (Part 1 of 2) - Encoding

When you hear HTTP and encryption typically first thought goes to SSL. SSL allows for encoding entire communication between client and server, but what if there is a need to encrypt the content in a way that the other side can only store it and whoever will request it in the future will need a proper key to decrypt it? The "Encrypted Content-Encoding for HTTP" (in version 07 at the moment of writing this) aims at providing standard solution for such scenarios. In this and next post I'm going to show how it can be used with HttpClient.

The "aes128gcm" encoding

The "Encrypted Content-Encoding for HTTP" introduces new value for Content-Encoding header - aes128gcm. This encoding allows for transferring encrypted data together with information necessary to decrypt them when somebody knows the key. The encoded body consists of a coding header and encrypted content represented by number of fixed size encrypted records (last record can be smaller than others and there is a basic mechanism for preventing removal or reordering of records). For the encryption purposes the AES in Galois/Counter mode with 128 bit key is being used.

Adding encoding capability to HttpClient

What I want to achieve is support for aes128gcm encoding on top of any content type. From HttpClient perspective it seems like something that can be easily achieved with custom HttpContent which would server as wrapper over other ones.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    private readonly HttpContent _contentToBeEncrypted;
    private readonly byte[] _key;
    private readonly string _keyId;
    private readonly int _recordSize;

    public Aes128GcmEncodedContent(HttpContent contentToBeEncrypted, byte[] key, string keyId, int recordSize)
    {
        _contentToBeEncrypted = contentToBeEncrypted;

        Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
        Headers.ContentEncoding.Add("aes128gcm");
    }
}

The key, key identifier and record size are parameters which control the encoding algorithm so they will be needed. I'm also setting Content-Encoding header to aes128gcm and Content-Type to application/octet-stream. The second one is suggested by specification but there is also an option of skipping Content-Type header - the goal is to prevent exposure of original Content-Type. In order to add the encoding I've decided to override the SerializeToStreamAsync method which should allow me to perform work over streams whenever possible.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    ...

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        Stream streamToBeEncrypted = await _contentToBeEncrypted.ReadAsStreamAsync();

        await Aes128GcmEncoding.EncodeAsync(streamToBeEncrypted, stream, _key, _keyId, _recordSize);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;

        return false;
    }
}

Implementing the encoding algorithm

At high level the encoding algorithm consists of just a few steps, so it is simpler to show the implementation and then describe each of the steps.

public static class Aes128GcmEncoding
{
    public static async Task EncodeAsync(Stream source, Stream destination, byte[] key, string keyId, int recordSize)
    {
        ...

        if ((key == null) || (key.Length != 16))
            throw new ArgumentException(
                $"The '{nameof(key)}' parameter must be 16 octets long.", nameof(key));

        if (recordSize < 18)
            throw new ArgumentException(
                $"The '{nameof(recordSize)}' parameter must be at least 18.", nameof(recordSize));

        byte[] salt = GenerateSalt();

        byte[] pseudorandomKey = HmacSha256(salt, key);
        byte[] contentEncryptionKey = GetContentEncryptionKey(pseudorandomKey);

        await WriteCodingHeaderAsync(destination, salt, keyId, recordSize);

        await EncryptContentAsync(source, destination, recordSize, pseudorandomKey, contentEncryptionKey);
    }
}

First some validations must be performed (I've skipped the null checks for brevity):

  • The key must be provided and must have exactly 128 bits. This comes from the encryption algorithm being used.
  • The record size must be at least 18 bytes. This comes from the fact that the encryption algorithm produces output longer by 16 bytes from the source and a delimiter byte is required (so if the record size will be 18 bytes it means we can put exactly 1 byte of data into it).

If the provided parameters are valid the salt can be generate.

public static class Aes128GcmEncoding
{
    ...

    private static readonly SecureRandom _secureRandom = new SecureRandom();

    ...

    private static byte[] GenerateSalt()
    {
        byte[] salt = new byte[16];
        _secureRandom.NextBytes(salt, 0, 16);

        return salt;
    }

    ...
}

The salt will be used to generate the content encryption key which will serve as the actual key for encryption. This is needed in order to prevent key exposure in cases when different content is being encrypted using the same keying material - without the random part this wouldn't be safe. In order to derive the content encryption key from salt and key the HKDF algorithm must be used. First step is calculating pseudorandom key which is a result of HMAC SHA-256 hash of salt with the key.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] HmacSha256(byte[] key, byte[] value)
    {
        byte[] hash = null;

        using (HMACSHA256 hasher = new HMACSHA256(key))
        {
            hash = hasher.ComputeHash(value);
        }

        return hash;
    }

    ...
}

With pseudorandom key the content encryption key can be calculated as truncated to 16 bytes HMAC SHA-256 hash of pseudorandom key with content encryption key info parameter. The content encryption key info parameter is ASCII-encoded Content-Encoding: aes128gcm string terminated by 0x00 and 0x01 bytes.

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _contentEncryptionKeyInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        _contentEncryptionKeyInfoParameter = GetInfoParameter("Content-Encoding: aes128gcm");
    }

    ...

    private static byte[] GetInfoParameter(string infoParameterString)
    {
        byte[] infoParameter = new byte[infoParameterString.Length + 2];
        Encoding.ASCII.GetBytes(infoParameterString, 0, infoParameterString.Length, infoParameter, 0);

        infoParameter[infoParameter.Length - 1] = 1;

        return infoParameter;
    }

    private static byte[] GetContentEncryptionKey(byte[] pseudorandomKey)
    {
        byte[] contentEncryptionKey = HmacSha256(pseudorandomKey, _contentEncryptionKeyInfoParameter);
        Array.Resize(ref contentEncryptionKey, 16);

        return contentEncryptionKey;
    }

    ...
}

Those are all the prerequisites and now the body of the message can be created. First the header must be written and then the encrypted records.

Writing the coding header

The coding header has four fields: salt, record size, key identifier size and key identifier. The size of the header is variable as it depends on length of the key identifier, this is why the key identifier size must be provided. The size is being kept as a single byte which means that key identifier can take up to 255 bytes. The key identifier is expected to be a UTF-8 encoded string, so with a simple method we can transform it to array of bytes and verify the length.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetKeyIdBytes(string keyId)
    {
        byte[] keyIdBytes = String.IsNullOrEmpty(keyId) ? new byte[0] : Encoding.UTF8.GetBytes(keyId);
        if (keyIdBytes.Length > Byte.MaxValue)
        {
            throw new ArgumentException($"The '{nameof(keyId)}' parameter is too long.", nameof(keyId));
        }

        return keyIdBytes;
    }

    ...
}

Second thing which needs to be transformed into byte array is the record size. There are 4 bytes reserved for record size in the header which means that it can store an unsigned 32-bit integer, but my implementation is limited to signed integer which makes it simpler (especially as some of the methods are available only over arrays which can't be bigger than 2GB).

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetRecordSizeBytes(int recordSize)
    {
        byte[] recordSizeBytes = BitConverter.GetBytes(recordSize);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSizeBytes);
        }

        return recordSizeBytes;
    }

    ...
}

The header needs to be put together and written to the target stream.

public static class Aes128GcmEncoding
{
    ...

    private static async Task WriteCodingHeaderAsync(Stream destination, byte[] salt, string keyId, int recordSize)
    {
        byte[] keyIdBytes = GetKeyIdBytes(keyId);
        byte[] recordSizeBytes = GetRecordSizeBytes(recordSize);

        byte[] codingHeader = new byte[21 + keyIdBytes.Length];

        salt.CopyTo(codingHeader, 0);
        recordSizeBytes.CopyTo(codingHeader, 16);
        codingHeader[20] = (byte)keyIdBytes.Length;
        keyIdBytes.CopyTo(codingHeader, 21);

        await destination.WriteAsync(codingHeader, 0, codingHeader.Length);
    }

    ...
}

Encrypting the content and writing the records

The implementation of AES GCM is beyond the scope of a blog post and I'm not even going to attempt doing that, instead I've chosen to use Bouncy Castle. As mentioned previously the encrypted content must be represented by fixed size records (last can be shorter), which means that source data needs to be properly split. Also the last record must be properly detected because it should be terminated with 0x02 byte while all the previous ones should be terminated with 0x01 byte. Detecting last record is easy if the content doesn't split equally between records as the final read will return smaller number of bytes than requested. The situation when content does split equally is a little bit more tricky - it requires checking if stream can be further read before writing the record to the target. The safest way to do that seems to be reading a single byte in advance and then adding it to next record. I've started the implementation with a helper method which reads the required number of bytes, prepends it with that "peeked" byte and initially sets the delimiter based on number of returned bytes.

public static class Aes128GcmEncoding
{
    ...

    private static async Task GetPlainTextAsync(Stream source, int recordDataSize, byte? peekedByte)
    {
        int readDataSize;
        byte[] plainText = new byte[recordDataSize + 1];

        if (peekedByte.HasValue)
        {
            plainText[0] = peekedByte.Value;
            readDataSize = (await source.ReadAsync(plainText, 1, recordDataSize - 1)) + 1;
        }
        else
        {
            readDataSize = await source.ReadAsync(plainText, 0, recordDataSize);
        }

        if (readDataSize == recordDataSize)
        {
            plainText[plainText.Length - 1] = 1;
        }
        else
        {
            Array.Resize(ref plainText, readDataSize + 1);
            plainText[plainText.Length - 1] = 2;
        }

        return plainText;
    }

    ...
}

The number of bytes to read is being calculated from record size by subtracting the previously mentioned overhead (17 bytes). This allows for putting the core of encryption routine together.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = 2;
                }
            }

            // TODO: Encrypt and write the record
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}

The AES GCM requires one more parameter which needs to be calculate - nonce. The aes128gcm encoding uses nonce additionally for removal and reordering protection by performing a XOR with record sequence number as last step. The first argument for that XOR is the result of same the HKDF function as the one discussed in context of content encryption key, the difference is key info parameter (Content-Encoding: nonce) and length (12 bytes).

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _nonceInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        ...
        _nonceInfoParameter = GetInfoParameter("Content-Encoding: nonce");
    }

    ...

    private static byte[] GetNonce(byte[] pseudorandomKey, ulong recordSequenceNumber)
    {
        byte[] nonce = HmacSha256(pseudorandomKey, _nonceInfoParameter);
        Array.Resize(ref nonce, 12);

        byte[] recordSequenceNumberBytes = BitConverter.GetBytes(recordSequenceNumber);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSequenceNumberBytes);
        }
        int leadingNullBytesCount = 12 - recordSequenceNumberBytes.Length;

        for (int i = 0; i < leadingNullBytesCount; i++)
        {
            nonce[i] = (byte)(nonce[i] ^ 0);
        }

        for (int i = 0; i < recordSequenceNumberBytes.Length; i++)
        {
            nonce[leadingNullBytesCount + i] =
                (byte)(nonce[leadingNullBytesCount + i] ^ recordSequenceNumberBytes[i]);
        }

        return nonce;
    }

    ...
}

With nonce calculated every record can be encrypted and written to the target.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = LAST_RECORD_DELIMITER;
                }
            }

            aes128GcmCipher.Reset();
            AeadParameters aes128GcmParameters = new AeadParameters(new KeyParameter(contentEncryptionKey),
                128, GetNonce(pseudorandomKey, recordSequenceNumber));
            aes128GcmCipher.Init(true, aes128GcmParameters);

            byte[] cipherText = new byte[aes128GcmCipher.GetOutputSize(plainText.Length)];
            int lenght = aes128GcmCipher.ProcessBytes(plainText, 0, plainText.Length, cipherText, 0);
            aes128GcmCipher.DoFinal(cipherText, lenght);

            await destination.WriteAsync(cipherText, 0, cipherText.Length);
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}

Take it for a spin

With all the pieces in place the Aes128GcmEncodedContent can be used to make an actual request.

using (HttpClient encryptedContentEncodingClient = new HttpClient())
{
    HttpContent contentToBeEncrypted = new StringContent("I am the walrus", Encoding.UTF8);

    byte[] key = Convert.FromBase64String("yqdlZ+tYemfogSmv7Ws5PQ==");
    HttpContent encryptedContent = new Aes128GcmEncodedContent(contentToBeEncrypted, key, null, 4096);

    await encryptedContentEncodingClient.PostAsync("<URL>", encryptedContent);
}

Both Aes128GcmEncodedContent and Aes128GcmEncoding can be grabbed directly from here.

In next post I'm going to focus on decoding part.