VestPocket: File-based Data Storage for AOT .NET Applications

As Zoolander villain Mugato might say if he were a .NET developer, “Ahead of Time (AOT) compilation is so hot right now.” AOT is one of the focuses of the .NET 8 release, with a lot of attention given to high-performance scenarios. For the uninitiated, the act of AOT is compiling a higher-level language into a lower-level language for a better execution profile at runtime. In the case of .NET AOT, it’s targeting builds for specific environments to get near-native performance.

While many folks will undoubtedly start looking at AOT as an option to squeeze more juice out of their apps, they may have to reconsider many of their dependencies that are not AOT compatible. In that dependency vacuum, a new class of libraries will emerge to offer developers a way forward.

In today’s blog post, let’s check out VestPocket. VestPocket is a file-based “database” closer to a key-value store than a full-blown database. It aims to provide developers with an in-memory record storage option while having them persisted to disk.

What is VestPocket?

VestPocket is a .NET library that allows developers to store entities in a single file. Records are serialized using System.Text.Json and the newer AOT-compatible serialization source generators in .NET 7. The library is meant to give the developer a local database instance without the overhead of a database engine.

Some use cases might involve:

Version and deploy application-specific data in a human-readable file.
Caching data locally in a single application instance.
As a proof-of-concept database for demos and samples.
You need more resilience than in-memory collections.

You can get a solution up and running with a bit of setup. Let’s do that now!

Getting Started with VestPocket

Before getting started, you’ll need to install the VestPocket NuGet package. As mentioned in the previous section, VestPocket uses System.Text.Json and source generators to serialize records into the file-based format. Let’s start by looking at setting up our entity models.

using System.Text.Json.Serialization;  
using VestPocket;  
  
namespace Vested;  
  
[JsonSerializable(typeof(Entity))]  
[JsonSourceGenerationOptions(WriteIndented = false)]  
public partial class DatabaseContext : JsonSerializerContext  
{  
}  
  
[JsonDerivedType(typeof(Entity), nameof(Entity))]  
[JsonDerivedType(typeof(Person), nameof(Person))]  
public record Entity(string Key, int Version, bool Deleted)  
    : IEntity  
{  
    public IEntity WithVersion(int version)  
        => this with { Version = version };  
}  
  
public record Person(  
    string Key,  
    string Name,  
    int Age,  
    int Version = 0,  
    bool Deleted = false  
) : Entity(Key, Version, Deleted);

All your VestPocket entities will need a base Entity type. This type is created within your application and must implement the IEntity interface from VestPocket. Once you have a base record, you can begin implementing derived entities. In the previous code, I implemented a new Person entity. Be sure to mark your Entity instance with all derived entities using JsonDerivedType. This attribute tells System.Text.Json the derived types that the JsonSerializerContext should be aware of when building serializers and deserializers to create for use at runtime. Finally, we set up the DatabaseContext with serialization options and known entities. Again, since everything will derive from Entity, we only need one JsonSerializable attribute.

Now, let’s write some code using our new data storage.

using Vested;  
using VestPocket;  
  
var options = new VestPocketOptions { FilePath = "test.vest" };  
var store = new VestPocketStore<Entity>(DatabaseContext.Default.Entity, options);  
  
// open the database  
await store.OpenAsync(CancellationToken.None);  
  
var khalid =   
    store.Get<Person>("person/khalid") ??  
    new("person/khalid", "Khalid Abuhakmeh", 40);  
  
var maarten =   
    store.Get<Person>("person/maarten") ??  
    new("person/maarten", "Maarten Balliauw", 39);  
  
// will save a new version or increment version  
await store.Save(new Entity[] { khalid, maarten });  
  
// get all people  
var people = store.GetByPrefix<Person>("person/");  
  
foreach (var (_, name, age, _, _) in people)  
{  
    Console.WriteLine($"{name} ({age})");  
}  
  
// maintenance (clean up previous versions)  
await store.ForceMaintenance();  
  
// close store (flush any pending writes)  
await store.Close(CancellationToken.None);

If you’ve used any .NET object-relation mapper, this style will likely not surprise you. Let’s break down the sample.

var options = new VestPocketOptions { FilePath = "test.vest" };  
var store = new VestPocketStore<Entity>(DatabaseContext.Default.Entity, options);

A VestPocketStore is your access method to your data storage. The constructor takes the Entity serializer generated by System.Text.Json and an instance of VestPocketOptions. Stores are thread-safe, so you can share a single store instance across your application.

// open the database  
await store.OpenAsync(CancellationToken.None);

To create the file or access an existing file, the OpenAsync method must be invoked.

var khalid =   
    store.Get<Person>("person/khalid") ??  
    new("person/khalid", "Khalid Abuhakmeh", 40);  
  
var maarten =   
    store.Get<Person>("person/maarten") ??  
    new("person/maarten", "Maarten Balliauw", 39);

This code may look weird, but for the sake of this sample, I check to see if the sample has already stored a previous version of our records. If so, we retrieve the existing element. Otherwise, we’ll create a new record in memory.

// will save a new version or increment version  
await store.Save(new Entity[] { khalid, maarten });

Here, we attempt to store the records. It’s important to note that VestPocket is an append-only write system. So, you might have multiple versions of the same record. Here is the test.vest file on disk.

{"Creation":"2023-09-05T13:20:35.044512-04:00","LastRewrite":"2023-09-05T13:30:11.911718-04:00","CompressedEntities":null}  
{"$type":"Person","Name":"Khalid Abuhakmeh","Age":40,"Key":"person/khalid","Version":3,"Deleted":false}  
{"$type":"Person","Name":"Maarten Balliauw","Age":39,"Key":"person/maarten","Version":3,"Deleted":false}  
{"$type":"Person","Name":"Khalid Abuhakmeh","Age":40,"Key":"person/khalid","Version":4,"Deleted":false}  
{"$type":"Person","Name":"Maarten Balliauw","Age":39,"Key":"person/maarten","Version":4,"Deleted":false}

As the previous code shows, you can always retrieve records by a specific key, but you can filter the results down based on key prefixes.

// get all people  
var people = store.GetByPrefix<Person>("person/");  
  
foreach (var (_, name, age, _, _) in people)  
{  
    Console.WriteLine($"{name} ({age})");  
}

You may also want to “clean up” the database of old versions eventually. This is where the call to ForceMaintenance helps. It removes all but the latest version of an entity.

// maintenance (clean up previous versions)  
await store.ForceMaintenance();

Finally, before our application exits, we want to ensure we’ve flushed all records to disk. We can do that by calling Close on our store instance.

// close store (flush any pending writes)  
await store.Close(CancellationToken.None);

There, you have it. A simple one-file usage of VestPocket.

Conclusion

With AOT on the horizon, it’s nice to see folks experimenting with it and providing the community with packages. VestPocket is a file-based data storage mechanism with an ESENT vibe, and that’s pretty neat. While still in the experimental phase, I can see it being helpful in the use cases the author describes. Head over to the VestPocket GitHub repository and give it a star.

As always, thank you for reading and sharing my posts with friends and colleagues. Cheers.