Roy van Kaathoven
Full Stack Developer

22 Oct

Isolating memory leaks in a concurrent system

While working on Advanced-Forms i have to deal with various memory issues which arise when running the server application.

The server component of Advanced-Forms is a continuously running background service which can run many different components in parallel at any given moment. Some of these components are third party maintained by other companies or open sources projects like FTP, Mail, Data Conversions, etc..

Sometimes a component will leak memory and it has to be traced down. When i tracked it down and it happens to be an open source project then i can patch it and send the patch back upstream. The change will be merged and i don't have to maintain a separate repository with my own fixes.

In other cases the component is from a company which does not accept patches or the leak is way harder to track down or the code changes are just to big to maintain a separate repository with patches.

In the second case the component can be isolated in a separate AppDomain and any leaked memory will be cleaned up when the AppDomain is released. What is a Appdomain you ask? Well it provides a layer of isolation, unloading, and security boundaries for executing managed code within a process. You would think that things like static variables are "per program" but it's actually per AppDomain.

AppDomains are costly to create so creating more then necessary is a waste of resources and will hurt performance. Reusing an AppDomain will drastically speed things up.

Lets define the interface of the Object Pool. The pool must be able to give an object and we must be able to return the object to be reused.

public interface IObjectPool where T : IDisposable
{
    T GetObject();
    void GetObject(Action func);
    void ReleaseObject(T obj);
}

The GetObject(Action func); is a convenience method to get and automatically release an object, it can be used as follows:

pool.GetObject(obj => obj.DoSomething())

The object pool must return an object, in this case an AppDomain when calling the GetObject method. When an object is already in use by another thread then create a new object and return it. This way a thread does not have to wait for other threads to complete their work, and objects will be reused.

When the objectpool is heavily used then there typically will be the same amount of objects in the pool as there are available threads (usually 2 for each available core)

public abstract class ConcurrentObjectPool : IObjectPool, IDisposable where T : IDisposable
{
    private readonly List _available = new List();

    private readonly List _inUse = new List();

    protected abstract T CreateObject();

    protected abstract void CleanUp(T obj);

    protected abstract bool ShouldCleanup(T obj);

    public T GetObject()
    {
        lock (_available)
        {
            if (_available.Count != 0)
            {
                var po = _available[0];
                _inUse.Add(po);
                _available.RemoveAt(0);
                return po;
            }
            else
            {
                var po = CreateObject();
                _inUse.Add(po);
                return po;
            }
        }
    }

    public void GetObject(Action func)
    {
        var obj = GetObject();

        try
        {
            func(obj);
        }
        finally
        {
            ReleaseObject(obj);
        }
    }

    public void ReleaseObject(T po)
    {
        lock (_available)
        {
            _inUse.Remove(po);

            if (ShouldCleanup(po))
            {
                CleanUp(po);
            }
            else
            {
                _available.Add(po);
            }
        }
    }

    public void Dispose()
    {
        _inUse.ForEach(x => x.Dispose());
        _available.ForEach(x => x.Dispose());
    }
}

Next up the AppDomainFactory which builds the AppDomains and returns an instance of the object which is located in the isolated AppDomain

public class AppDomainFactory : IDisposable
{
    protected AppDomain Domain;

    public int NumberOrRuns { get; private set; }

    public ProcessorServiceDomain()
    {
        NumberOrRuns = 0;
    }

    protected AppDomain GetDomain()
    {
        if (Domain == null)
        {
            var appDomainSetup = new AppDomainSetup
            {
                ShadowCopyFiles = "false",
                ApplicationBase = AppDomain.CurrentDomain.SetupInformation.ApplicationBase
            };

            Domain = AppDomain.CreateDomain(
                "AppDomain-" + Guid.NewGuid(),
                AppDomain.CurrentDomain.Evidence,
                appDomainSetup);
        }

        return Domain;
    }

    public T CreateInstance() where T : MarshalByRefObject
    {
        NumberOrRuns++;
        return (T)GetDomain().CreateInstanceAndUnwrap(typeof(T).Assembly.FullName, typeof(T).FullName);
    }

    public void Dispose()
    {
        AppDomain.Unload(Domain);
        Domain = null;
    }
}

Notice that the T must be a covariant of MarshalByRefObject. If the object is not MarshalByRefObject then CreateInstanceAndUnwrap() will create a new object in the new AppDomain, serialize it, transfer the serialized instance to the original AppDomain, deserialize it and call the method on the deserialized instance. Which does not contain memory leaks.

One important setting is the ShadowCopyFiles (why the hell is that a string) which must be set to false to make sure DLL/EXE are not being locked.

The implementation is really easy now, just return the object when CreateObject is called and cleanup every x runs.

public class ObjectPoolImpl : ConcurrentObjectPool
{
    protected override AppDomainFactory CreateObject()
    {
        return new AppDomainFactory();
    }

    protected override void CleanUp(AppDomainFactory obj)
    {
        obj.Dispose();
    }

    protected override bool ShouldCleanup(AppDomainFactory obj)
    {
        return obj.NumberOrRuns > 500;
    }
}

The pool can now be used:

// Default method
var pool = new ObjectPoolImpl();
var object = pool.GetObject();
object.CreateInstance().DoSomething();
pool.ReleaseObject(object);

// Alternative
pool.GetObject(obj => {
    obj.CreateInstance().DoSomething();
});

Make sure that when objects are being passed to any functions or methods that the objects are marked [Serializable] because the object has to be serialized between AppDomains