I am using a BlockingCollection in a producer/consumer scenario, where the Ts are big objects with variable memory size. Some Ts can be as large as 700 MB, and others can be as small as 10 MB. I want to set a limit on the total memory of all the objects that are stored at any time in the collection. I don't want it to exceed 1 GB of memory, because then I get OutOfMemoryExceptions. The BlockingCollection doesn't provide this functionality out of the box. I can limit the capacity of the collection on a specific number of items, but not on a specific maximum total weight. So I might have to implement something similar by hand.
My question is: how can I implement a collection similar to the BlockingCollection, that has the basic functionality shown below?
class WeighedBlockingCollection
{
public WeighedBlockingCollection(long maximumTotalWeight);
public void Add(long itemWeight, Func itemFactory);
public void CompleteAdding();
public IEnumerable GetConsumingEnumerable();
}
The method Add has a Func parameter, because I want to delay the creation of the object until the collection has enough empty space to accept it. The itemFactory should be called synchronously during the Add call (i.e. it should be called by the producer, not the consumer). Other than that, the collection should behave exactly the same as the built-in BlockingCollection. It must be thread-safe, and support multiple producers and consumers.
With multiple producers, the Add should be roughly FIFO. It should not prioritize smaller objects over bigger objects.
In case the Add is called with an itemWeight larger than the specified maximumTotalWeight, or negative, an exception should be thrown.
Note: This question is contrived. It was inspired by this recent question.