I have encountered a problem with the way InlineUIContainer allocates size for an image when used in a TextBlock and I cannot find a way to work around it. Below is a simple WPF application that demonstrates the issue.

public partial class MainWindow : Window
{
    private InlineUIContainer container = new InlineUIContainer(new Border()
    {
        Child = new Image()
        {
            Source = new BitmapImage(new Uri("pack://application:,,,/Smiley.png"))
        },
        Background = Brushes.Red
    });
    public MainWindow()
    {
        InitializeComponent();
        tb.Inlines.Add(container);
    }

    private void Button_Click(object sender, RoutedEventArgs e)
    {
        if (tb.Inlines.Count == 0)
        {
            tb.Inlines.Add(container);
        }
        else
        {
            tb.Inlines.Clear();
        }
    }
}

Smiley.png is a 100px X 100px image. I added the border with a background around the image just to show the space that was being taken up, this same behavior is present when the only child of the InlineUIContainer is an Image. When the image is first rendered it appears correct, however if the image is removed and readded by manually clearing and adding to the Inlines property of the TextBlock the space allocated for the image becomes the full width, as though the StretchDirection property is not being taken into account. Some examples showing the issue:

First, the full-width size without setting StretchDirection:

Now with the StretchDirection property set to DownOnly. Initially the image shows up correctly:

Pressing the toggle button twice to remove then readd the image results in the image displaying at the proper 100px X 100px size but it takes up the full width of the TextBlock:

The project in which I am encountering this bug is far more sophisticated than this example and involves limitations to how the issue can be solved. Setting the StretchDirection property on the Image itself will allow it to render correctly every time it is displayed but I'm not able to do that, the property has to be set in a style. Removing/adding the inlines to the TextBox is functionally the same as toggling visibility but in my case I am swapping between multiple sets of inlines rather than just removing and adding so any solution that doesn't involve directly manipulating the Inlines property is not usable. If the Inlines property is cleared and added to in a single click of the button this issue does not present itself, however due to the necessity to swap between multiple sets of inlines this is also not a useable solution.

Any clues as to why this is happening and what could be done to remedy the issue with the given limitations?