A common pattern you see across software targeting a recent Gtk+ stack requires the use of GtkPopover. Choosing the right default-size for that popover is an exercise in futility.
Really, what you want is for the popover to shrink and grow in size based on the child, but only up to a certain point. EggScrolledWindow makes this relatively quick and easy for you to do.
Simply replace your use of GtkScrolledWinodw with EggScrolledWindow (in a template .ui file I hope) and set the max-content-width and max-content-height properties.
<object class="EggScrolledWindow">
<property name="visible">true</property>
<property name="min-content-height">20</property>
<property name="max-content-height">500</property>
<property name="min-content-width">100</property>
<property name="max-content-width">400</property>
<child>
<object ...>
</object>
</child>
</object>
Update: Patrick Griffis ported EggScrolledWindow to Python, which can be found found here.