Hi, thanks for your input.
> I went ahead and increased the memory limit already, but the feature seems broken.
Do you mean it does not work or do you mean that the rest of the message explains why it's broken?
Without manually increasing the memory limit, I was unable to perform basic operations in Siril. It would just randomly declare that I had insufficient free memory. With the memory limit function disabled, it works as expected.
> The OS can page out little-used pages, creating more free memory.
Yes, no problem with that, it's still more physical memory that is free to use by our program.
> It can even do this without swap, by releasing memory mapped pages.
I'm not familiar with this. But as I understand it, it will make more free space than less, so it's not a problem.
I think you're missing my point.
The OS will not reclaim pages for your use, unless you try to allocate them
. You have to apply pressure to the VM subsystem to get it to reclaim.
By checking the "free" status, all you're doing is causing Siril to return spurious errors which the OS would have been able to satisfy.
If your concern is that users might accidentally initiate an operation which uses an unusual amount of RAM, you could issue a warning like "Warning: this operation will use more than 50% of your physical RAM. Continue? (yes) (no)"
We could indeed fix the maximum amount of memory the program will use. I see three modes then: unlimited (= the OS manages the required space as swap, which may be even too large for the swap but that's what Stephen proposed), limited to a ratio of free space like it is now, and limited to some absolute amount of memory as you suggest.
Please take a look at how the GIMP manages memory. Like Siril it has to be able to process very large image files. It has a configurable tile cache, where over a certain limit (user configurable) it will start paging portions of images to disk. Because the images are managed in tiles, this is a better fit for paging than the typical row/column buffer. GIMP users would surely be very surprised if it just refused to open a large file when they have more than three tabs open in their web browser (due to a transient free-space check).
This sort of approach should have better liveliness compared to the unlimited approach above, simply because it de-prioritizes the graphics application compared to other users of memory. You could also imagine the tile cache being strictly processed in the background without impacting the GUI event loop.
You can actually simulate this to a degree on Linux by just putting Siril in a memory cgroup with the limit set to half of RAM (and swap enabled, of course). In this configuration Siril will start being paged to disk even when there's free space, which increases system liveliness. Of course, this isn't applicable to normal users.
I don't think Siril really needs to go to these sorts of extremes, but having batch processes fail to execute because of some random system load is highly unexpected. The only other software which I normally expect to fail due to physical RAM allocation is a VM hypervisor.