Le forum des utilisateurs de Siril

Nouvelles:

  • L'équipe de développement de Siril est heureuse de vous accueillir dans ce forum.

    La version 0.9.11 est disponible!

    Le forum a été migré vers l'adresse www.forum.siril.org mettez à jour vos favoris !!!
  • We are delighted to welcome you to this forum

    Siril's version 0.9.11 is available!

    The forum has been migrated to www.forum.siril.org update your bookmarks!!!

Nouvelles

L'équipe de développement de Siril est heureuse de vous accueillir dans ce forum.

La version 0.9.11 est disponible!

Le forum a été migré vers l'adresse www.forum.siril.org mettez à jour vos favoris !!!

Normalisation error

stephen2615 · 235

Hors ligne stephen2615

  • Mercurien
  • *
    • Messages: 5
    • Karma: +0/-0
le: mai 28, 2019, 03:36:24 03
Greetings
I downloaded the new version and decided to give it a try. I selected the Cosmetic and Drizzle script and let it run. I returned to find the following:

10:57:58: Computing normalization...
10:57:58: Your system does not have enough memory to normalize images for stacking operation (3598 MB free for 5548 MB required)

My computer is a Windows 10 system with 4 GB of RAM.

Something has changed as I never had this before. Any ideas?

Regards
Stephen
« Modifié: mai 28, 2019, 03:39:01 03 par stephen2615 »



Hors ligne lock042

  • Administrator
  • Martien
  • *****
    • Messages: 339
    • Karma: +13/-0
Réponse #1 le: mai 28, 2019, 07:01:47 07
Yes. Now siril computes to see if you have enough memory.
Ad you can see, you don't have enough memory on your disk to make this operation.
Citer
(3598 MB free for 5548 MB required)
« Modifié: mai 28, 2019, 07:31:58 07 par lock042 »



Hors ligne stephen2615

  • Mercurien
  • *
    • Messages: 5
    • Karma: +0/-0
Réponse #2 le: mai 28, 2019, 11:23:57 11
I certainly have enough disk as you mistakenly stated that: "you do not have enough memory on your disk to make this operation". The error states that my RAM is insufficient. A program that needs more RAM than a computer can readily provide usually lets the operating system control the access to RAM and that is what demand paging does. This is the first time that I have ever seen a program "run out of RAM". I realise that Siril is not a native Windows program but all OS's handle paging the same way. I would like to see if this happens with a Linux system.

I just ran it again and the system was paging like I have never seen it page before. It failed with 22 GB of storage available. My disk is a SSD. Running it again with Drizzle and no Cosmetic processing also resulted in the memory error. I used Drizzle in previous versions and never had this problem.
« Modifié: mai 28, 2019, 01:45:29 13 par stephen2615 »



Hors ligne vinvin

  • Administrator
  • Mercurien
  • *****
    • Messages: 25
    • Karma: +4/-0
Réponse #3 le: mai 28, 2019, 03:17:07 15
Hello Stephen, the goal was indeed to not run out of memory. If you want to enable siril access to paged memory, you can increase the memory ratio use between 1 and 2, a value of 1 meaning all free memory will be used, a value of 2 increases this to twice this value. But I'd still recommend getting more RAM than using disks for that, it'll be slower and increases disk wear.

You should be able to do it with this command: https://free-astro.org/index.php/Siril:Commands#setmem    because I'm not sure it can be done from the GUI settings.
« Modifié: mai 28, 2019, 03:23:23 15 par vinvin »



Hors ligne lock042

  • Administrator
  • Martien
  • *****
    • Messages: 339
    • Karma: +13/-0
Réponse #4 le: mai 28, 2019, 03:23:33 15
Here the setting window where you can change parameters.



Hors ligne stephen2615

  • Mercurien
  • *
    • Messages: 5
    • Karma: +0/-0
Réponse #5 le: mai 29, 2019, 12:51:40 00
Thanks for the response. I still am interested in why you don't allow the operating system to manage the memory. I had no problems with the prior version of Siril and I am sure that there are many people out there that just can't throw extra RAM onto their systems for any number of reasons. When using drizzle, the size of the files are enormous (over 500 MB per file) and trying to stack them is going to push many systems to the edge.



Hors ligne vinvin

  • Administrator
  • Mercurien
  • *****
    • Messages: 25
    • Karma: +4/-0
Réponse #6 le: mai 29, 2019, 01:17:47 01
The reason is quite simple: many systems don't have swap (paged memory), that the three computers I tested siril on. When you reach the limit of free memory in that case, most often the only way out is to pull the plug. Then you have to restart everything the computer was doing. We may clarify this option in the next release, by adding a checkbox next to the ratio maybe, that would indicate "automatic OS memory management" and disable checks.



Hors ligne elladan

  • Mercurien
  • *
    • Messages: 7
    • Karma: +0/-0
Réponse #7 le: juin 02, 2019, 09:58:47 09
I've also been having a problem with this new feature, on Linux. I went ahead and increased the memory limit already, but the feature seems broken.

Tracking "free" memory is wrong, since on a well functioning OS no memory is ever free. Rather, it's used as cache.

Assuming you're treating cache as free, it's still wrong. The OS can page out little-used pages, creating more free memory. It can even do this without swap, by releasing memory mapped pages.

Rather than trying to track "free" pages, a better approach is to maintain a maximum program size which defaults to, say, 1/2 of physical RAM but which the user can also change.




Hors ligne vinvin

  • Administrator
  • Mercurien
  • *****
    • Messages: 25
    • Karma: +4/-0
Réponse #8 le: juin 02, 2019, 02:08:43 14
Hi, thanks for your input.
>  I went ahead and increased the memory limit already, but the feature seems broken.
Do you mean it does not work or do you mean that the rest of the message explains why it's broken?

> Assuming you're treating cache as free, it's still wrong.
I have not checked the implementation on all OS, but on Linux it's the case. And for me it works fine on systems without swap.

> The OS can page out little-used pages, creating more free memory.
Yes, no problem with that, it's still more physical memory that is free to use by our program.

> It can even do this without swap, by releasing memory mapped pages.
I'm not familiar with this. But as I understand it, it will make more free space than less, so it's not a problem.

We could indeed fix the maximum amount of memory the program will use. I see three modes then: unlimited (= the OS manages the required space as swap, which may be even too large for the swap but that's what Stephen proposed), limited to a ratio of free space like it is now, and limited to some absolute amount of memory as you suggest.



Hors ligne elladan

  • Mercurien
  • *
    • Messages: 7
    • Karma: +0/-0
Réponse #9 le: juin 03, 2019, 09:29:31 09
Hi, thanks for your input.
>  I went ahead and increased the memory limit already, but the feature seems broken.
Do you mean it does not work or do you mean that the rest of the message explains why it's broken?

Without manually increasing the memory limit, I was unable to perform basic operations in Siril. It would just randomly declare that I had insufficient free memory. With the memory limit function disabled, it works as expected.

Citer
> The OS can page out little-used pages, creating more free memory.
Yes, no problem with that, it's still more physical memory that is free to use by our program.

> It can even do this without swap, by releasing memory mapped pages.
I'm not familiar with this. But as I understand it, it will make more free space than less, so it's not a problem.

I think you're missing my point.

The OS will not reclaim pages for your use, unless you try to allocate them. You have to apply pressure to the VM subsystem to get it to reclaim.

By checking the "free" status, all you're doing is causing Siril to return spurious errors which the OS would have been able to satisfy.

If your concern is that users might accidentally initiate an operation which uses an unusual amount of RAM, you could issue a warning like "Warning: this operation will use more than 50% of your physical RAM. Continue? (yes) (no)"

Citer
We could indeed fix the maximum amount of memory the program will use. I see three modes then: unlimited (= the OS manages the required space as swap, which may be even too large for the swap but that's what Stephen proposed), limited to a ratio of free space like it is now, and limited to some absolute amount of memory as you suggest.

Please take a look at how the GIMP manages memory. Like Siril it has to be able to process very large image files. It has a configurable tile cache, where over a certain limit (user configurable) it will start paging portions of images to disk. Because the images are managed in tiles, this is a better fit for paging than the typical row/column buffer. GIMP users would surely be very surprised if it just refused to open a large file when they have more than three tabs open in their web browser (due to a transient free-space check).

This sort of approach should have better liveliness compared to the unlimited approach above, simply because it de-prioritizes the graphics application compared to other users of memory. You could also imagine the tile cache being strictly processed in the background without impacting the GUI event loop.

You can actually simulate this to a degree on Linux by just putting Siril in a memory cgroup with the limit set to half of RAM (and swap enabled, of course). In this configuration Siril will start being paged to disk even when there's free space, which increases system liveliness. Of course, this isn't applicable to normal users.

I don't think Siril really needs to go to these sorts of extremes, but having batch processes fail to execute because of some random system load is highly unexpected. The only other software which I normally expect to fail due to physical RAM allocation is a VM hypervisor.



Hors ligne vinvin

  • Administrator
  • Mercurien
  • *****
    • Messages: 25
    • Karma: +4/-0
Réponse #10 le: juin 03, 2019, 10:10:41 22
> Without manually increasing the memory limit, I was unable to perform basic operations in Siril. It would just randomly declare that I had insufficient free memory. With the memory limit function disabled, it works as expected.
For me, this is the expected behaviour, not using the swap because it's too slow or because it leads to system failure if you run out of memory when you don't have any swap. As I said before, I understand most people need this and I will add some options to allow it as you and Stephen recommended, it's just not my personal need.

I would not recommend the user dialog because most people process their images using scripts or the console mode now, but I agree for an informative output.

I am not familiar with The Gimp's memory management, I don't know if the tile cache is managed by the OS or The Gimp in what you say, but we will certainly not develop that on our side, and if the user's OS can manage it, so should it indeed.