[torqueusers] oom killer and vmem / swap usage.

[Fernando Silva]

Hi Everyone, Torque/Moab user here.

First off, we are noticing that every once in awhile OOM kicks in and
will kill a job or process on our compute nodes, how is this possible
or preventable, torque should not allow to over commit mem/job to a
node.

On another note, ideally we would like the compute nodes to never use
swap, unfortunately it seems you have to submit vmem with your
submissions. It's my understanding that vmem is actually referencing
virtual address spacing which will include the swap, correct?. Tips on
preventing jobs touching swap? That's probably a step in the right
direction stopping OOM killer.

Thanks.

________________________________

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

[Gustavo Correa]

Hi Fernando

You could write a qsub wrapper / submit filter to include vmem on the node resources requested
by the user:

http://docs.adaptivecomputing.com/torque/4-1-4/help.htm#topics/12-appendices/jobSubmissionFilter.htm

You can disable the oom killer, changing the kernel parameters
vm.overcommit_memory and vm.overcommit_ratio (in /etc/sysctl.conf)

sysctl vm.overcommit_memory=2
sysctl vm.overcommit_ratio = 95

and in /etc/sysctl.conf for the next reboot:

# vm overcommit & oom
vm.overcommit_memory=2
vm.overcommit_ratio=95

The jobs should fail when they try to allocate more memory than available,
but not kill processes or the node.

https://serverfault.com/questions/141988/avoid-linux-out-of-memory-application-teardown
engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/

Gus Correa

Post by Fernando Silva
Hi Everyone, Torque/Moab user here.
First off, we are noticing that every once in awhile OOM kicks in and
will kill a job or process on our compute nodes, how is this possible
or preventable, torque should not allow to over commit mem/job to a
node.
On another note, ideally we would like the compute nodes to never use
swap, unfortunately it seems you have to submit vmem with your
submissions. It's my understanding that vmem is actually referencing
virtual address spacing which will include the swap, correct?. Tips on
preventing jobs touching swap? That's probably a step in the right
direction stopping OOM killer.
Thanks.
________________________________
This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.
_______________________________________________
torqueusers mailing list
http://www.supercluster.org/mailman/listinfo/torqueusers