[torqueusers] Limit max jobs submitted

Discussion:

Andrus, Brian Contractor

2011-11-22 21:15:29 UTC

All,

Ok, using torque/moab latest versions.

Have someone submitting array jobs with t>100000
This causes much grief.

How to I limit a user from having X number of total jobs in the queue (to include each individual array element)?

I see MAXJOBS, but that doesn't do it.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20111122/90ea7a1e/attachment.html

Lloyd Brown

2011-11-22 21:20:39 UTC

Permalink

You could try one of the per-queue limits
(http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/4.1queueconfig.php).
In particular:

- max_queuable
- max_running
- max_user_queuable
- max_user_run

Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

Post by Andrus, Brian Contractor
All,
Ok, using torque/moab latest versions.
Have someone submitting array jobs with t>100000
This causes much grief.
How to I limit a user from having X number of total jobs in the queue
(to include each individual array element)?
I see MAXJOBS, but that doesn?t do it.
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

glen.beane

2011-11-22 21:23:08 UTC

Permalink

Post by Lloyd Brown
You could try one of the per-queue limits
(http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/4.1queueconfig.php).
- max_queuable
- max_running
- max_user_queuable
- max_user_run

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

Andrus, Brian Contractor

2011-11-22 21:28:56 UTC

Permalink

They do work, but they do not do what I need.

See when someone submits >100000 array jobs, it fills up the job list that is used to schedule the jobs.

That is MAXJOB tells how many jobs to work with within moab to decide priority and who to run. So if MAXJOB is set to 50000, and someone submits an array of 100000, then 1/2 of their jobs get pulled in and the rest are ignored (for now) by moab.
Now along comes supersensitive.user who submits his interactive job, which will sit for way too long because moab isn't even going to schedule it. In fact, moab is ignoring it.

I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.

Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

-----Original Message-----
From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of glen.beane at gmail.com
Sent: Tuesday, November 22, 2011 1:23 PM
To: Torque Users Mailing List
Cc: torqueusers at supercluster.org
Subject: Re: [torqueusers] Limit max jobs submitted

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

glen.beane

2011-11-22 21:37:04 UTC

Permalink

Sent from my iPhone

Post by Andrus, Brian Contractor
They do work, but they do not do what I need.
See when someone submits >100000 array jobs, it fills up the job list that is used to schedule the jobs.
That is MAXJOB tells how many jobs to work with within moab to decide priority and who to run. So if MAXJOB is set to 50000, and someone submits an array of 100000, then 1/2 of their jobs get pulled in and the rest are ignored (for now) by moab.
Now along comes supersensitive.user who submits his interactive job, which will sit for way too long because moab isn't even going to schedule it. In fact, moab is ignoring it.
I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.
Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?

Yes
The torque parameters Lloyd mentioned and I confirmed

Post by Andrus, Brian Contractor
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
-----Original Message-----
From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of glen.beane at gmail.com
Sent: Tuesday, November 22, 2011 1:23 PM
To: Torque Users Mailing List
Cc: torqueusers at supercluster.org
Subject: Re: [torqueusers] Limit max jobs submitted

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

glen.beane

2011-11-22 21:39:58 UTC

Permalink

Post by glen.beane
Sent from my iPhone

Yes
The torque parameters Lloyd mentioned and I confirmed

Specifically you want max_user_queable

Post by glen.beane

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

Lloyd Brown

2011-11-22 21:38:02 UTC

Permalink

Brian,

There is a subtle but significant difference between the different
parameters.

MAXJOB is a moab parameter that specifies how many jobs it will pay
attention to from it's resource manager (torque, in this case).

What we're talking about are torque parameters that can limit the number
of queued jobs either on a per-queue basis, or a per-user, per-queue basis.

For example, if you never want to allow any user to queue more than 1000
jobs, you can do something like this:

qmgr -c 'set queue batch max_user_queueable = 1000'

Now, I don't know how job arrays would interact with this (especially
given the discrepancy between torque arrays and moab arrays), but with
traditional jobs, when they have 1000 or more enqueued total (no matter
the state), qsub will refuse to queue any more for that user.

Sorry that wasn't clear earlier.

Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

glen.beane

2011-11-22 21:41:45 UTC

Permalink

Post by Lloyd Brown
Brian,
There is a subtle but significant difference between the different
parameters.
MAXJOB is a moab parameter that specifies how many jobs it will pay
attention to from it's resource manager (torque, in this case).
What we're talking about are torque parameters that can limit the number
of queued jobs either on a per-queue basis, or a per-user, per-queue basis.
For example, if you never want to allow any user to queue more than 1000
qmgr -c 'set queue batch max_user_queueable = 1000'
Now, I don't know how job arrays would interact with this (especially
given the discrepancy between torque arrays and moab arrays), but with
traditional jobs, when they have 1000 or more enqueued total (no matter
the state), qsub will refuse to queue any more for that user.

As of torque 2.5 these attributes are checked during array submission

Post by Lloyd Brown
Sorry that wasn't clear earlier.
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

Steve Crusan

2011-11-22 21:37:13 UTC

Permalink

I think your are looking for MAXIJOB (idle jobs). MAXIJOB was designed to prevent queue stuffing. For instance on our systems, we only allow a maximum of 50 idle jobs, meaning Moab will only actively increase priority and make attempts to schedule those jobs in the idle queue. Any jobs above the 50 are placed in a blocked state, and they are basically ignored.

http://www.adaptivecomputing.com/resources/docs/mwm/6-0/6.2throttlingpolicies.php#idle

Post by Andrus, Brian Contractor
I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.
Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
-----Original Message-----
From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of glen.beane at gmail.com
Sent: Tuesday, November 22, 2011 1:23 PM
To: Torque Users Mailing List
Cc: torqueusers at supercluster.org
Subject: Re: [torqueusers] Limit max jobs submitted

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

----------------------
Steve Crusan
System Administrator
Center for Research Computing
University of Rochester
https://www.crc.rochester.edu/

Christopher Samuel

2011-11-23 03:03:18 UTC

Permalink

Post by Steve Crusan
I think your are looking for MAXIJOB (idle jobs).

Note that if you use MAXIJOB you might want to also set
(in the moab.cfg on the login nodes):

DISPLAYFLAGS HIDEBLOCKED

This will mean users won't see other users blocked jobs with
showq (makes them less worried that their job will ever run)
but still shows users their own blocked jobs.

cheers,
Chris
- --
Christopher Samuel - Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
http://www.vlsci.unimelb.edu.au/

Martin Siegert

2011-11-22 22:45:30 UTC

Permalink

Hi Brian,

I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and thus
preventing it from scheduling jobs in priority order is to

1) set MAXJOB to some value X
2) use

set queue exec max_user_queuable = Y

for the execution queues AND additionally set

set queue rte max_user_queuable = Y

for ALL routing queues that route jobs to the relevant execution queues.

Y must be much smaller than X.
Unfortunately, there currently is no limit like

set server max_user_queuable = Y

which would be a more logical way of preventing this denial-of-service
attack against moab.

Cheers,
Martin

--
Martin Siegert
Simon Fraser University

Andrus, Brian Contractor

2011-11-23 00:53:13 UTC

Permalink

Martin,

Thanks! That looks like what I need. I have a routing queue and folks use qsub rather than msub. So I will limit how many they are allowed to have in the routing queue.

Hopefully this will do the trick.

Brian

Post by Martin Siegert
Hi Brian,

I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and thus
preventing it from scheduling jobs in priority order is to
1) set MAXJOB to some value X
2) use
set queue exec max_user_queuable = Y
for the execution queues AND additionally set
set queue rte max_user_queuable = Y
for ALL routing queues that route jobs to the relevant execution queues.
Y must be much smaller than X.
Unfortunately, there currently is no limit like
set server max_user_queuable = Y
which would be a more logical way of preventing this denial-of-service
attack against moab.
Cheers,
Martin
--
Martin Siegert
Simon Fraser University

glen.beane

2011-11-23 01:24:55 UTC

Permalink

Post by Martin Siegert
Hi Brian,

I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and thus
preventing it from scheduling jobs in priority order is to
1) set MAXJOB to some value X
2) use
set queue exec max_user_queuable = Y
for the execution queues AND additionally set
set queue rte max_user_queuable = Y
for ALL routing queues that route jobs to the relevant execution queues.
Y must be much smaller than X.
Unfortunately, there currently is no limit like
set server max_user_queuable = Y

This is a reasonable feature request. I have been thinking about implementing it for a few years...

Post by Martin Siegert
which would be a more logical way of preventing this denial-of-service
attack against moab.
Cheers,
Martin
--
Martin Siegert
Simon Fraser University
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

Gareth.Williams

2011-11-23 03:28:35 UTC

Permalink

Post by Andrus, Brian Contractor
-----Original Message-----
From: glen.beane at gmail.com [mailto:glen.beane at gmail.com]
Sent: Wednesday, 23 November 2011 12:25 PM
To: Torque Users Mailing List
Subject: Re: [torqueusers] Limit max jobs submitted

Post by Martin Siegert
Hi Brian,
On Tue, Nov 22, 2011 at 09:28:56PM +0000, Andrus, Brian Contractor

Post by Andrus, Brian Contractor
They do work, but they do not do what I need.
See when someone submits >100000 array jobs, it fills up the job

list that is used to schedule the jobs.

Post by Martin Siegert

Post by Andrus, Brian Contractor
That is MAXJOB tells how many jobs to work with within moab to

decide priority and who to run. So if MAXJOB is set to 50000, and
someone submits an array of 100000, then 1/2 of their jobs get pulled
in and the rest are ignored (for now) by moab.

Post by Martin Siegert

Post by Andrus, Brian Contractor
Now along comes supersensitive.user who submits his interactive job,

which will sit for way too long because moab isn't even going to
schedule it. In fact, moab is ignoring it.

Post by Martin Siegert

Post by Andrus, Brian Contractor
I could set MAXJOB to 500000, but that still doesn't prevent a user

from submitting too many jobs such that the list that is looked at does
not over-fill.

Post by Martin Siegert

Post by Andrus, Brian Contractor
Is there a setting were if someone were to submit >X jobs (array or

otherwise), torque/moab will not even allow it in?

Post by Martin Siegert

Post by Andrus, Brian Contractor
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and

thus

Post by Martin Siegert
preventing it from scheduling jobs in priority order is to
1) set MAXJOB to some value X
2) use
set queue exec max_user_queuable = Y
for the execution queues AND additionally set
set queue rte max_user_queuable = Y
for ALL routing queues that route jobs to the relevant execution

queues.

Post by Martin Siegert
Y must be much smaller than X.
Unfortunately, there currently is no limit like
set server max_user_queuable = Y

This is a reasonable feature request. I have been thinking about
implementing it for a few years...

Hi Glen (and all),

I find your statement confusing. Do you mean you have been thinking about adding this to your torque configuration or and you thinking of implementing a new feature in torque?

On a related note, I posted a while ago that such a routing queue setup with limits does not play nicely with job dependencies. I'd probably move to such a setup if this issue were resolved. There would still be a potential issue of a user wanting to submit higher priority jobs when they already have filled their limit. I guess they have to hold their existing queued jobs if they want to handle such a situation.

It would be nice if the scheduler was more graceful about dealing with many jobs but I can see there are technical issues that make that hard. We recently had to increase limits in moab to get more jobs considered but then ran out of memory which was a worse issue.

Gareth

Post by Andrus, Brian Contractor

Post by Martin Siegert
which would be a more logical way of preventing this denial-of-

service

Post by Martin Siegert
attack against moab.
Cheers,
Martin
--
Martin Siegert
Simon Fraser University
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

glen.beane

2011-11-23 03:49:27 UTC

Permalink

Post by Gareth.Williams

Post by Martin Siegert
Hi Brian,
On Tue, Nov 22, 2011 at 09:28:56PM +0000, Andrus, Brian Contractor

Post by Andrus, Brian Contractor
They do work, but they do not do what I need.
See when someone submits >100000 array jobs, it fills up the job

list that is used to schedule the jobs.

Post by Martin Siegert

Post by Andrus, Brian Contractor
That is MAXJOB tells how many jobs to work with within moab to

decide priority and who to run. So if MAXJOB is set to 50000, and
someone submits an array of 100000, then 1/2 of their jobs get pulled
in and the rest are ignored (for now) by moab.

Post by Martin Siegert

Post by Andrus, Brian Contractor
Now along comes supersensitive.user who submits his interactive job,

which will sit for way too long because moab isn't even going to
schedule it. In fact, moab is ignoring it.

Post by Martin Siegert

Post by Andrus, Brian Contractor
I could set MAXJOB to 500000, but that still doesn't prevent a user

from submitting too many jobs such that the list that is looked at does
not over-fill.

Post by Martin Siegert

Post by Andrus, Brian Contractor
Is there a setting were if someone were to submit >X jobs (array or

otherwise), torque/moab will not even allow it in?

Post by Martin Siegert

Post by Andrus, Brian Contractor
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and

thus

queues.

Post by Martin Siegert
Y must be much smaller than X.
Unfortunately, there currently is no limit like
set server max_user_queuable = Y

This is a reasonable feature request. I have been thinking about
implementing it for a few years...

Hi Glen (and all),
I find your statement confusing. Do you mean you have been thinking about adding this to your torque configuration or and you thinking of implementing a new feature in torque?

New feature

Post by Gareth.Williams
On a related note, I posted a while ago that such a routing queue setup with limits does not play nicely with job dependencies. I'd probably move to such a setup if this issue were resolved. There would still be a potential issue of a user wanting to submit higher priority jobs when they already have filled their limit. I guess they have to hold their existing queued jobs if they want to handle such a situation.
It would be nice if the scheduler was more graceful about dealing with many jobs but I can see there are technical issues that make that hard. We recently had to increase limits in moab to get more jobs considered but then ran out of memory which was a worse issue.
Gareth

Post by Andrus, Brian Contractor

Post by Martin Siegert
which would be a more logical way of preventing this denial-of-

service

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

David Beer

2011-11-22 21:41:16 UTC

Permalink

----- Original Message -----

Post by Andrus, Brian Contractor
They do work, but they do not do what I need.
See when someone submits >100000 array jobs, it fills up the job list
that is used to schedule the jobs.
That is MAXJOB tells how many jobs to work with within moab to decide
priority and who to run. So if MAXJOB is set to 50000, and someone
submits an array of 100000, then 1/2 of their jobs get pulled in and
the rest are ignored (for now) by moab.
Now along comes supersensitive.user who submits his interactive job,
which will sit for way too long because moab isn't even going to
schedule it. In fact, moab is ignoring it.
I could set MAXJOB to 500000, but that still doesn't prevent a user
from submitting too many jobs such that the list that is looked at
does not over-fill.
Is there a setting were if someone were to submit >X jobs (array or
otherwise), torque/moab will not even allow it in?

Moab will reject jobs that exceed MAXJOB if they are submitted via msub. To govern jobs that are submitted using qsub, you have to use the TORQUE parameters. It can become a little tedious to manage, especially if you are using multiple queues, but that's what is currently available to you. Does that take care of what you're trying to do?

David

Post by Andrus, Brian Contractor
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
-----Original Message-----
From: torqueusers-bounces at supercluster.org
[mailto:torqueusers-bounces at supercluster.org] On Behalf Of
glen.beane at gmail.com
Sent: Tuesday, November 22, 2011 1:23 PM
To: Torque Users Mailing List
Cc: torqueusers at supercluster.org
Subject: Re: [torqueusers] Limit max jobs submitted

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

Post by Andrus, Brian Contractor
All,
Ok, using torque/moab latest versions.
Have someone submitting array jobs with t>100000
This causes much grief.
How to I limit a user from having X number of total jobs in the
queue
(to include each individual array element)?
I see MAXJOBS, but that doesn?t do it.
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

--
David Beer
Direct Line: 801-717-3386 | Fax: 801-717-3738
Adaptive Computing
1712 S East Bay Blvd, Suite 300
Provo, UT 84606

David Beer

2011-11-22 21:54:04 UTC

Permalink

----- Original Message -----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think your are looking for MAXIJOB (idle jobs). MAXIJOB was
designed to prevent queue stuffing. For instance on our systems, we
only allow a maximum of 50 idle jobs, meaning Moab will only
actively increase priority and make attempts to schedule those jobs
in the idle queue. Any jobs above the 50 are placed in a blocked
state, and they are basically ignored.
http://www.adaptivecomputing.com/resources/docs/mwm/6-0/6.2throttlingpolicies.php#idle

This won't completely reject the jobs, but it will help. It will prevent a user from stuffing the queue with jobs, but it won't get Moab to reject things submitted via qsub. Generally speaking, it is not possible for Moab to do that.

David

Post by Andrus, Brian Contractor
I could set MAXJOB to 500000, but that still doesn't prevent a user
from submitting too many jobs such that the list that is looked at
does not over-fill.
Is there a setting were if someone were to submit >X jobs (array or
otherwise), torque/moab will not even allow it in?
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
-----Original Message-----
From: torqueusers-bounces at supercluster.org
[mailto:torqueusers-bounces at supercluster.org] On Behalf Of
glen.beane at gmail.com
Sent: Tuesday, November 22, 2011 1:23 PM
To: Torque Users Mailing List
Cc: torqueusers at supercluster.org
Subject: Re: [torqueusers] Limit max jobs submitted
On Nov 22, 2011, at 4:20 PM, Lloyd Brown <lloyd_brown at byu.edu>

As of TORQUE 2.5 these should work correctly with job arrays

Post by Lloyd Brown
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

Post by Andrus, Brian Contractor
All,
Ok, using torque/moab latest versions.
Have someone submitting array jobs with t>100000
This causes much grief.
How to I limit a user from having X number of total jobs in the
queue
(to include each individual array element)?
I see MAXJOBS, but that doesn?t do it.
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

----------------------
Steve Crusan
System Administrator
Center for Research Computing
University of Rochester
https://www.crc.rochester.edu/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJOzBYTAAoJENS19LGOpgqKnqQIAIQxzt3cSjJK5Q8cCgMtlScX
YDttvfWlVaEh7dDV/zL2/MsbDiYBuTBPXb3xGOqvNcalR8SVYMJrFtRW7AkrwBpK
RG3c8uzL6yqGnIEirWbEFBrdy0FK7+LRILEEg2zJx5OEX2CePECrXbLqvJwka+PX
TPdSFNfYayrQ4RUxInUK7/wLI/x1oBoHq39O2yHlvbh+X043epnO2RfmvOqGW3D7
q1/S+BbIyy+ZR4NM+inXJ9FvOeSRus+ytw4ZKd1KsnX6lyxmb7dGsF9xUWmmrh+G
pupR/eqOf9YhXzO0vyF2QfAzAAi53qXeziR1jda+gB+Iw8PuKDwtU79MMCMtaAg=
=8w9h
-----END PGP SIGNATURE-----
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers

--
David Beer
Direct Line: 801-717-3386 | Fax: 801-717-3738
Adaptive Computing
1712 S East Bay Blvd, Suite 300
Provo, UT 84606