Difference between revisions of "Rocky Python Prime Array"
(One intermediate revision by the same user not shown) | |||
Line 68: | Line 68: | ||
<pre> | <pre> | ||
$ pwd | |||
/home/test_user/projects/python/prime-array/ | /home/test_user/projects/python/prime-array/ | ||
$ ls | |||
logs prime_array.py python-prime-array.run | logs prime_array.py python-prime-array.run | ||
$ sbatch python-prime-array.run | |||
Submitted batch job 5771 | Submitted batch job 5771 | ||
</pre> | </pre> | ||
Line 83: | Line 81: | ||
Here we can see the job is queued with all 100 jobs. | Here we can see the job is queued with all 100 jobs. | ||
<pre> | <pre> | ||
$ squeue | |||
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | ||
5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources) | 5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources) | ||
Line 91: | Line 89: | ||
Here we can see the jobs are beginning to run. Two of the jobs have completed, 20 of them are currently running, and the rest are still queued. | Here we can see the jobs are beginning to run. Two of the jobs have completed, 20 of them are currently running, and the rest are still queued. | ||
<pre> | <pre> | ||
$ squeue | |||
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | ||
5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources) | 5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources) | ||
Line 118: | Line 117: | ||
Once jobs are no longer listed in the queue, we see there is a log file for every task in the job array. Each one contains the prime numbers in their respective chunk. | Once jobs are no longer listed in the queue, we see there is a log file for every task in the job array. Each one contains the prime numbers in their respective chunk. | ||
<pre> | <pre> | ||
$ ls logs/*.out | sort -V | |||
logs/python_prime_array_5771-0.out | logs/python_prime_array_5771-0.out | ||
logs/python_prime_array_5771-1.out | logs/python_prime_array_5771-1.out | ||
Line 145: | Line 144: | ||
We can see the results in order by using the cat command and then sorting the results. | |||
<pre> | <pre> | ||
$ cat logs/python_prime_array_5771-*.out | sort -V | |||
3 | 3 | ||
Line 176: | Line 176: | ||
999979 | 999979 | ||
999983 | 999983 | ||
</pre> | |||
We can also use the wc command to count how many primes were found. | |||
<pre> | |||
$ cat logs/python_prime_array_5771-* | wc -l | |||
78498 | |||
</pre> | </pre> |
Latest revision as of 21:30, 21 April 2023
Job Array
Job arrays allow you to run the same code many times with a different task id. The task id can then be used to determine which subset of your data to process. This strategy breaks your large job up into multiple smaller jobs that not only execute more quickly but can run concurrently.
In the example of discovering prime numbers, lets say we want to discover all the primes in the first 1 million numbers. We could just create code that goes from 1 to 1000000. But if we use a job array, we could create 100 jobs that each search 10000 numbers.
Batch File
There are three differences when turning this into a job array.
First, we've added a SBATCH parameter to define not only how many jobs but the range of task ids to produce. In our example, we're making the range 0 to 99 (we could have also done 1-100).
Secondly, for the log file pattern, we're using %A and %a instead of %j. These are patterns specific to job arrays. You can read more about the file patterns at this link
Lastly, we pass the environment variable $SLURM_ARRAY_TASK_ID as a parameter to our code. We will need to read in this parameter and use it to determine what data to process. We know from our array definition that it will be a number from 0 to 99.
python-prime-array.run
#!/bin/bash #SBATCH --job-name=PYTHON_PRIME_ARRAY #SBATCH --output=logs/python_prime_array_%A-%a.out #SBATCH --array=0-99 module load Python python prime_array.py $SLURM_ARRAY_TASK_ID
Python Code
In the python code, we'll need determine the MIN and MAX values to search. As long as we know our CHUNKSIZE, we should be able to calculate those values using the task id being passed in as a parameter. This way, each execution of the code will process different chunks of numbers.
prime_array.py
import sys # How many numbers to check for prime from each job CHUNKSIZE = 10000 ARRAYID=0 if len(sys.argv) > 1: ARRAYID = int(sys.argv[1]) MIN = ARRAYID * CHUNKSIZE MAX = MIN + CHUNKSIZE def is_prime(num): if num <= 1: return False else: for i in range(2, num): if (num % i) == 0: return False return True for i in range(MIN, MAX+1): if is_prime(i): print(i)
Running Job
$ pwd /home/test_user/projects/python/prime-array/ $ ls logs prime_array.py python-prime-array.run $ sbatch python-prime-array.run Submitted batch job 5771
Here we can see the job is queued with all 100 jobs.
$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)
Here we can see the jobs are beginning to run. Two of the jobs have completed, 20 of them are currently running, and the rest are still queued.
$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources) 5771_22 compute_all PYTHON_P test_use R 0:00 1 rocky2 5771_20 compute_all PYTHON_P test_use R 0:01 1 rocky2 5771_21 compute_all PYTHON_P test_use R 0:01 1 rocky2 5771_3 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_4 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_5 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_6 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_7 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_8 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_9 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_10 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_11 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_12 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_13 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_14 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_15 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_16 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_17 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_18 compute_all PYTHON_P test_use R 0:03 1 rocky2 5771_19 compute_all PYTHON_P test_use R 0:03 1 rocky2
Once jobs are no longer listed in the queue, we see there is a log file for every task in the job array. Each one contains the prime numbers in their respective chunk.
$ ls logs/*.out | sort -V logs/python_prime_array_5771-0.out logs/python_prime_array_5771-1.out logs/python_prime_array_5771-2.out logs/python_prime_array_5771-3.out logs/python_prime_array_5771-4.out logs/python_prime_array_5771-5.out logs/python_prime_array_5771-6.out logs/python_prime_array_5771-7.out logs/python_prime_array_5771-8.out logs/python_prime_array_5771-9.out logs/python_prime_array_5771-10.out logs/python_prime_array_5771-11.out [truncated] logs/python_prime_array_5771-90.out logs/python_prime_array_5771-91.out logs/python_prime_array_5771-92.out logs/python_prime_array_5771-93.out logs/python_prime_array_5771-94.out logs/python_prime_array_5771-95.out logs/python_prime_array_5771-96.out logs/python_prime_array_5771-97.out logs/python_prime_array_5771-98.out logs/python_prime_array_5771-99.out
We can see the results in order by using the cat command and then sorting the results.
$ cat logs/python_prime_array_5771-*.out | sort -V 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 [truncated] 999863 999883 999907 999917 999931 999953 999959 999961 999979 999983
We can also use the wc command to count how many primes were found.
$ cat logs/python_prime_array_5771-* | wc -l 78498