Rocky Python Prime Array - Revision history

Jstratt7 at 21:30, 21 April 2023

2023-04-21T21:30:52Z

← Older revision		Revision as of 21:30, 21 April 2023
Line 68:		Line 68:

	<pre>		<pre>
	~~[test_user@rocky7 prime]~~$ pwd		$ pwd
	/home/test_user/projects/python/prime-array/		/home/test_user/projects/python/prime-array/
	~~</pre>~~
	~~<pre>~~		$ ls
	~~[test_user@rocky7 prime]~~$ ls
	logs prime_array.py python-prime-array.run		logs prime_array.py python-prime-array.run
	~~</pre>~~
	~~<pre>~~		$ sbatch python-prime-array.run
	~~[test_user@rocky7 prime]~~$ sbatch python-prime-array.run
	Submitted batch job 5771		Submitted batch job 5771
	</pre>		</pre>
Line 83:		Line 81:
	Here we can see the job is queued with all 100 jobs.		Here we can see the job is queued with all 100 jobs.
	<pre>		<pre>
	~~[test_user@rocky7 prime]~~$ squeue		$ squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)		5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)
Line 91:		Line 89:
	Here we can see the jobs are beginning to run. Two of the jobs have completed, 20 of them are currently running, and the rest are still queued.		Here we can see the jobs are beginning to run. Two of the jobs have completed, 20 of them are currently running, and the rest are still queued.
	<pre>		<pre>
			$ squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)		5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)
Line 118:		Line 117:
	Once jobs are no longer listed in the queue, we see there is a log file for every task in the job array. Each one contains the prime numbers in their respective chunk.		Once jobs are no longer listed in the queue, we see there is a log file for every task in the job array. Each one contains the prime numbers in their respective chunk.
	<pre>		<pre>
	~~[test_user@rocky7 prime_array]~~$ ls logs/*.out \| sort -V		$ ls logs/*.out \| sort -V
	logs/python_prime_array_5771-0.out		logs/python_prime_array_5771-0.out
	logs/python_prime_array_5771-1.out		logs/python_prime_array_5771-1.out
Line 147:		Line 146:
	We can see the results in order by using the cat command and then sorting the results.		We can see the results in order by using the cat command and then sorting the results.
	<pre>		<pre>
	~~[test_user@rocky7 prime_array]~~$ cat logs/python_prime_array_5771-*.out \| sort -V		$ cat logs/python_prime_array_5771-*.out \| sort -V

	3		3
Line 177:		Line 176:
	999979		999979
	999983		999983
			</pre>


			We can also use the wc command to count how many primes were found.
			<pre>
			$ cat logs/python_prime_array_5771-* \| wc -l
			78498
	</pre>		</pre>

Jstratt7 at 20:48, 21 April 2023

2023-04-21T20:48:06Z

← Older revision		Revision as of 20:48, 21 April 2023
Line 145:		Line 145:


			We can see the results in order by using the cat command and then sorting the results.
	<pre>		<pre>
	[test_user@rocky7 prime_array]$ cat logs/python_prime_array_5771-*.out \| sort -V		[test_user@rocky7 prime_array]$ cat logs/python_prime_array_5771-*.out \| sort -V

Jstratt7 at 20:46, 21 April 2023

2023-04-21T20:46:55Z

@@ Line 79: / Line 79: @@
 Submitted batch job 5771
 </pre>
 Here we can see the job is queued with all 100 jobs.
@@ Line 87: / Line 88: @@
 </pre>
-Here we can see the jobs are beginning to run.  A few of the jobs have already completed, 20 of them are currently running, and the rest are still queued.
 <pre>
               JOBID   PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
@@ Line 114: / Line 116: @@
 <pre>
-[test_user@rocky7 prime]$ ls logs/
+[test_user@rocky7 prime_array]$ cat logs/python_prime_array_5771-*.out | sort -V
-</pre>
-<pre>
-[test_user@rocky7 prime]$ cat
 </pre>

Jstratt7 at 20:40, 21 April 2023

2023-04-21T20:40:51Z

← Older revision		Revision as of 20:40, 21 April 2023
Line 9:		Line 9:
	There are three differences when turning this into a job array.		There are three differences when turning this into a job array.

	First, we've added a SBATCH parameter to define not only how many jobs but the range of task ids to produce. In our example, we're making the range 0 to 100.		First, we've added a SBATCH parameter to define not only how many jobs but the range of task ids to produce. In our example, we're making the range 0 to 99 (we could have also done 1-100).

	Secondly, for the log file pattern, we're using %A and %a instead of %j. These are patterns specific to job arrays. You can read more about the file patterns at [https://slurm.schedmd.com/sbatch.html#SECTION_%3CB%3Efilename-pattern%3C/B%3E this link]		Secondly, for the log file pattern, we're using %A and %a instead of %j. These are patterns specific to job arrays. You can read more about the file patterns at [https://slurm.schedmd.com/sbatch.html#SECTION_%3CB%3Efilename-pattern%3C/B%3E this link]

	Lastly, we pass the environment variable $SLURM_ARRAY_TASK_ID as a parameter to our code. We will need to read in this parameter and use it to determine what data to process. We know from our array definition that it will be a number from 0 to ~~100~~.		Lastly, we pass the environment variable $SLURM_ARRAY_TASK_ID as a parameter to our code. We will need to read in this parameter and use it to determine what data to process. We know from our array definition that it will be a number from 0 to 99.


Line 22:		Line 22:
	#SBATCH --job-name=PYTHON_PRIME_ARRAY		#SBATCH --job-name=PYTHON_PRIME_ARRAY
	#SBATCH --output=logs/python_prime_array_%A-%a.out		#SBATCH --output=logs/python_prime_array_%A-%a.out
	#SBATCH --array=0-~~100~~		#SBATCH --array=0-99

	module load Python		module load Python
Line 77:		Line 77:
	<pre>		<pre>
	[test_user@rocky7 prime]$ sbatch python-prime-array.run		[test_user@rocky7 prime]$ sbatch python-prime-array.run
	Submitted batch job ~~3877~~		Submitted batch job 5771
	</pre>		</pre>

Line 83:		Line 83:
	<pre>		<pre>
	[test_user@rocky7 prime]$ squeue		[test_user@rocky7 prime]$ squeue
			JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
			5771_[0-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)
			</pre>

			Here we can see the jobs are beginning to run. A few of the jobs have already completed, 20 of them are currently running, and the rest are still queued.
			<pre>
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	~~5667_~~[0-~~100~~] compute_all PYTHON_P test_use PD 0:00 1 (Resources)		5771_[23-99] compute_all PYTHON_P test_use PD 0:00 1 (Resources)
			5771_22 compute_all PYTHON_P test_use R 0:00 1 rocky2
			5771_20 compute_all PYTHON_P test_use R 0:01 1 rocky2
			5771_21 compute_all PYTHON_P test_use R 0:01 1 rocky2
			5771_3 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_4 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_5 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_6 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_7 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_8 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_9 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_10 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_11 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_12 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_13 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_14 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_15 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_16 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_17 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_18 compute_all PYTHON_P test_use R 0:03 1 rocky2
			5771_19 compute_all PYTHON_P test_use R 0:03 1 rocky2
	</pre>		</pre>

Jstratt7 at 05:03, 21 April 2023

2023-04-21T05:03:18Z

@@ Line 15: / Line 15: @@
 Lastly, we pass the environment variable $SLURM_ARRAY_TASK_ID as a parameter to our code.  We will need to read in this parameter and use it to determine what data to process.  We know from our array definition that it will be a number from 0 to 100.
 <pre>
 #!/bin/bash
@@ Line 32: / Line 34: @@
 In the python code, we'll need determine the MIN and MAX values to search.  As long as we know our CHUNKSIZE, we should be able to calculate those values using the task id being passed in as a parameter.  This way, each execution of the code will process  different chunks of numbers.
 <pre>
 import sys
@@ Line 58: / Line 62: @@
      if is_prime(i):
          print(i)
 </pre>
@@ Line 63: / Line 90: @@
-= Running Job =
+</pre>

Jstratt7 at 04:50, 21 April 2023

2023-04-21T04:50:00Z

← Older revision		Revision as of 04:50, 21 April 2023
Line 29:		Line 29:

	= Python Code =		= Python Code =

			In the python code, we'll need determine the MIN and MAX values to search. As long as we know our CHUNKSIZE, we should be able to calculate those values using the task id being passed in as a parameter. This way, each execution of the code will process different chunks of numbers.

	<pre>		<pre>

Jstratt7: Created page with "= Job Array = Job arrays allow you to run the same code many times with a different task id. The task id can then be used to determine which subset of your data to process. This strategy breaks your large job up into multiple smaller jobs that not only execute more quickly but can run concurrently. In the example of discovering prime numbers, lets say we want to discover all the primes in the first 1 million numbers. We could just create code that goes from 1 to 1000..."

2023-04-21T04:42:47Z

Created page with "= Job Array = Job arrays allow you to run the same code many times with a different task id. The task id can then be used to determine which subset of your data to process. This strategy breaks your large job up into multiple smaller jobs that not only execute more quickly but can run concurrently. In the example of discovering prime numbers, lets say we want to discover all the primes in the first 1 million numbers. We could just create code that goes from 1 to 1000..."

New page

= Job Array =
Job arrays allow you to run the same code many times with a different task id. The task id can then be used to determine which subset of your data to process. This strategy breaks your large job up into multiple smaller jobs that not only execute more quickly but can run concurrently.

In the example of discovering prime numbers, lets say we want to discover all the primes in the first 1 million numbers. We could just create code that goes from 1 to 1000000. But if we use a job array, we could create 100 jobs that each search 10000 numbers.

= Batch File =

There are three differences when turning this into a job array.

First, we've added a SBATCH parameter to define not only how many jobs but the range of task ids to produce. In our example, we're making the range 0 to 100.

Secondly, for the log file pattern, we're using %A and %a instead of %j. These are patterns specific to job arrays. You can read more about the file patterns at [https://slurm.schedmd.com/sbatch.html#SECTION_%3CB%3Efilename-pattern%3C/B%3E this link]

Lastly, we pass the environment variable $SLURM_ARRAY_TASK_ID as a parameter to our code. We will need to read in this parameter and use it to determine what data to process. We know from our array definition that it will be a number from 0 to 100.

<pre>
#!/bin/bash

#SBATCH --job-name=PYTHON_PRIME_ARRAY
#SBATCH --output=logs/python_prime_array_%A-%a.out
#SBATCH --array=0-100

module load Python

python prime_array.py $SLURM_ARRAY_TASK_ID
</pre>

= Python Code =

<pre>
import sys

# How many numbers to check for prime from each job
CHUNKSIZE = 10000

ARRAYID=0
if len(sys.argv) > 1:
ARRAYID = int(sys.argv[1])

MIN = ARRAYID * CHUNKSIZE
MAX = MIN + CHUNKSIZE

def is_prime(num):
if num <= 1:
return False
else:
for i in range(2, num):
if (num % i) == 0:
return False
return True

for i in range(MIN, MAX+1):
if is_prime(i):
print(i)
</pre>

= Running Job =