3. Job Execution Method

Here is an example of how to run the quantum simulator software mpiQulacs on a quantum simulator system.
(For more information about mpiQulacs, see the mpiQulacs documentation in the sidebar.)
The flow of job execution is as follows:

Log in to the compute nodes from the login server by Interactive job and build the environment
Return to the login server and create a shell script
Submit a job into the Batch queue from the login server (running a shell script)
Check the execution results

3.1. Environmental Construction

3.1.1. Installing Python packages

You log in to the compute nodes (FX 700) from the login server using an Interactive job.
The following example sets aside one node, one hour, for interactive execution of a compute node belonging to the Interactive queue.
See Job assignment (salloc) for other option.

Note

Since the number of nodes is limited, please exit when you are finished.

[username@loginvm-XXX ~]$ salloc -N 1 -p Interactive --time=1:00:00

You create an execution environment on the compute node. The following example creates a Python virtual environment in the example directory and installs the packages needed to use mpiQulacs.

[username@fx-XX-XX-XX ~]$ mkdir example
[username@fx-XX-XX-XX ~]$ cd example
[username@fx-XX-XX-XX ~]$ python3.8 -m venv qenv
[username@fx-XX-XX-XX ~]$ source ./qenv/bin/activate
(qenv) [username@fx-XX-XX-XX ~]$ pip install --upgrade pip wheel
(qenv) [username@fx-XX-XX-XX ~]$ pip install mpi4py  # Required to use mpiQulacs
(qenv) [username@fx-XX-XX-XX ~]$ pip install mpiQulacs
(qenv)  [username@fx-XX-XX-XX ~]$ exit

3.2. Creating scripts

3.2.1. Creating a job script file for MPI execution

mpiQulacs performs parallel processing using MPI.

(See Overview of MPI Parallel Computing in Python for a description of MPI.)

This is where you create the necessary script file to run MPI.

Create a script file referring to the following ~/example/job.sh .

~/example/job.sh

#!/usr/bin/env bash

# Usage:
#     mpirun -n <num_ranks> -npernode 1 job.sh <path/to/venv> <command> <command arguments>
#
# Example:
#     (python): mpirun -n 2 -npernode 1 job.sh ~/example/venv python ./sample.py
#     (pytest): mpirun -n 2 -npernode 1 job.sh ~/example/venv pytest

# Setting the Environment Variables required to use MPI in the quantum simulator system
export UCX_IB_MLX5_DEVX=no
export OMP_PROC_BIND=TRUE

# Number of threads used for OpenMP parallelization
# Due to the OpenMP multi-threading behavior, some libraries (such as scipy) can introduce small calculation errors, which can lead to inconsistent calculation results between processes during MPI execution. When using mpiQulacs, set it to 1.
export OMP_NUM_THREADS=1

# Number of threads used by mpiQulacs (if not set, the value of OMP_NUM_THREADS is used)
export QULACS_NUM_THREADS=48

#
source $1/bin/activate
shift

### Workaround for the glibc bug (https://bugzilla.redhat.com/show_bug.cgi?id=1722181)).
if [ -z "${LD_PRELOAD}" ]; then
    export LD_PRELOAD=/lib64/libgomp.so.1
else
    export LD_PRELOAD=/lib64/libgomp.so.1:$LD_PRELOAD
fi

#
LSIZE=${OMPI_COMM_WORLD_LOCAL_SIZE}
LRANK=${OMPI_COMM_WORLD_LOCAL_RANK}
COM=$1
shift

if [ $LSIZE -eq 1 ]; then
    numactl -m 0-3 -N 0-3 ${COM} "$@"
elif [ $LSIZE -eq 4 ]; then
    numactl -N ${LRANK} -m ${LRANK} ${COM} "$@"
else
    ${COM} "$@"
fi

After writing the code, execute the following command to grant execute permission.

$ chmod u+x ~/example/job.sh

3.2.2. Python script example of quantum program

Use mpiQulacs to write quantum programs in Python.

Basically, you can write a program in the same way as Qulacs . See Usage for details.

~/example/example.py

from qulacs import QuantumCircuit, QuantumState
from mpi4py import MPI  # import is required for MPI execution even if MPI classes are not used

seed=1234

# Get process rank
comm = MPI.COMM_WORLD
rank = comm.Get_rank()

# creation of quantum states
num_qubits = 10
state = QuantumState(num_qubits, use_multi_cpu=True)

# Define quantum circuits and update quantum states
circuit = QuantumCircuit(num_qubits)
circuit.add_H_gate(0)
for i in range(num_qubits - 1):
    circuit.add_CNOT_gate(i, i + 1)
circuit.update_quantum_state(state)

# Sampling values from quantum states
sampled_values = state.sampling(20, seed)

# Show results only for a process with rank 0 (node01)
if rank == 0:
    print(sampled_values)

Note

To run MPI, you must write an import statement “from mpi4py import MPI” before running the quantum circuit.

Note

The program (example.py) runs simultaneously on multiple compute nodes. Simply calling the print method will print the message to standard output multiple times for as many compute nodes. Therefore, the print statement is limited to node01. See Overview of MPI Parallel Computing in Python for details.

3.2.3. Creating a batch file

~/example/sim.job

#!/bin/bash

#SBATCH -p Batch # Specify Batch queue
#SBATCH -o test-%j # Output file name
#SBATCH -N 64 # Number of nodes allocated
#SBATCH -t 06:00:00 # Limit on the job execution time

mpirun -npernode 4 job.sh ~/example/qenv python example.py # The -n option is not required when executing with the number of nodes allocated above.
#   mpirun -n 1 -npernode 1 job.sh ~/example/qenv python example.py # To run on more than the allocated number of nodes, give the -n option; in this example, one node.

You create a file for job execution.
Fujitsu tells you how many nodes are available. If you have any questions, please contact the person in charge.
This example creates a Python program and runs it in parallel on two compute nodes.
-npernode specifies the number of processes per node.
Following job.sh is the path to the Python virtual environment, followed by the command and then the command arguments.
In the example above, ~/example/qenv specifies the path to the Python virtual environment created in Environmental Construction .
It then specifies the command python followed by the command argument example.py .
This causes python example.py to run in parallel on 64 compute nodes via job.sh.

Note

The optimal number of parallel circuits depends on the size and configuration of the circuit. Please consult with Fujitsu Researcher and adjust the parallel number accordingly.

Warning

The default job execution time limit is 6 hours.
Although it is not necessary to specify the job execution time limit, setting the execution time limit may cause the job to execute earlier.
(By default, a job will not run if there is scheduled maintenance within 6 hours, but it may run if the execution time is short.
In addition, jobs with a small number of nodes and a short job execution time are executed first if there are any unused nodes before the job is executed, even if there are jobs waiting.)
However, note that if the job does not end within the specified execution time, it will end even if the job is still running.

3.3. Job Submission

You run the batch file you created.

[username@loginvm-XXX example]$ sbatch sim.job

3.4. Check Output Results

When the job runs, the test-<JOBID> file is created and the results are output. If the JOBID is 10000, the results are output to the file test-10000 .

~/example/test-10000

[username@loginvm-XXX example]$ cat test-10000
[1023, 0, 1023, 1023, 0, 1023, 1023, 1023, 0, 1023, 1023, 0, 0, 1023, 0, 1023, 0, 0, 0, 1023]