How to create MPI performance models using Hockney model parameters? - time-complexity

Question

I understand that the parameters α and β can be used in the Hockney model to represent latency and bandwidth in peer to peer communications with m representing the message size. For example:
T(m) = α + β · m
I have been trying to model some OpenMPI algorithms using this technique and can't figure out this following algorithm for MPI_Scatter:
int
ompi_coll_base_scatter_intra_linear_nb(const void *sbuf, int scount,
struct ompi_datatype_t *sdtype,
void *rbuf, int rcount,
struct ompi_datatype_t *rdtype,
int root,
struct ompi_communicator_t *comm,
mca_coll_base_module_t *module,
int max_reqs)
{
int i, rank, size, err, line, nreqs;
ptrdiff_t incr;
char *ptmp;
ompi_request_t **reqs = NULL, **preq;
rank = ompi_comm_rank(comm);
size = ompi_comm_size(comm);
/* If not root, receive data. */
if (rank != root) {
err = MCA_PML_CALL(recv(rbuf, rcount, rdtype, root,
MCA_COLL_BASE_TAG_SCATTER,
comm, MPI_STATUS_IGNORE));
if (MPI_SUCCESS != err) {
line = __LINE__; goto err_hndl;
}
return MPI_SUCCESS;
}
if (max_reqs <= 1) {
max_reqs = 0;
nreqs = size - 1; /* no send for myself */
} else {
/* We use blocking MPI_Send (which does not need a request)
* every max_reqs send operation (which is size/max_reqs at most),
* therefore no need to allocate requests for these sends. */
nreqs = size - (size / max_reqs);
}
reqs = ompi_coll_base_comm_get_reqs(module->base_data, nreqs);
if (NULL == reqs) {
err = OMPI_ERR_OUT_OF_RESOURCE;
line = __LINE__; goto err_hndl;
}
err = ompi_datatype_type_extent(sdtype, &incr);
if (OMPI_SUCCESS != err) {
line = __LINE__; goto err_hndl;
}
incr *= scount;
/* I am the root, loop sending data. */
for (i = 0, ptmp = (char *)sbuf, preq = reqs; i < size; ++i, ptmp += incr) {
/* simple optimization */
if (i == rank) {
if (MPI_IN_PLACE != rbuf) {
err = ompi_datatype_sndrcv(ptmp, scount, sdtype, rbuf, rcount,
rdtype);
}
} else {
if (!max_reqs || (i % max_reqs)) {
err = MCA_PML_CALL(isend(ptmp, scount, sdtype, i,
MCA_COLL_BASE_TAG_SCATTER,
MCA_PML_BASE_SEND_STANDARD,
comm, preq++));
} else {
err = MCA_PML_CALL(send(ptmp, scount, sdtype, i,
MCA_COLL_BASE_TAG_SCATTER,
MCA_PML_BASE_SEND_STANDARD,
comm));
}
}
if (MPI_SUCCESS != err) {
line = __LINE__; goto err_hndl;
}
}
err = ompi_request_wait_all(preq - reqs, reqs, MPI_STATUSES_IGNORE);
if (MPI_SUCCESS != err) {
line = __LINE__; goto err_hndl;
}
return MPI_SUCCESS;
err_hndl:
if (NULL != reqs) {
/* find a real error code */
if (MPI_ERR_IN_STATUS == err) {
for (i = 0; i < nreqs; i++) {
if (MPI_REQUEST_NULL == reqs[i]) continue;
if (MPI_ERR_PENDING == reqs[i]->req_status.MPI_ERROR) continue;
if (reqs[i]->req_status.MPI_ERROR != MPI_SUCCESS) {
err = reqs[i]->req_status.MPI_ERROR;
break;
}
}
}
ompi_coll_base_free_reqs(reqs, nreqs);
}
OPAL_OUTPUT((ompi_coll_base_framework.framework_output,
"%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank));
(void)line; /* silence compiler warning */
return err;
}
So far I understand that from looking at the code that the model should be
T(NP, m) = (NP − 1) · (α + m · β).
With NP being the number of processes (As Scatter distributes using all processes apart from the root).
This does not account for the use of non-blocking sends that are send using MPI_Isend. (on the condition found in the code snippet) I am unsure of how to account for both the non-blocking and blocking sends using simply the Hockney Model.
Any help would be very much appreciated as non of the papers that I have read on the subject seem to explain the process well.

Answer 1

First of all, the source file mentions that this implementation is probably only for small numbers of processes: for larger numbers you probably want to do something treewise. Next, the max_reqs parameter controls how many isend calls you do before a blocking send call. So the running time of this algorithm would be equal to the number of times you do a blocking send. In an ideal world. In practice, non-blocking sends still have to be serialized out.
My best guess is that this algorithm can handle the case where there are multiple network cards or multiple ports per network card. If you can send 4 messages at a time (physically!) then this code sets up 3 non-blocking & 1 blocking send, and when the blocking send has gone through, your network ports are ready for the next batch of messages.