PSparselib uses reverse communication to abstract away from particulars of the communication layer and the data structure: the fgmres routine returns control to the user for each matrix-vector product and preconditioning operation. However, inner products are still performed by hard MPI instructions in the fgmres routine.