Discuss the differences and similarities between MPI and PVM Discuss the benefits and?

Answer 1

MPI

(Message

P

assing

In

terface)

is

sp

eciØcation

for

message-passing

libraries

that

can

b

e

used

for

writing

p

ortable

parallel

programs.

What

do

es

MPI

do?

When

w

e

sp

eak

ab

out

parallel

programming

using

MPI,

w

e

imply

that:

≤

A

Øxed

set

of

pro

cesses

is

created

at

program

initialization,

one

pro

cess

is

created

p

er

pro

cessor

≤

Eac

h

pro

cess

kno

ws

its

p

ersonal

n

um

b

er

≤

Eac

h

pro

cess

kno

ws

n

um

b

er

of

all

pro

cesses

≤

Eac

h

pro

cess

can

comm

unicate

with

other

pro

cesses

≤

Pro

cess

can't

create

new

pro

cesses

(in

MPI{1),

the

group

of

pro

cesses

is

static

What

is

PVM?

PVM

(P

arallel

Virtual

Mac

hine)

is

a

soft

w

are

pac

k

age

that

allo

ws

a

heterogeneous

collection

of

w

orkstations

(host

p

o

ol)

to

function

as

a

single

high

p

erformance

parallel

virtual

mac

hine.

PVM,

through

its

virtual

mac

hine,

pro

vides

a

simple

y

et

useful

distributed

op

erating

system.

It

has

daemon

running

on

all

computers

making

up

the

virtual

mac

hine.

PVM

daemon

(p

vmd)

is

UNIX

pro

cess,

whic

h

o

v

ersees

the

op

eration

of

user

pro

cesses

within

a

PVM

application

and

co

ordinates

in

ter-mac

hine

PVM

comm

unications.

Suc

h

p

vmd

serv

es

as

a

message

router

and

con

troller.

One

p

vmd

runs

on

eac

h

host

of

a

virtual

mac

hine,

the

Ørst

p

vmd,

whic

h

is

started

b

y

hand,

is

designated

the

master,

while

the

others,

started

b

y

the

master,

are

called

sla

v

es.

It

means,

that

in

con

trast

to

MPI,

where

master

and

sla

v

es

start

sim

ultaneously

,

in

PVM

master

m

ust

b

e

started

on

our

lo

cal

mac

hine

and

then

it

automatically

starts

daemons

on

all

other

mac

hines.

In

PVM

only

the

master

can

start

new

sla

v

es

and

add

them

to

conØguration

7

or

delete

sla

v

e

hosts

from

the

mac

hine.

Eac

h

daemon

main

tains

a

table

of

conØguration

and

handles

information

relativ

e

to

our

parallel

virtual

mac

hine.

Pro

cesses

comm

unicate

with

eac

h

other

through

the

daemons:

they

talk

to

their

lo

cal

daemon

via

the

library

in

terface

routines,

and

lo

cal

daemon

then

sends/receiv

es

messages

to/from

remote

host

daemons.

General

idea

of

using

MPI

and

PVM

is

the

follo

wing:

The

user

writes

his

application

as

a

collection

of

co

op

erating

pro

cesses

(tasks),

that

can

b

e

p

er-

formed

indep

enden

tly

in

diÆeren

t

pro

cessors.

Pro

cesses

access

PVM/MPI

resources

through

a

library

of

standard

in

terface

routines.

These

routines

allo

w

the

initiation

and

termination

of

pro-

cesses

across

the

net

w

ork

as

w

ell

as

comm

unication

b

et

w

een

pro

cesses.

3.3

What

is

not

diÆeren

t?

Despite

their

diÆerences,

PVM

and

MPI

certainly

ha

v

e

features

in

common.

In

this

section

w

e

review

some

of

the

similarities.

3.3.1

P

ortabilit

y

Both

PVM

and

MPI

are

p

ortable;

the

sp

eciØcation

of

eac

h

is

mac

hine

indep

enden

t,

and

im-

plemen

tations

are

a

v

ailable

for

a

wide

v

ariet

y

of

mac

hines.

P

ortabilit

y

means,

that

source

co

de

written

for

one

arc

hitecture

can

b

e

copied

to

a

second

arc

hitecture,

compiled

and

executed

without

mo

diØcation.

3.3.2

MPMD

Both

MPI

and

PVM

p

ermit

diÆeren

t

pro

cesses

of

a

parallel

program

to

execute

diÆeren

t

exe-

cutable

binary

Øles

(This

w

ould

b

e

required

in

a

heterogeneous

implemen

tation,

in

an

y

case).

That

is,

b

oth

PVM

and

MPI

supp

ort

MPMD

programs

as

w

ell

as

SPMD

programs,

although

again

some

implemen

tation

ma

y

not

do

so

(MPICH,

LAM

{

supp

ort).

3.3.3

In

terop

erabilit

y

The

next

issue

is

in

terop

erabilit

y

{

the

abilit

y

of

diÆeren

t

implemen

tations

of

the

same

sp

eciØ-

cation

to

exc

hange

messages.

F

or

b

oth

PVM

and

MPI,

v

ersions

of

the

same

implemen

tation

(Oak

Ridge

PVM,

MPICH,

or

LAM)

are

in

terop

erable.

3.3.4

Heterogeneit

y

The

next

imp

ortan

t

p

oin

t

is

supp

ort

for

heterogeneit

y

.

When

w

e

wish

to

exploit

a

collection

of

net

w

ork

ed

computers,

w

e

ma

y

ha

v

e

to

con

tend

with

sev

eral

diÆeren

t

t

yp

es

of

heterogeneit

y

[GBD

+

94]:

≤

arc

hitecture

The

set

of

computers

a

v

ailable

can

include

a

wide

range

of

arc

hitecture

t

yp

es

suc

h

as

PC

class

mac

hines,

high-p

erformance

w

orkstations,

shared-memory

m

ultipro

cessors,

v

ector

sup

ercom-

puters,

and

ev

en

large

MPPs.

Eac

h

arc

hitecture

t

yp

e

has

its

o

wn

optimal

programming

metho

d.

Ev

en

when

the

arc

hitectures

are

only

serial

w

orkstations,

there

is

still

the

prob-

lem

of

incompatible

binary

formats

and

the

need

to

compile

a

parallel

task

on

eac

h

diÆeren

t

mac

hine.

8

≤

data

format

Data

formats

on

diÆeren

t

computers

are

often

incompatible.

This

incompatibilit

y

is

an

imp

or-

tan

t

p

oin

t

in

distributed

computing

b

ecause

data

sen

t

from

one

computer

ma

y

b

e

unreadable

on

the

receiving

computer.

Message

passing

pac

k

ages

dev

elop

ed

for

heterogeneous

en

viron-

men

ts

m

ust

mak

e

sure

all

the

computers

understand

the

exc

hanged

data;

they

m

ust

include

enough

information

in

the

message

to

enco

de

or

deco

de

it

for

an

y

other

computer.

≤

computational

sp

eed

Ev

en

if

the

set

of

computers

are

all

w

orkstations

with

the

same

data

format,

there

is

still

heterogeneit

y

due

to

diÆeren

t

computational

sp

eeds.

The

problem

of

computational

sp

eeds

can

b

e

v

ery

subtle.

The

programmer

m

ust

b

e

careful

that

one

w

orkstation

do

esn't

sit

idle

w

aiting

for

the

next

data

from

the

other

w

orkstation

b

efore

con

tin

uing.

≤

mac

hine

load

Our

cluster

can

b

e

comp

osed

of

a

set

of

iden

tical

w

orkstations.

But

since

net

w

ork

ed

com-

puters

can

ha

v

e

sev

eral

other

users

on

them

running

a

v

ariet

y

of

jobs,

the

mac

hine

load

can

v

ary

dramatically

.

The

result

is

that

the

eÆectiv

e

computational

p

o

w

er

across

iden

tical

w

orkstations

can

v

ary

b

y

an

order

of

magnitude.

≤

net

w

ork

load

Lik

e

mac

hine

load,

the

time

it

tak

es

to

send

a

message

o

v

er

the

net

w

ork

can

v

ary

dep

ending

on

the

net

w

ork

load

imp

osed

b

y

all

the

other

net

w

ork

users,

who

ma

y

not

ev

en

b

e

using

an

y

of

the

computers

in

v

olv

ed

in

our

computation.

This

sending

time

b

ecomes

imp

ortan

t

when

a

task

is

sitting

idle

w

aiting

for

a

message,

and

it

is

ev

en

more

imp

ortan

t

when

the

parallel

algorithm

is

sensitiv

e

to

message

arriv

al

time.

Th

us,

in

distributed

computing,

heterogeneit

y

can

app

ear

dynamically

in

ev

en

simple

setups.

Both

PVM

and

MPI

pro

vide

supp

ort

for

heterogeneit

y

.

As

for

MPI,

diÆeren

t

datat

yp

es

can

b

e

encapsulated

in

a

single

deriv

ed

t

yp

e,

thereb

y

allo

wing

comm

unication

of

heterogeneous

messages.

In

addition,

data

can

b

e

sen

t

from

one

arc

hitecture

to

another

with

data

con

v

ersion

in

heterogeneous

net

w

orks

(big-endian,

little-endian).

Although

MPI

sp

eciØcation

is

designed

to

encourage

heterogeneous

implemen

tation,

some

implemen

tations

of

MPI

ma

y

not

b

e

used

in

a

heterogeneous

en

vironmen

t.

Both

the

MPICH

and

LAM

are

implemen

tations

of

MPI,

whic

h

supp

ort

heterogeneous

en

vironmen

ts.

The

PVM

system

supp

orts

heterogeneit

y

in

terms

of

mac

hines,

net

w

orks,

and

applications.

With

regard

to

message

passing,

PVM

p

ermits

messages

con

taining

more

than

one

datat

yp

e

to

b

e

exc

hanged

b

et

w

een

mac

hines

ha

ving

diÆeren

t

data

represen

tations.

In

summary

,

b

oth

PVM

and

MPI

are

systems

designed

to

pro

vide

users

with

libraries

for

writing

p

ortable,

heterogeneous,

MPMD

programs.

3.4

DiÆerences

PVM

is

built

around

the

concept

of

a

virtual

mac

hine

whic

h

is

a

dynamic

collection

of

(p

oten-

tially

heterogeneous)

computational

resources

managed

as

a

single

parallel

computer.

The

virtual

mac

hine

concept

is

fundamen

tal

to

the

PVM

p

ersp

ectiv

e

and

pro

vides

the

basis

for

heterogeneit

y

,

p

ortabilit

y

,

and

encapsulation

of

function

that

constitute

PVM.

In

con

trast,

MPI

has

fo

cused

on

message-passing

and

explicitly

states

that

resource

managemen

t

and

the

concept

of

a

virtual

mac

hine

are

outside

the

scop

e

of

the

MPI

(1

and

2)

standard

[GKP96

].

9

3.4.1

Pro

cess

Con

trol

Pro

cess

con

trol

refers

to

the

abilit

y

to

start

and

stop

tasks,

to

Ønd

out

whic

h

tasks

are

running,

and

p

ossibly

where

they

are

running.

PVM

con

tains

all

of

these

capabilities

{

it

can

spa

wn/kill

tasks

dynamically

.

In

con

trast

MPI

{1

has

no

deØned

metho

d

to

start

new

task.

MPI{2

con

tains

functions

to

start

a

group

of

tasks

and

to

send

a

kill

signal

to

a

group

of

tasks

[NS02].

3.4.2

Resource

Con

trol

In

terms

of

resource

managemen

t,

PVM

is

inheren

tly

dynamic

in

nature.

Computing

resources

or

"hosts"

can

b

e

added

and

deleted

at

will,

either

from

a

system

"console"

or

ev

en

from

within

the

user's

application.

Allo

wing

applications

to

in

teract

with

and

manipulate

their

computing

en

vironmen

t

pro

vides

a

p

o

w

erful

paradigm

for

≤

load

balancing

|

when

w

e

w

an

t

to

reduce

idle

time

for

eac

h

mac

hine

in

v

olv

ed

in

computation

≤

task

migration

|

user

can

request

that

certain

tasks

execute

on

mac

hines

with

particular

data

formats,

arc

hitectures,

or

ev

en

on

an

explicitly

named

mac

hine

≤

fault

tolerance

Another

asp

ect

of

virtual

mac

hine

dynamics

relates

to

e±ciency

.

User

applications

can

exhibit

p

oten

tially

c

hanging

computational

needs

o

v

er

the

course

of

their

execution.

F

or

example,

con-

sider

a

t

ypical

application

whic

h

b

egins

and

ends

with

primarily

serial

computations,

but

con

tains

sev

eral

phases

of

hea

vy

parallel

computation.

PVM

pro

vides

∞exible

con

trol

o

v

er

the

amoun

t

of

computational

p

o

w

er

b

eing

utilized.

Additional

hosts

can

b

e

added

just

for

those

p

ortions

when

w

e

need

them.

MPI

lac

ks

suc

h

dynamics

and

is,

in

fact,

sp

eciØcally

designed

to

b

e

static

in

nature

to

impro

v

e

p

erformance.

Because

all

MPI

tasks

are

alw

a

ys

presen

t,

there

is

no

need

for

an

y

time-consuming

lo

okups

for

group

mem

b

ership.

Eac

h

task

already

kno

ws

ab

out

ev

ery

other

task,

and

all

com-

m

unications

can

b

e

made

without

the

explicit

need

for

a

sp

ecial

daemon.

Because

all

p

oten

tial

comm

unication

paths

are

kno

wn

at

startup,

messages

can

also,

where

p

ossible,

b

e

directly

routed

o

v

er

custom

task-to-task

c

hannels.

3.4.3

Virtual

T

op

ology

On

the

other

hand,

although

MPI

do

es

not

ha

v

e

a

concept

of

a

virtual

mac

hine,

MPI

do

es

pro

vide

a

higher

lev

el

of

abstraction

on

top

of

the

computing

resources

in

terms

of

the

message-

passing

top

ology

.

In

MPI

a

group

of

tasks

can

b

e

arranged

in

a

sp

eciØc

logical

in

terconnection

top

ology

[NS02,

F

or94]

.

A

virtual

top

ology

is

a

mec

hanism

for

naming

the

pro

cesses

in

a

group

in

a

w

a

y

that

Øts

the

comm

unication

pattern

b

etter.

The

main

aim

of

this

is

to

mak

e

subsequen

t

co

de

simpler.

It

ma

y

also

pro

vide

hin

ts

to

the

run-time

system

whic

h

allo

w

it

to

optimize

the

comm

unication

or

ev

en

hin

t

to

the

loader

ho

w

to

conØgure

the

pro

cesses.

F

or

example,

if

our

pro

cesses

will

comm

unicate

mainly

with

nearest

neigh

b

ours

after

the

fashion

of

a

t

w

o-dimensional

grid

(see

Figure

3),

w

e

could

create

a

virtual

top

ology

to

re∞ect

this

fact.

What

w

e

gain

from

this

creation

is

access

to

con

v

enien

t

routines

whic

h,

for

example,

compute

the

rank

of

an

y

pro

cess

giv

en

its

co

ordinates

in

the

grid,

taking

prop

er

accoun

t

of

b

oundary

conditions.

In

particular,

there

are

routines

to

compute

the

ranks

of

our

nearest

neigh

b

ours.

The

rank

can

then

b

e

used

as

an

argumen

t

to

message{passing

op

erations.

10