
% Template for a simple journal or conference style article

\documentstyle[12pt, fullpage]{article}
\input epsf
\newcommand{\cepsffig}[1]{\begin{center}{\mbox{\epsffile{#1}}}\end{center}}
\newcommand{\noi}{\noindent}
\begin{document}
\pagestyle{plain}
\title{\bf Implementation of Farming Under PVM}
\author {Sameer S. Shende\\ Department of Computer and Information Science \\
University of Oregon}

\maketitle
\vspace*{.3in}

\begin{abstract} 
Dynamic load balancing can improve the performance of a parallel application 
significantly. This is needed in the PVM environment because different nodes in the network have different processing capacities, networks have different
data carrying capacities,
in a timesharing environment, other users' processes can slow down
a particular task in a parallel 
application. Also a large number of data parallel algorithms require unequal
processing times for each quantum of data given to the processor. 

This paper explains the design and implementation of one such load balancing 
scheme under PVM. PVM is fast becoming a popular parallel programming environment both on a network of workstations, multiprocessors and other parallel and
vector platforms. 

{\bf Keywords :} dynamic load balancing, PVM, farming, heterogeneous computing, parallel and distributed processing.
\end{abstract}


\section{Introduction}

This paper focuses on an implementation of process farms under the parallel virtual machine environment. PVM uses a cluster of workstations to form a heterogenous network based MIMD computing environment. The problems associated with 
such an environment to form a parallel application are that, typically point 
to point communication constructs are provided to distribute data in a data parallel approach. Most parallel applications need some form of load balancing 
technique to equitably distribute work packets and keep all processors busy.
This is usually a static off line process in which the application
is evaluated for a particular problem size and a log is examined to determine the
placement of 
tasks on the nodes by examining the computation to communication time ratios for 
the tasks. The application has to explicitly distribute the data so as to have equal load on all processors.

\section {Load Balancing}
\subsection{Need for Dynamic Load Balancing under PVM}

In PVM, the user tasks which comprise a parallel application are distributed
over the nodes in a network. The nodes could be parallel/vector machines or 
workstations. Thus, a heterogenous computing environment is used to solve the
problem. To extract the maximum performance from the parallel application, it is
important to keep the idle time of tasks to a barest minimum. To ensure that
each process gets an optimum load to be processed, a dynamic load balancing 
scheme is desirable. The relevant issues that the application developer has to
take into consideration are :
\begin{enumerate}
\item All nodes in the network do not have identical computation capacities.
Even among workstation class machines, a variety of processor architectures,
network interfaces, and I/O subsystems can be encountered. Some machines 
may have processing rates which are an order of magnitude higher than some 
other machines in the network.
\item A parallel application running over a wide area network will encompass
a range of networks which have different transfer rates and latencies. For
e.g., ATM supports 622 Mb/s and 100 Mb/s Ethernet LANs have 10 Mb/s rates, WANs have 64 Kb/s (X.25), Token ring
LANs have 16 Mb/s peak transfer rates. This results in different message transfer rates
between tasks in the parallel application.

\item There are background jobs in most time sharing multiuser systems (typical
unix workstations) so the CPU time that a task under PVM can get may 
vary from one run to another depending on the usage of the workstation.

\item A large number of algorithms are not structured nicely, and on breaking
the data to be processed into work packets, each work packet may require a 
different processing time (even if we keep the above factors constant). This
unequal processing time requirement cannot be always determined apriori and can  be 
difficult to account for while designing the application.
\end{enumerate}

\subsection{Dynamic Load Balancing}

Dynamic load balancing in an application can address these concerns. Dynamic load balancing can
be implemented in a heterogenous environment either at the operating system 
level or at a process level. 
\begin{enumerate}
\item At the {\em operating system level}, the emphasis is to
keep all the {\em processors} busy all the time. This is accomplished by process
migration from a heavily loaded node in the network to a lightly loaded one.
This raises issues about doing this effectively at runtime on a heterogenous
collection of nodes. While it may be easy to incorporate such a scheme for a
``state-less'' server process by killing one instance and starting another 
instance on the other node (different binaries being used for dissimilar 
processors and operating systems), it can be difficult to migrate a process
at run-time when the the state of a process needs to be maintained. The process
migration approach can be seen in systems such as Condor.

\item At the {\em process level}, it is important to keep all the {\em user tasks}
busy all the time. If the data to be processed is divided into independent work packets, then the load balancing scheme at the process level can be implemented
 by giving more work packets to 
the tasks on faster processors and less work packets to the slower ones. This way, the
data can be distributed on the basis of the processing capacity of a particular
task. This approach takes into account the differences in processing times for
different data packets. The aim of the paper is to implement a dynamic load
balancing scheme under PVM using this approach by constructing {\em process farms}. 
\end{enumerate}

\section {Farming}
\subsection{Farming - Definition of terms}
Under the farming model, the parallel application is divided into one or more process farms. A process farm comprises one farmer task and one or more worker classes. A 
worker class has one or more worker tasks which are grouped together because
they perform similar functions. A worker (task) can be associated with one and
only one farmer task (and in turn one worker class). In a {\em hierarchical farm}, a worker task can in turn be a farmer to some other set of worker classes and worker tasks. Again, the worker can be a farmer to more than one farm. This could be useful in subdivision of a work packet into sub parts for
processing by another worker class hierarchy. Also, the farm implementation should be such that a worker task should be free to leave a process farm (dynamic process
migration to another farm) and join any other farm if needed. 

To implement an algorithm using the farming paradigm, the data parallel application would need to create one or more farms. The farmer task would need to take
the data to be processed and split it up into a number of work packets which 
would be  independent of each other and could be processed by a worker task. It
would then give out the work packets to a worker class without knowing which
particular worker task would process it. The operating environment would decide
the appropriate worker to forward the work packet to. After processing the work
packet, the worker would create a reply packet and address it to the farmer task.
The farmer task would collect 
all the work packets from a particular worker class and reassemble these 
appropriately. More than one farm and more than one worker class could be active
at a particular instant creating, sending or receiving packets. The farmer could
terminate a farm after it is no longer needed, at which stage the worker tasks 
would automatically be expelled from the farm and informed accordingly. However,
before this happens, a worker should also be able to leave the farm and stop
receiving work packets from the worker class.   

\subsection {Farming - Classification}
To focus our attention on the operating environment (or a farm daemon task), it should be able to manage
the creation of the farms and operations such as termination of a farm, leaving
a farm (by a worker task), and routing of work packets to the worker. There are
ways in which the packet distribution to a worker can occur. Farms are classifed into the following categories.
\begin{enumerate}
\item {\em Static Farm}

In this case, the farm daemon could distribute all the work packets, that it 
has at any instant, equally to all the worker tasks. The advantage of this 
scheme is that the worker task 
would not need to request the farm daemon for a work packet but would retrieve
it locally from its queue of messages. The disadvantage of this scheme is that at
run-time if another worker task joins a worker class after all work packets 
have been distributed, then a redistribution of work packets would not take
place and the new worker task could be idle. Also, this approach does not 
take into account the different processing requirements for each work packet.
So, in this approach the farm is static in the sense that all worker tasks can
process number of packets that is independent of their processing capacities. So this does not provide for dynamic load balancing.

\item {\em Dynamic Farm}

In this case, the farm daemon would distribute the work packets to workers
only when the worker would request for a work packet. Thus, different workers
could receive different number of work packets depending upon their respective
processing capacities. Also, this scheme is suitable to a farm which allows 
migration of a worker task from one farm to another. When this happens, the new worker task could receive a work packet as all the work packets are contained in the central farm daemon task and
it can be integrated into the farm easily. The advantage of the dynamic farm
is that it can support worker task migration (between farms) efficiently, it can
distribute work packets depending upon the processing capacities of the worker
tasks and keep all the workers busy most of the times and so it implements a 
dynamic load balancing scheme. The disadvantage of a dynamic farm is that it 
requires the work packets for a farm to be stored at a central location which
requires a large storage to be allocated at the farm daemon. Also, the worker
tasks need to request for each work packet explicitly, so there is a greater latency 
in getting a work packet as compared to the static farm case where all
the work packets are locally stored and can be retrieved without any additional
messages to be sent to the central server (farm daemon). 

Different distributed data management schemes with respect to locks are 
discussed in \cite{th:tom} . These schemes are analogous to managing the work
packets in the case of process farms.
\begin{figure}
\input{f1.pic}
\caption{Representation of Farming}
\end{figure}
\end{enumerate} 

\subsection{Goals of the farming implementation}

The aim of this paper was to implement a hierarchical farm which supported
worker task migration from one farm to another and to implement a dynamic load
balancing scheme by using a dynamic farm model described above. It needed to 
support multiple farms at each level with multiple worker classes at each farm
level. Furthermore, it was necessary to formulate an Application Programmers
Interface (API) for the farm which was independent of the nature of the farm
(static or dynamic farm implementation) and would work with the PVM environment.
It was important to design it in such a way that there would be {\em ``busy-waiting'' condition}
at any stage in the components. Any polling for an event to happen would only
degrade the system performance.
Also, it was a goal to use the farming constructs as a layer above the PVM
software and to use the interface of PVM to build the farming model. This was
necessary to support the farm implementation across different versions of 
PVM (starting from 3.2.0) so that it would give backward source code compatibilty (as long as PVM maintained backward compatibility).

\subsection{Previous work in this area - Comparison}
Work related to the implementation of process farms has been done on transputer
based environments which are typically distributed memory MIMD message passing
environments. The work done at EPCC \cite{pul:tf} on process farming runs on top
of their CHIMP environment which also works as a heterogenous computing platform
using workstation clusters. Their implementation of farming however 
consists of a source process and a sink process (instead of a single farmer 
task). 
Implementations of static farms with queue lengths (of
the local worker task's queue of work packets) specified by worker tasks have been done at C-DAC, India 
\cite{corec} and TUL, India \cite{sss} but have concentrated on a homogeneous
environment. The advantage of \cite{sss} over this paper's implementation is
that the farm related information is maintained in a distributed manner, 
so when multiple farms are implemented it would have no bottlenecks at the 
central server site. This may happen in our case if all farms simultaneously
dispatch  work packets to the workers. Work on a single process farm (non 
hierarchical) based application implementation also appears in 3L\cite{3lc} in
Inmos, U.K. 

The author is not aware of any process farm implementation under the PVM 
heterogenous computing environment to implement dynamic load balancing. This work aims at bridging this
gap by providing a dynamic load balancing construct to expand the PVM API.

\section{Implementation and Design Issues}
The implementation of process farms consisted of
\begin{itemize}
\item Defining the API for farming.
\item Designing the software components viz., the farm daemon (pvmfarmd - 
the server task) and the library of the API (libfarm.a - the client) which 
would be used by both the farmer and the worker tasks.
\item Defining the protocol for interprocess communication between the
farmer task and the farm daemon task, and between the worker task and the
farm daemon task so as to avoid any polling by any of the three components.
\end{itemize}
\subsection{API definition}
{\em At the farmer end:}
\begin{enumerate}
\item
{\bf int 	pvm\_farm\_init (char *farm\_name); }

The routine pvm\_farm\_init() creates a process farm with a name specified by the 
argument {\em farm\_name}. In an application, the farm name is unique (i.e., the 
user should not invoke this routine twice with the same name). Multiple farms 
can be used with a separate farm names. This name is used by the worker task to enroll in a farm.
	Returns OK or ERROR or PvmFarmNamePresent.
\begin{verbatim}
	e.g.,
	pvm_farm_init("Farmer1");
\end{verbatim}
\item
{\bf int pvm\_get\_worker\_class\_id (char *farm\_name, char *worker\_class\_name); }

The routine pvm\_get\_worker\_class\_id() returns the identifier that 
represents a worker class named {\em worker\_class\_name} associated 
with the farm named {\em farm\_name}.  This is used by the farmer to uniquely
identify a particular worker class.
Returns ERROR on error or a positive integer {\em id}. This identifier
is used by subsequent farmer calls to send data packets to this worker class or
 to receive replies from the worker class. This call blocks till the worker class is created by a worker task.
\begin{samepage}
\begin{verbatim}
	e.g.,
	id = pvm_get_worker_class_id("Farmer1", "WC1");
	if (id == ERROR)
	{
		printf("Error in getting worker class id\n");
		pvm_exit();
		exit(1);
	}
\end{verbatim}
\end{samepage}
\item
{\bf int	pvm\_send\_work\_packet(int id); 	 }

The routine pvm\_send\_work\_packet() sends the message to one worker task associated with the 
worker class whose identifier is specified by {\em id}. The data should be packed in
a buffer (the active send buffer) created by pvm\_initsend() or pvm\_mkbuf();
PvmDataDefault is used to create the buffer so that XDR encoding can be done
for communication between dissimilar hosts (little endian and big endian hosts).
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	int id, data, bufid; 
	id = pvm_get_worker_class_id("Farmer1","WC1");
	bufid = pvm_mkbuf(PvmDataDefault); /* pvm call creates buffer */
	pvm_setsbuf(bufid); /* sets this as the active send buffer
		This is also the default action taken */
	data = 500; /* gets data ready for packing in the buffer */
	pvm_pkint(&data, 1, 1); /* packs one int data item, stride = 1 */
	/* calls to pack the data */
	pvm_send_work_packet(id, 34);
	pvm_freebuf(bufid); /* frees the buffer */
\end{verbatim}

\item 
{\bf int	bufid = pvm\_recv\_reply\_packet(id); }

This blocking receive reply packet routine will wait until has arrived from the
worker class specified by the identifier {\em id}.  This call 
places the message in a new active receive buffer that is created. The previous
active receive buffer is cleared unless it has been saved explicitly with a pvm\_setrbuf() call.
\begin{verbatim}
	e.g.,	
	pvm_recv_reply_packet(id);
\end{verbatim}

\item 
{\bf int	pvm\_farm\_terminate(char *farm\_name); }
	
This routine shuts down a farm and frees the worker tasks in the worker classes associated with the farm whose name is specified by {\em farm\_name}.
The worker tasks waiting on a receive are sent a message that the farm has terminated. After executing this call, a new farm with the same name may be created.
Returns OK or ERROR or PvmNoSuchFarm or PvmNotFarmOwner.
\begin{verbatim}
	e.g.,
	pvm_farm_terminate("Farmer1");
\end{verbatim}

\item 
{\bf int	pvm\_start\_farmd(void); }

This routine starts the pvmfarmd daemon which is responsible for farming 
related calls. This daemon can be started manually from the pvm shell or by
using this call. This should be called only once in the entire application.
Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_start_farmd(); /* starts the daemon.*/
\end{verbatim}
\item 
{\bf  int	pvm\_stop\_farmd(void); }

This routine sends a message to the pvmfarmd to terminate itself. This should
also be called by only one task in the application. This should be done by the
farmer task after it has terminated the farm.
Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_stops_farmd(); /* stops the daemon.*/
\end{verbatim}

	
{\em At the worker end:}
\item 
{\bf int	pvm\_init\_worker\_class(char *farm\_name, char *worker\_class\_name); }

The pvm\_init\_worker\_class() routine creates (if it already doesn't exist) a worker class
named {\em worker\_class\_name} and enrolls the calling task in the worker class.
This worker class is associated with the farmer task specified by {\em farm\_name}. This call blocks till a farm is created (in case the farmer has not created the farm).
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_init_worker_class("Farmer1", "WC1");
\end{verbatim}
\item
{\bf int	bufid = pvm\_recv\_work\_packet(void);  }

The pvm\_recv\_work\_packet()  routine waits until a message arrives from the farmer task. A 
worker task can be associated with at most one farmer (and a worker class)  at
a time. The previous active receive buffer is cleared unless it has been 
saved with a pvm\_setrbuf() call. This call returns bufid or PvmFarmTerminated
if the farmer has executed a farm\_terminate() call. When PvmFarmHasTerminated is returned, the worker task is
free to do any other activity not associated with the current farm.
\begin{verbatim}
	e.g.,
	buffer_id = pvm_recv_reply_packet();
	/* All unpacking routines */
\end{verbatim}

\item 
{\bf int	pvm\_send\_reply\_packet(void); }

	This routine sends a reply packet which has been packed into the 
active send buffer, to the farmer task.
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	/* initialise the buffer for sending */
	int bufid,data;
	bufid = pvm_mkbuf(PvmDataDefault); /* makes a new buffer */
	pvm_setsbuf(bufid); /* done by default - sets this 
		as the active send buffer used by pvm_send_reply_packet() */
	data = 5; /* some reply packet */
	pvm_pkint(&data, 1, 1); /* packs the integer into the buffer */
	pvm_send_reply_packet(); /* sends the buffer to the farmer */
	pvm_freebuf(bufid);
\end{verbatim}
\item 
{\bf int	pvm\_leave\_farm(void); }

	This routine is called by the worker task to disassociate itself from
the farm in which it is currently enrolled. This call removes the task related
information of this task from the worker class information repository maintained
by the central pvmfarmd daemon (farm daemon). After this call is executed, the
calling worker task no longer receives work packets from the farmer. The worker
can enroll in any farm subsequently, by calling the pvm\_init\_worker\_class()
routine. This allows for the worker process to migrate from one farm to another
dynamically at runtime.  
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_leave_farm(); /* leaves the farm */
\end{verbatim}
	
\end{enumerate}

\subsection{Design issues}
The process farms were implemented following the design of the farm daemon
and the definition of the protocol for interprocess communication as described 
in Appendix A of the paper. 
\begin{figure}
\input{f2.pic}
\caption{pvmfarmd - Internals}
\end{figure}
The farm daemon (pvmfarmd) was implemented using lists. The entire code was
written in C++ making list as a class with member functions like append, delete,
pop, search by name, search by value, etc. The farm daemon had a list of all
farms (each farm was identified by a unique name and farmer task id). Each item
in the farm list had a list of worker classes that were associated with it. 
Each item in the list of worker classes had a unique worker class identifier,
the list of work packets awaiting distribution and a list of worker tasks 
enrolled in the worker class. Apart from these lists, there was a global list
of all pending requests (e.g., the farmer requested for a worker class id and
the worker class was not created, so it had to wait for a worker class to be
created, etc.). When a work packet arrived at the farm daemon, it had to be 
forwarded to a single worker task. To search for the worker class to which the packet
had to be addressed, one approach was to search each item in the farmer list
for a matching farm name, then to get to the worker class list and then to 
search for the matching worker class id item and then to get a worker task that
had no pending work packets. This was inefficient
when a large number of work packets were sent to the farm daemon. Thus, to 
improve performance, a cache of worker class ids was maintained. The entries
in this cache array pointed to the worker class item in the list of worker 
classes. So, in addition to the lists, the array of worker class items (indexed
by the unique worker class id) was maintained. This structure is illustrated in figure 2.

\section{Results}

\begin{figure}
\cepsffig{back/comp0-100.ps}
\caption{Matrix multiplication for small sizes of matrices}
\end{figure}
\begin{figure}

\cepsffig{back/comp0-500.ps}
\caption{Matrix multiplication for large sizes of matrices}
\end{figure}
To compare the performance of an application using the farming API under PVM,
a matrix multiplication application was written. It might have been easy to show the
performance gain by using farming in an application where each work packet 
required a different number of instructions for processing it. So, instead matrix
multiplication application was chosen because each row in the product matrix 
required $n^2$ operations (uniformly). To get the timings of the best possible
case without load balancing, the two matrices (A and B in C = A x B) were kept
on all the nodes. The application was divided into 6 worker tasks on 6 nodes in
the network and one master task which gave out the row number to the workers.
The worker on getting the row number would calculate the corresponding row of
the C matrix and return it to the master. Rows to be calculated were interleaved
among the workers in the case where farming was not used. This caused each
worker task to process a fixed amount of rows only. So, the application speedup
was limited by the slowest worker in the application. 

Farming was used on an identical setup and the farmer task sent out the row 
numbers to a worker class which contained the worker tasks. The farm 
daemon gave out the rows to the worker tasks depending on their requests. The
network used was a token ring network and 6 SUN4 nodes running SunOS 4.1.3 were
used. The Sun workstations were of different types ranging from Sparc station 1
to a four processor Sun 670/MP multiprocessor. Also, background jobs were 
running on the machines. 

The results obtained for different matrix sizes indicated that when the processing requirement was small (overall time was less than 5 secs), the application without
load balancing performed well (in the absence of the farming overhead of forwarding work packets). However, as the size of the problem grew, the application 
with dynamic load balancing outperformed the other sometimes by an order of
magnitude. This was because, when the application was using the farming construct, all the tasks were busy all the time. This led to the optimum processor
utilization and produced the results in an efficient way.
\section{Future Work}

\subsection {Fault Tolerance}
A feature of fault tolerance can be incorporated in the farm daemon. This can
be used when a node having a worker task goes down. PVM provides constructs
to notify a task (if it has so indicated) when a task exits PVM or the node
on which it is running halts due to some error \cite{man:pvm} . This feature could be useful
in the case where the farmer task needs ``at least one'' reply  to come from
the worker task for every work packet that is sent out. It would be possible 
to incorporate fault tolerance in the farm daemon because it could maintain a list of work
packets which were currently being processed (it needn't keep a list of all 
work packets, as when a worker task requests for a new work packet, the earlier
one can be removed from the list). When PVM would notify the farm daemon of a worker
task exiting or a node malfunctioning, then the farm daemon could either fork the
task on another node or using the existing nodes could resend the work packet
to the other worker tasks after removing the entry about the dead task(s) from 
its lists. 
\subsection{Additional farm daemons}
If an application uses multiple farms simultaneously, then a single farm 
daemon could become a bottleneck. In that case, a farm daemon
for every farm might give better results. This would need to be verified since
the additional tasks could have an overhead in the sense that there would be 
more tasks on a fixed number of nodes in the network. Also,
since farms are based on a named service, an additional name server task would
need to be introduced (if two tasks were to  simultaneously request for creating a farm with the same
name, one would get an error and the other would be allowed to continue). It
would need to be studied whether adding extra farm daemons would improve the
performance and how often would a large number of farms would be used in an application. This could be used for future work in this area. 

\section{Conclusion}
Farming was implemented under PVM and it was observed that dynamic load balancing could sometimes improve the performance of an application under PVM by an 
order of magnitude. 

However, to use the construct of farming, an application would need to be
rewritten to take advantage of this performance improvement. This may not be 
always feasible for existing applications but new applications could take 
advantage of this approach.

\begin{thebibliography}{99}
\bibitem{th:tom} Anderson T. E., Lazowska E. D., and Levy H. M., ``The Performance
Implications of Thread Management Alternatives for Shared Memory Multiprocessors
'', Department of Computer Science, University of Washington, Technical Report
88-09-04, Sep. 1988.
\bibitem{pul:tf} ``PUL-TF Prototype User Guide'', {\em EPCC User Guide}, 1992.
\bibitem{sss} Shende, S., and B. Ananda Krishna , ``Simulation of Concurrent
Runtime Environment of PARAM on UNIX'', {\em High Performance Computing, 
Proceedings of the Southern Regional Convention, Computer Society of India}, July 1994.
\bibitem{corec} ``CORE Concurrent Runtime Environment for C Users Manual'', C-DAC 
, India, 1991.
\bibitem{paras} ``PARAS Overview Manual'', C-DAC, India, 1991.
\bibitem{3lc} ``3LC Users Guide'', Inmos U.K., 1990.

\bibitem{man:pvm} Beguelin A., Dongarra J. J., Geist G. A., Manchek R., Sundaram 
V. S., ``A users' guide to PVM parallel virtual machine'', {\em Technical 
Report ORNL/TM-11826, Oak Ridge National Laboratory}, July 1991.

\end{thebibliography}

\appendix 
\clearpage
\section{Appendix- Algorithms for implementing farming under pvm}
\begin{enumerate}

\subsection  {\bf int 	pvm\_farm\_init (char *farm\_name); }
 
\subsubsection{Client side}
\begin{enumerate}
\item Establish a connection with the server by seeing the name of the server in the list of tasks and extracting its tid. Also, find the farmer process name (opt) and calling process task id.
\item The server task id should be stored in a global variable and if it 
contains a server tid, then the search for the server is not required (say a
a server tid, then the search for the server is not required (say a
previous pvm\_farm\_init() call has already set it).
\item Send a message to the server of the type DEF\_MSGTAG (1) which contains
	farm\_name, opcode INIT\_FARM
\item Receive a message from the server containing the result of the operation.
\item Return the value read from the server. This value can be OK or PvmFarmName\-Present or ERROR.
\end{enumerate}

\subsubsection*{Server side }

\begin{enumerate}
\item Wait for any request from any task (tid = -1, msgtag = -1) (wildcard recv).
\item On getting the message, extract the opcode from the buffer
\item If opcode is INIT\_FARM, then get tid and  farm name from the client.
\item Search the list of farm names for a farm of the name received from the client.
\item If such a farm does not exist, append the farm name (and farmer tid) to 
the list of farm names. This creates a new farm item in the list of farms. Return OK the client if it is created successfully or ERROR if an error occurs.
\item If the farm name already exists, then send a message to the client containing the return value FARM\_NAME\_PRESENT.
\item Look at the list of pending requests and service these.
\item Go back and wait for other messages
\end{enumerate}


\subsection {\bf int	pvm\_get\_worker\_class\_id (char 	*farm\_name, char	*worker\_class\_name); }
\subsubsection*{Client side }
\begin{enumerate}
\item Take the global variable which identifies the server task (its tid) and
send a message to the server.
\item The message should contain 
opcode GET\_WORKER\_CLASS\_ID, farm name and the worker class name.
\item Receive reply from the server 
\item Unpack the message and extract the return value.
\item The return value (if nonnegative) will contain the worker class id 
(wcid) or the error message PvmFarmNotCreated if the farm is not created by 
the farmer prior to invoking this call.
\item return the integer received from the server.
\end{enumerate}

\subsubsection*{Server side }
\begin{enumerate}
\item if opcode is GET\_WORKER\_CLASS\_ID
\item extract (unpack) farm name and worker class name.
\item Search for the farm item by farm name.
\item If item is not found send a reply containing FARM\_NOT\_CREATED error message.
\item Get the farm item (FItem), using this get access to the worker class list (WCList).
\item Search for worker class item by the worker class name.
\item If not found then, put the request on the list of pending requests and  look for other messages (while the farmer task blocks for a reply).
\item If item is found, then see the WCItem and get the worker class id (wcid).
\item Send the reply containing the wcid.
\end{enumerate}


\subsection 
{\bf int	pvm\_init\_worker\_class(char *farm\_name, char *worker\_class\_name); }

\subsubsection*{Client side }
\begin{enumerate}
\item Get the tid of the worker (pvm\_mytid )
\item Search for the server task by looking at the list of tasks and get the tid
of the server. If there's no server, then return ERROR.  (check the global variable containing the server task id).
\item Send a message to the server. It should contain
opcode INIT\_WORKER\_CLASS, farm name, and worker class name.
\item Wait for the reply  from the server. Get the status ERROR, or wcid and store it in a global variable to be used later.
\item Get and store the farmer name and the farmer task id  in a global variable which could be used later.

\item return OK
\end{enumerate}


\subsubsection*{Server side }
\begin{enumerate}
\item If opcode is INIT\_WORKER\_CLASS
\item Get the farm name, worker class name and worker task id.
\item Search the list of the farmers using this farm name.
\item If an item is not found then add the request to the list of pending requests
and service some other message (client remains blocked)
\item If the farmer item is found then, get the farm item (FItem) and look for
the list of worker classes (WCList).
\item Search the list of the worker classes using the worker class name (search by name).
\item If the item is not found then Append an item in the list of worker classes and get the worker class id.
\item If the worker class id exceeds the hard limit, return an ERROR.
\item Make an entry in the Cache of worker class ids regarding
the worker class item and the worker class id.
\item Append a worker task item in the list (using the wlist ) and send the wcid  to the client task.
\item If the worker class item is found then get the WCItem which contains the worker class id.
\item Append the worker task to this list 
\item Send the wcid to the worker task (client)
\item Service other messages
\end{enumerate}

\subsection*{Algorithm for ProcessPending() at the Server side.}
\begin{enumerate}
\item Check the number of pending items in the global list of the pending requests related to farm setup.
\item Pop one by one all the requests from the list.
\item If the opcode in the pending request is GET\_WORKER\_CLASS\_ID, then get the farm name, the worker class name, and the task id of the farmer. 
\item Execute ProcessGetWorkerClassId() with these parameters. This function is the same as above (when a pvm\_get\_worker\_class\_id() call is executed).
\item If the opcode is INIT\_WORKER\_CLASS, then get the farm name, the worker class name and the task id of the worker task.
\item Execute ProcessInitWorkerClass() with these parameters. This function is illustrated above. 
\item Note that all the algorithms of farm set up functions are executed as functions with parameters. This helps so that either the main server task can call these when the client requests for the service, or the ProcessPending() can call these.
\end{enumerate}

\subsection {\bf int	pvm\_send\_work\_packet(int id); 	}

\subsubsection*{Client side }
\begin{enumerate}
\item Get the currently used buffer id and save it. Create another buffer to
use for sending a message to the server task.
\item The message should contain, the opcode 
	SEND\_WORK\_PACKET and the the worker class id of the following message.\item After sending the message, make the user message buffer the current active
send buffer and send the user specified message.
\item Return OK.
\end{enumerate}

\subsubsection*{Server side }
\begin{enumerate}
\item Extract the opcode and the worker class id of the message. Also get the tid of the sender task (farmer task).

\item Using the tid, and the default message tag, receive another message from the 
farmer task. This would contain the work packet to be processed.
\item Call the scheduler with the worker class id and the message buffer id and the message tag of the buffer message.
\item In the scheduler, using the worker class id, locate the WCitem which contains the information about the work packets. 
\item If no worker class item is found, then return an ERROR. 
\item Find out if any worker task is free by checking if the item returned by asking for an item which has no packets pending, is NULL.
\item If yes, then using the tid of the task, send the message buffer to the worker task giving the message tag information and the buffer id.
\item Clear the buffer used to receive the message and look for other messages.
\item If there are no workers which can handle the message, then check the list of work packets. If it does not exist, create it and use the pointer to
the list of work packets and append the current packet to the end of the list of work packets associated with the worker class.
\item Go back and check for other messages arriving at the server.
\end{enumerate}



\subsection {\bf int	bufid = pvm\_recv\_work\_packet(void); }

\subsubsection*{Client side }
\begin{enumerate}
\item Create a buffer and send a message to the server containing op\-code RECV\_\-WORK\_\-PACKET and the worker class id, which is stored in a global variable.
\item Block on a receive from the server task with a wildcard message tag ($-1$). 
\item After receiving the message find out the message tag of 
the buffer received. 
\item If the message tag is FARM\_HAS\_TERMINATED, then return the constant PvmFarmHasTerminated (a negative number).
\item Else, return the positive integer which contains the buffer id of the message received.
\end{enumerate}
 
\subsubsection*{Server side }
\begin{enumerate}
\item If the opcode matched RECV\_WORK\_PACKET, extract the unique worker class id of the client and the tid of the client (using the buffer id of the received message).
\item Using this worker class id, search for the worker class item using the Cache of worker class ids (array index is wcid).
\item If the worker class item is not present (pointer is NULL), then that means that the farm associated with the current worker class id has terminated and the data structure associated with the farm has been deleted. Send a message to the client which has a message tag of FARM\_HAS\_TERMINATED and look for other service requests.
\item If the worker class item is present then using the client tid, search for a worker item. If not present then return ERROR.
\item Pop a work packet from the list of work packets pending and associated with the current worker class. 
\item If there are no pending items (pointer is NULL), then set the  number of packets pending field of the worker task as 0 and look for other service requests.
\item If there is a packet pending then set the active send buffer id to that of the packet item which has been popped.
\item Send the message to the client and free the buffer which was sent and the pointer of the popped item (housekeeping).
\item Look for other service requests.
\end{enumerate}


\subsection {\bf int	pvm\_send\_reply\_packet(void); }


\subsubsection*{Client side }

\begin{enumerate}
\item Get the worker class id and the farmer tid of the worker
task and validate these (if these are improper, return ERROR).
\item Send the message  in the active send buffer with message tag equal to the worker class id to the farmer task directly.
\item Return OK.
\end{enumerate}

\subsubsection*{Server side }
\begin{enumerate}
\item No action is performed by the server as the packet is sent directly from the worker task to the farmer task.
\end{enumerate}


\subsection {\bf int	bufid = pvm\_recv\_reply\_packet(worker\_class\_id); }
\subsubsection*{Client side }
\begin{enumerate}
\item Block on a wildcard receive (-1 - from anyone),having the message tag of the worker class id.
\item If the buffer id is negative, return ERROR, else return the buffer id of the received message.
\end{enumerate}
 
\subsubsection*{Server side }
\begin{enumerate}
\item Nothing is done explicitly by the server as this involves only the client and the farmer task.
\end{enumerate}


\subsection {\bf int	pvm\_farm\_terminate(char *farm\_name); }

\subsubsection*{Client side }
\begin{enumerate}
\item Send a message to the server containing opcode FARM\_TERMINATE and the farm name.
\item Wait for a reply from the server and unpack the return value from the message once it is received.
\item Return this value. It could be OK or PvmNoSuchFarm or PvmNotFarmOwner.
\end{enumerate}

\subsubsection*{Server side }
\begin{enumerate}
\item Receive the message and check if opcode is  FARM\_TERMINATE and get the tid of the client.
\item Search for the farm using the farm name.
\item If its not found, return a value NO\_SUCH\_FARM to the client.
\item If its found, check the client tid and the tid of the creator of the farm. If they do not match send a return value NOT\_FARM\_OWNER to the client.
\item If everything is OK, keep on popping all worker classes from the list of worker classes till there are no more worker classes.
\item In each worker class, get the worker class id and reset the entry in the Cache of the worker classes which contains the worker class id and the pointer to the worker class item. 
\item Then in the worker class while there are still worker items which have no packets pending (the ones which are blocked on the pvm\_recv\_work\_packet() request), process each worker item.
\item Send a message to the worker tasks which contains the inte\-ger FARM\-\_HAS\-\_TERMINATED with message tag of FARM\_HAS\_TERMINATED.
\item Set the number of packets pending as a non-zero number ($1$) so that the next search does not return the same entry.
\item Do the housekeeping with the buffer used for sending the message to the worker.
\item After all the blocked workers in the worker class have been sent this message, delete the worker list and the pending work packet list associated with the worker class.
\item After all  the worker classes have been processed, delete the worker class list (thereby destroying all worker classes associated with the farm).
\item Delete the farm item entry from the list of farms using the farm name.
\item Return a message OK to the farmer waiting for conformation on the farm.
\item Look for other messages.
\end{enumerate}

\subsection 
{\bf int	pvm\_start\_farmd(void);	}
\subsubsection*{Client side} 
\begin{enumerate}
\item Spawn a pvm task whose name is SERVER\_NAME. 
\item If it is spawned properly, return OK.
\item If there's an error in spawning the task, return ERROR.
\end{enumerate}
\subsubsection*{Server side}
\begin{enumerate}
\item Starts the pvmfarmd - farm daemon on one of the nodes.
\item Execute the function Server() which services requests from the tasks.
\end{enumerate}

\subsection 
{\bf int	pvm\_stop\_farmd(void); 	}
\subsubsection*{Client side}
\begin{enumerate}
\item Get the server task id. If it is not located, return ERROR.
\item Create a buffer and send an opcode STOP\_FARMD to the server process using DEF\_MSGTAG message tag.
\item Return OK.
\end{enumerate}
\subsubsection*{Server side}
\begin{enumerate}
\item If the opcode is STOP\_FARMD, then exit from the pvm system.
\item Exit from the operating system calling exit(0);
\item This shuts down the farming related server and should only be called once in the application after all useful farm related work is done. Even if this is not executed, the pvmfarmd can be shutdown manually from the pvm shell after the application has exited.
\end{enumerate}
\end{enumerate}



\end{document}

% ------------ end of document ----------------------

% here are some templates of different environments you can cut and paste:

\begin{figure}
\vspace{3in}
\caption{}
\end{figure}

\begin{itemize}
\item
...
\end{itemize}

\begin{enumerate}
\item
...
\end{enumerate}


\end{document}

% ------------ end of document ----------------------

% here are some templates of different environments you can cut and paste:

\begin{figure}
\vspace{3in}
\caption{}
\end{figure}

\begin{itemize}
\item
...
\end{itemize}

\begin{enumerate}
\item
...
\end{enumerate}

