Evaluation of J-Sim
October 17, 2003
In what follows, we summarize
notable research efforts on network simulation and compare J-Sim
against them wherever appropriate. Some of the simulation
packages are out-dated (but the results of which have been
leveraged in subsequent projects) and are listed here for
historical purpose:
-
NEtwork Simulation Testbed (NEST)
is a general-purpose simulation package designed
to simulate distributed networked systems and protocols. It provides
a client-server based graphical
environment for simulation construction and execution control.
-
Maryland Routing Simulator (MaRS)
is a network simulation package based on a general-purpose
network simulator NetSim developed earlier.
As the name suggests, it is mainly used to study different routing
algorithms.
-
The REalistic And Large (REAL) network simulator
is a substantially improved, and faster, version of NEST and
is designed specifically for studying different congestion and flow
control mechanisms in TCP/IP networks.
-
Ns-2 is part of the collaborative VINT
project involving USC/ISI, Xerox PARC, LBNL, and UC Berkeley. Ns-2
began as a variant of the REAL network simulator
in 1989, and has evolved substantially over the past few years. It provides
substantial support for simulation of TCP, routing, and multicast protocols,
but due to the special node structure in ns-2,
it is non-trivial, and sometimes difficult, to include other
protocols/algorithms or accommodate other network architectures in ns-2.
In addition, the not-so-structured software architecture and the mixture
of compiled and interpreted classes make it difficult to understand
and validate ns-2 codes.
-
The S3 project and
SSF
are collaboration efforts of Rutgers University, Dartmouth College,
Georgia Tech, and Boston University, and come closest to
our proposed work. The project focuses on distributed and parallel
simulation for large network simulation and proceeds along two fronts:
design and implementation of (i) a simulator-independent network modeling framework, called
Scalable Simulation Framework (SSF); and (ii) parallel and distributed simulation
kernels (such as Georgia Time Warp (GTW)). SSF consists of 5 base
classes: Entity, inChannel, outChannel, Process, and Event,
and defines APIs to separate simulation modeling from parallel simulation internals.
Two independent implementations of the SSF API have been provided: JSSF (written
in Java) and DaSSF (written in C++).
They also include a Domain Modeling Language (DML) to synthesize a model
and instantiate a simulation.
SSF and J-Sim are
similar in terms of the (i) object-oriented, layered software architecture design and (ii)
isolation of modeling from simulation internals. There are nevertheless several
important differences between SSF and J-Sim, which are tabulated in Table 1.
Table 1. A comparison between SSF and J-Sim.
|
Aspect |
SSF |
J-Sim |
|
Object-oriented paradigm |
Five base classes:
(i) Entity serves as a
container mechanism for defining alignment of a group
of components to common local time (e.g., ProtocolGraph
extends Entity);
(ii) inChannel and
(iii) outChannel are communication endpoints
for event exchange;
(iv) Event is
the base class for information exchange; and
(v) Process is
the base class for describing dynamic behavior. Each
instance of process is owned by a specific entity and
may wait for input to arrive on channels owned by the
entity, wait for some amount of simulation time to
elapse, or do both.
|
A simple and well-defined
component-based software architecture: Component
is the base class. Ports are the only
interfaces of a component to send/receive data. When
data arrives at a port, an execution context (a Java
thread) is created for the component to process the
data. Components are asynchronous in the sense that
two components may process different data at the same
time without synchronizing between each other.
Components can be
hierarchically structured. A component may be itself
a container mechanism and consist of subcomponents.
This facilitates hierarchical modeling of complex
systems.
|
|
Simulation framework |
Classes interact with the
underlying simulation engine with defined APIs |
Simulation engine is built in
the runtime and is transparent to components. |
|
Process-oriented modeling |
A set of wait() methods
is provided for modelers to write process-based
models. |
Provides, in addition to the
set of wait() methods, synchronization methods to
further extend programming flexibility |
|
Simulation technique |
Discrete event simulation |
Can work with both discrete
event simulation and real-time process-based
simulation |
|
Performance scalability |
Highly scalable by use of
parallel simulation kernels |
Will implement a parallel
simulation engine (that takes advantage of the conservative
approach) in the autonomous component architecture
|
As INET/J-Sim is laid
atop the autonomous component architecture,
a component layer, called SSFNET, is positioned
on top of SSF, contains classes for network hosts and routers, and
implements major Internet protocols (IP, TCP, UDP, OSPFv2,
BGP-4, and the pseudo-network interface drivers,
sockets), global Internet topology construction and IP address assignment.
The differences between SSFNET and INET are tabulated in Table 2.
Table 2. A comparison
between SSFNET and INET/J-Sim.
|
Aspect |
SSFNET |
INET/J-Sim |
|
Configuration |
The configuration files are in
the Domain Modeling Language (DML) format.
They are used to synthesize a model and instantiate a
simulation run with the help of the configuration
database package SSF.DML. The language is,
however, not suitable to be used in an interaction
environment. |
A dual-language environment is
provided: Java is used to create components and a
script language is used as a glue or control language
that integrates components in a desired fashion at
run time and provides high-level, dynamic control.
This facilitates fast prototyping of customized
simulation scenarios, and runtime configuration and
diagnosis. |
|
Modeling |
A node is an Entity. The
infrastructure inside a node does not follow the
container semantics in SSF but follows the protocol
graph/protocol session semantics in x-kernel. It is
unclear on how to extend the structure. |
A node is a composite component
and all modules in a node are components. A general
node structure is developed in
INET
to facilitate constructing specific
node structures. |
Network and Protocol
Models |
Support major Internet
protocols (IP, TCP, UDP, OSPFv2, BGP-4, and the
pseudo-network interface drivers NIC, sockets),
global Internet topology construction and IP address
assignment. |
Support, in addition to those
supported by SSFNET (except BGP), also - multicast routing
protocols (such as DVMRP, MOSPF, CBT),
- QoS-driven
protocols (such as QoS extension to OSPFv2,
QoS extension to CBT, and various rate-based
message scheduling disciplines),
- protocols in the
integrated services architecture (such as
RSVP signaling),
- components in the
differentiated services architecture (marker/tagger
at edge routers and buffer management
algorithms at core routers).
|
- Fluid simulation also
aims to improve scalability, but from a different perspective. It consists of two
major approaches: (i) hierarchical fluid
modeling: it abstracts a potentially complex component of
the simulation by a simpler model and substitution
of this simpler model into the larger simulation;
(ii) modeling at multiple time scales: it
simulates system behavior at coarser time-scales,
allowing for significant savings in simulation execution times.
Conceptually, network traffic is
modeled as a continuous fluid model and time-driven
simulation is used to discretize the
continuous fluid mode and simulate the
fluid backlogs (e.g., the back-logged fluid
at a server, the departure process at
a server) at discrete times. They prove that the
discretization error is bounded
by a constant proportional to the discretization interval
length. Although conceptually
innovative, it is not clear whether or not most, if not
all, traffic can be well characterized in fluid
models without losing modeling accuracy.
- Although the
Ptolemy project of UC Berkeley
is not focused on network simulation, it has
also come to our attention because of its similarity to the autonomous
component architecture in building component-based
architectures for general-purpose simulation.
Ptolemy mainly focuses on modeling and simulation of concurrent,
real-time, embedded systems, but shares similar philosophy with us in
terms of component-based models. They use well-defined models of computation
to govern the interaction between components. The models of computation they have
developed include communicating sequential processes, continuous
time, discrete events, distributed discrete events and so on.
These models, however, have their own communication semantics.
Components derived from different models can not be mixed
together to form truly heterogeneous systems.
J-Sim, on the other hand, has a simple, unified and concrete
component model. Components developed in different domains of application can
easily be integrated as long as the contracts are matched.
Scenario: Multiple TCP connections through one bottleneck link,
n TCP connections, 2n hosts, 2 routers and one bottleneck link with buffer
4n Kbytes, packet size=1Kbytes.
Parameter: Number of TCP connections, n; varied from 10 to 10,000.
Characteristics of the test:
- The number of the packets being transmitted over the bottleneck link
(and therefore the number of packet events) is irrelevant to n.
- Event queue is at least n (TCP timeout event).
- Routing table size is (n+1).
Results collected: Scenario setup time (time to create and configure the network before
simulation starts) and time to complete 100-, 500-, and 1000-second simulation.
Testee: J-Sim1.0, Ns-2 v2.1b8a and SSFNET v1.3.
Machine:Dual PentiumIII 600MHz, 256MByte RAM, Linux 2.4.8.
JVM: (for J-Sim and SSFNET) IBM JDK1.3 and
Blackdown v1.3.1.
Arguments passed to JVM: "-Xms200000000 -Xmx200000000".
Result (Setup time): For Javasim, we set up the scenario in
three different ways: Java program (J-Sim/Java), Tcl script (J-Sim/Tcl)
and Python script (J-Sim/Python).
* Results vary a lot when running SSFNET over 900 TCP connections probably due to
the garbage collection process in IBM's JVM.
* Out-of-memory exception occurs when setting up the scenario for more than 7,500 TCP connections
in SSFNET.
Result (Completion time, IBM JDK1.3): For J-Sim, we implement both a real-time simulation engine
and a pure event-based simulation engine to observe the additional overhead incurred by the real-time
simulation technique.
* Results vary a lot when running SSFNET over 900 TCP connections probably due to
the garbage collection process in IBM's JVM. As such, the results of SSFNET
is not shown in the following figures.
* The sawtooth pattern in the ns-2 curves (including those in the figures below) is due to the instability
of the calendar queue. That is, the performance of the calendar queue is sensitive to the enqueue pattern
of the simulation events.
Result (Completion time, Blackdown1.3.1):
* Out-of-memory exception occurs when running simulation for more than 3,310 TCP connections
in SSFNET.
Conclusions from the experiment:
- Scenario setup time, often neglected, takes a non-negligible, and even larger portion
of the entire time of a simulation run.
For example, it takes about 393 seconds for ns-2 to set up the 4000 TCP connections scenario
and takes about 181 seconds to complete a 1000-second simulation.
- J-Sim scales better in terms of memory usage. While Javasim can run through 10,000 TCP
connections without using up memory, ns-2 causes system to thrash at about 4,500 TCP
connections, and SSFNET throws out-of-memory exception at about 3,310 TCP connections.
- J-Sim suffers from inter-component communication overhead and JVM. And the real-time
simulation engine incurs more overhead than the pure event simulation engine. The offsets
of the completion time at small number of TCP connections in all figures are the evidence.
- As for the scalability in terms of the completion time v.s. number of connections,
J-Sim performs better than SSFNET as J-Sim's curves grow more slowly than SSFNET's.
J-Sim also performs better than ns-2 when simulation time is 100 seconds, but worse
than ns-2 when simulation time is 500 and 1000 seconds, before ns-2 uses up memory.
Scripts and modified source files:
- J-Sim: (v1.0)
- Java: OneBottleneck.java.
Usage:
java OneBottleneck <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
Argument:
interval<nn>: interval (in simulation time) to report wall time elapsed
event: use event simulation engine instead of real-time simulation engine (default)
radix: use radix-tree routing table instead of multifield routing table (default)
Example:
java -Xms200000000 -Xmx200000000 OneBottleneck 100 1000 interval200 event radix
- Tcl: onebottleneck.tcl.
Usage:
java drcl.ruv.System -u onebottleneck.tcl <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
Argument:
See "Java" above.
Example:
java -Xms200000000 -Xmx200000000 drcl.ruv.System -u onebottleneck.tcl 100 1000 interval200 event radix
- Python: onebottleneck.py.
Usage:
java drcl.ruv.System -u -s drcl.ruv.ShellPython onebottleneck.py <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
Argument:
See "Java" above.
Example:
java -Xms200000000 -Xmx200000000 drcl.ruv.System -u -s drcl.ruv.ShellPython onebottleneck.py 100 1000 interval200 event radix
- ns-2: (v2.1b8a)
- SSFNET: (v1.3)
|
