FreePastry release notes

Release 2.0_04,  26 May, 2008.

FreePastry is a modular, open source implementation of the Pastry p2p structured overlay network.


Peter Druschel, Eric Engineer , Romer Gil , Andreas Haeberlen, Jeff Hoye, Y. Charlie Hu , Sitaram Iyer , Andrew Ladd , Alan Mislove, Animesh Nandi , Ansley Post, Charlie Reis , Dan Sandler , Jim Stewart, Atul Singh , and RongMei Zhang contributed to the FreePastry code. The code is based on algorithms and protocols described in the following papers:


The software requires a Java runtime, version 1.5.0+. The software was developed using Sun's SDK, version 1.5.0+

Changes since release 2.0_03

Bug Fixes

Changes since release 2.0_02

Bug Fixes

Changes since release 2.0_01

Bug fix in routing -- When you routed to a key that fully matched the id of a node in the network, the routing table repair mechanism would throw a NPE.

Changes since release 2.0

Bug fix for consistency -- When you called endpoint.route(null,msg,target); this message would still be dropped for possible consistency violations. There is no violation for such a request because it is for a specific node. This required an update to the Binary Format for the RouteMessage (version 1). See the Protocol Specification for the specific change.

To gracefully upgrade a ring you can specify:

pastry_protocol_router_routeMsgVersion = 0
to force the routing protocol to send version 0 messags until the majority of your ring is properly upgraded. Note that both versions 0 and 1 can still be received, but only version 0 will be sent, even if it receives a version 1 message. Note that you will still be subject to the bug while using version 0.

Changes since release 2.0beta2

Changes since release 2.0beta

Changes since release 1.4.4

Limitations of the beta.

Changes since release 1.4.3_02

Changes since release 1.4.3_01

This release is fixes some bugs with the direct simulator.

Changes since release 1.4.3

This release is fixes some bugs with the direct simulator.

Changes since release 1.4.2

This release is primarily performance enhancements and bug fixes.

Changes since release 1.4.1

Aside from various bug fixes, this version includes rice.environment which allows for node virtualization within the same JVM. FreePastry and it's p2p applilcations all use the new features. the Environment provides the following:

How to use parameters:

The Parameters interface contains: Default implementation:

When you construct the environment (new Environment()) it will use rice.environment.params.simple.SimpleParameters to manage parameters. The Environment constructor with no parameters will look for a file called freepastry.params in the classpath. This file is the default file, and is immutable, thus calling the store() method on the Parameters object will do nothing. The format for freepastry.params is explained here.
If you want to specify a file to save dynamic user parameters, or to override the default freepastry parameters, you can specify a single String fileName in the Environment constructor.
To override the default params filename, or to use a chain of defaults (for example epost uses a secondary default file to override and add to freepastry's defaults), specify a String[] of the defaults in order (for example freepastry, then epost), then specify a String for the dynamic user settings.

How to use logging:

You can access the environment from the rice.p2p.commonapi.Node interface. A programatic call to the Logger would typically look like:

  Node n;
  n.getEnvironment().getLogManager().getLogger(MyClass.class, instance).log(Logger.WARNING, "This is a warning");
  n.getEnvironment().getLogManager().getLogger(MyClass.class, instance).logException(Logger.WARNING, "This is an exception", new Exception());
Of course you can use helper methods to reduce typing.

The available log levels can be found in the rice.environment.logging.Logger interface, but the match java's log Levels.

By default the logging will go to System.out (standard output). To make the logging go elsewhere, construct the environment with a SimpleLogManager that is constructed with a PrintStream that you want (such as System.err, or a File).

The following parameters affect logging: Logging to file (used when you have multiple nodes in the same JVM, and want logging to go to separate files, with a name that is based on the node's id):

Other notes on the Environment:

Other changes:

Changes since release 1.4

Version 1.4.1 introduces the ConsistentJoinProtocol and some changes to the PeriodicLeafSetProtocol. These improvements provide stronger guarantees regarding routing consistency.

What is consistency?

Routing is consistent if no overlay node ever delivers a lookup message when it is not the current root node for the message's destination key. -- Microsoft Technical Report MSR-TR-2003-94

Wasn't FreePastry always consistent? What could have caused routing inconsistencies in previous versions?

Routing could be inconsistent in FreePastry if at some point there existed two adjacent nodes that are both active and do not know of each other, thus believing that they are responsible for some of the same keyspace. This could temporarily happen during concurrent joins, or if a node incorrectly determined its direct neighbor to be faulty, and therefore took over that section of the keyspace. The result of an inconsistency is that routing to a key (for example a DHT put/get) could cause the message to be delivered to different nodes, depending on the origin of the message. Thus you may do a PUT but then a subsequent GET still retrieves the old value!

How does FreePastry prevent routing inconsistency?

Our consistent join protocol is similar to that described in Microsoft Technical Report MSR-TR-2003-94, except that it does not require that all nodes are reachable from any other node at all times. The idea is that when a node joins, before it accepts messages, it must first contact its entire leafset and either receive a reply from each member or determine them faulty. This causes concurrent joiners to be aware of each other before both can become active. We rely on dynamic source routing introduced in FreePastry Version 1.4 to ensure that a node is considered alive as long as it is reachable by one of its leafset members.

The new protocol also dramatically reduces the overhead of leafset maintenance. By default the system is configured to send and receive a message from each neighbor every 20 seconds, and if it takes more than 30 seconds to hear from a neighbor, then a checkLiveness() starts. This ensures that a portion of ringspace never remains unclaimed for longer than 50 seconds (20 seconds between pings, then 30 seconds to wait for replies) after the death of a node.

We ran tests on PlanetLab to characterize the overhead of leafset maintence at various neighbor ping intervals. All of the data for the following graphs were collected on a ring running with no application and induced churn to give an average node lifetime of 60 minutes.

This graph represents the fraction of traffic that is due to leafset maintenance. One hundred percent would mean that all overlay traffic was due to leafset maintence. Leafset maintenance amounts to a modest 12.4% of traffic in packets even at a fairly aggressive neighbor ping rate of once every 20 seconds (the default). We expect the overhead as measured in bytes to go down in a future release of FreePastry that uses a binary wire protocol rather than java serialization.

This graph shows the average traffic due to leafset maintenance in bytes per second (smaller numbers are better). The error bars are at one standard deviation among the nodes in the overlay. At a 20 second neighbor ping interval leafset maintenance traffic is 71.5% of overlay traffic (top graph) but as this graph shows, that amounts to a data rate of only 500 bytes/second.

This graph shows the average traffic due to leafset maintenance in packets per second (smaller numbers are better). The error bars are at one standard deviation among the nodes in the overlay. At a 20 second neighbor ping interval, leafset maintence requires about 0.15 packets per second, or about 10 packets per minute.

Applications can trade off between the amount of maintence overhead they are willing to tolerate and the length of time they are willing to allow ringspace to be unclaimed by adjusting the neighbor ping interval.

New behavior:


We have done extensive routing consistency testing on Planetlab. The only times we find routing inconsistencies is during a network partition (usually the loss of a subnet). Routing in this case is only inconsistent across the partitions.

Changes since release 1.3.2

Changes since release 1.3.1

Changes since release 1.3

Changes since release 1.2

Changes since release 1.1


Release 1.4 has the following limitations.
(Background: To start a Pastry node, the IP address (and port number, unless the default port is used) of a "bootstrap" or "contact" node must be provided. If no such node is provided, and no other Pastry node runs on the local machine, then FreePastry creates a new overlay network with itself as the only node. Any node that is already part of the Pastry node can serve as the bootstrap node.)



To use the binary distribution, download the pastry jar file and set the Java classpath to include the path of the jar file. This can be done using the "-cp" command line argument, or by setting the CLASSPATH variable in your shell environment. For some applications you may need the 3rd party libraries included with the distribution. These are available in the source distributions. Simply unpack the distribution and include the jars in the lib/ directory in your classpath.

To compile the source distribution we have switched to ant for the build process. You will need to have ANT installed (available from on your system. Expand the archive (FreePastry-1.4.2-source.tgz or into a directory. Execute "ant" in the top level directory (you may have to increase the maximum memory for ant by setting the environment variable ANT_OPTS=-Xmx128m), then change to the "classes" directory to run FreePastry. 

You may have to provide a Java security policy file with sufficient permissions to allow FreePastry to contact other nodes. The simplest way to do this is to install a ".java.policy" file with the following content into your home directory:

grant {

Running FreePastry

1. To run a HelloWorld example:

     java [-cp pastry.jar] rice.pastry.testing.DistHelloWorld
          [-msgs m] [-nodes n] [-port p] [-bootstrap bshost[:bsport]] [-protocol [socket]]
          [-verbose|-silent|-verbosity v] [-help]

        Without -bootstrap bshost[:bsport], only localhost:p is used for bootstrap.
        Default verbosity is 8, -verbose is 1, and -silent is 10 (error msgs only).

        (replace "pastry.jar" by "FreePastry-<version>.jar", of course)

    Some interesting configurations:

  a. java rice.pastry.testing.DistHelloWorld

        Starts a standalone Pastry network, and sends two messages
        essentially to itself. Waits for anyone to connect to it,
        so terminate with ^C.

  b. java rice.pastry.testing.DistHelloWorld -nodes 2

        One node starts a Pastry network, and sends two messages to
        random destination addresses. At some point another node
        joins in, synchronizes their leaf sets and route sets, and
        sends two messages to random destinations. These may be
        delivered to either node with equal probability. Note how
        the sender node gets an "enroute" upcall from Pastry before
        forwarding the message.

  c. java rice.pastry.testing.DistHelloWorld -nodes 2 -verbose

        Also prints some interesting transport-level messages.

  d. pokey$ java rice.pastry.testing.DistHelloWorld
     gamma$ java rice.pastry.testing.DistHelloWorld -bootstrap pokey

        Two machines coordinate to form a Pastry network.

  e. pokey$ java rice.pastry.testing.DistHelloWorld
     gamma$ java rice.pastry.testing.DistHelloWorld -bootstrap pokey

        wait a few seconds, and interrupt with <ctrl-C>

     gamma$ java rice.pastry.testing.DistHelloWorld -bootstrap pokey

        The second client restarts with a new NodeID, and joins the
        Pastry network. One of them sends messages to the now-dead
        node, finds it down, and may or may not remove it
from the leaf sets. (repeat a few times to observe both
possibilities, i.e., leaf sets of size 3 or 5). If the
latter, then leaf set maintenance kicks in within a minute
on one of the nodes, and removes the stale entries.

f. pokey$ java rice.pastry.testing.DistHelloWorld
gamma$ java rice.pastry.testing.DistHelloWorld -bootstrap pokey -nodes 2

The client on gamma instantiates two virtual nodes, which
are independent in identity and functionality. Note how the
second virtual node bootstraps from the first (rather than
from pokey). Try starting say 10 or 30 virtual nodes, killing
with a <ctrl-C>, starting another bunch, etc.

2. To run the same HelloWorld application on an emulated network:

     java [-cp pastry.jar] rice.pastry.testing.HelloWorld [-msgs m] [-nodes n] [-verbose|-silent|-verbosity v] [-simultaneous_joins] [-simultaneous_msgs] [-help]

    Some interesting configurations:

  a. java rice.pastry.testing.HelloWorld

        Creates three nodes, and sends total three messages from
        randomly chosen nodes to random destinations addresses
        (which are delivered to the node with the numerically
        closest address).

  b. java rice.pastry.testing.HelloWorld -simultaneous_joins -simultaneous_msgs

        Join all three nodes at once, then issue three messages,
        then go about delivering them.

3. To run a regression test that constructs 500 nodes connected by an emulated network:

     java [-cp pastry.jar] rice.pastry.testing.DirectPastryRegrTest

4. To run a simple performance test based on an emulated network with successively larger numbers of nodes:

     java [-cp pastry.jar] rice.pastry.testing.DirectPastryPingTest

Writing applications on top of FreePastry

Applications that wish to use the native Pastry API must extend the class rice.pastry.client.PastryAppl. This class implements the Pastry API. Each application consists minimally of an application class that extends rice.pastry.client.PastryAppl, and a driver class that implements main(), creates and initializes one of more nodes, etc. Example applications and drivers can be found in rice.pastry.testing; the Hello World suite (,, may be a good starting point.

Another sample Pastry application is rice.scribe.

Application writers are strongly encouraged to base newly written applications on the new common API. Such applications should import the package rice.p2p.commonapi.

Running Scribe

1. To run a simple distributed test:

     java [-cp pastry.jar] rice.p2p.scribe.testing.ScribeRegrTest [-nodes n] [-port p] [-bootstrap bshost[:bsport]] [-protocol (direct|socket)] [-help]

        Ports p and bsport refer to contact port numbers (default = 5009).
        Without -bootstrap bshost[:bsport], only localhost:p is used for bootstrap.
        (replace "pastry.jar" by "FreePastry-<version>.jar", of course)

Running PAST

1. To run a simple distributed test requires placing the xmlpull.jar in the classpath:

     java -cp [.|pastry.jar];xmlpull<version>.jar;xpp<version>.jar rice.p2p.past.testing.PastRegrTest [-nodes n] [-protocol (direct|socket)]
In some unix shells, you may need to specify the classpath inside double quotes.
This creates a network of n nodes (10 by default), and then runs the Past regression test over these nodes.

Running SplitStream

The FreePastry implementation of SplitStream implements the system described in the SOSP '03 paper. class provides an interface that can be used by applications to create SplitStream instances. Each SplitStream forest is represented by a channel object (, where a channel object encapsulates multiple stripe trees. Each stripe tree for a SplitStream forest is represented by a class (, which handles the data reception and subscription failures.

Applications can configure the maximum capacity each channel can accommodate in terms of number of children it is willing to accept. Applications can control total outgoing capacity they are willing to provide by changing the value in

1. To run a simple distributed test:

     java [-cp pastry.jar] rice.p2p.splitstream.testing.SplitStreamRegrTest [-nodes n] [-protocol (direct|socket)]
This creates a network of n nodes (10 by default), and then runs the SplitStream regression test over these nodes.