AllPeers Hacker's Getting Started Guide

AllPeers is a Mozilla-based platform that implements peer-to-peer capabilities, strong identity, presence and a framework for generic resource management. It is currently available as a Firefox extension. There are plans to make it available as a standalone application as well, on top of which alternate UIs can be constructed using interprocess communication.

As Mark Twain once said, “Everyone talks about the weather, but no one does anything about it.” Similarly, there has been a lot of talk about what a next-generation web browser might resemble, but so far relatively few real innovations have found there way in front of consumers. This will doubtless change over time with efforts like Mozilla Labs picking up steam.

AllPeers offers the possibility to see some interesting new browser features in action today. Do you have an application idea that could benefit from every user having a unique identity signed by a digital certificate, with real-time presence information to track which users are online? How about a generic resource framework that lets you replicate structured data across the network without the need for a centralized server, and store it in an efficient SQL-backed data store? Or maybe your application needs to manage the distribution of large media files efficiently using P2P infrastructure? If so, our platform may be what you’re looking for.

If you’re an AllPeers user, you may have been struck by all the potential synergies with other applications, both web-hosted and desktop-based. Wouldn’t it be nice to drag-and-share your photos and videos onto Flickr, YouTube, MySpace and other media sharing websites? Or you could enqueue music straight from AllPeers into the media player of your choice. Another interesting avenue of exploration is integrating AllPeers with other devices, such as mobile phones and set-top boxes. We can’t do all this ourselves, but many of these applications would be fairly straightforward for outside contributors to build on top of our platform.

We won’t lie to you. Hacking AllPeers right now is no trivial endeavor. First of all, you really need to be a competent Mozilla hacker. We have plans to provide new APIs at some point in the future that will make AllPeers functionality accessible to anyone with web development skills. But for now, if you’re not comfortable pulling and building Mozilla, this might not be for you.

A wealth of information and pointers about hacking Mozilla is available here: Mozilla Hacker's Getting Started Guide. Pretty much everything in that document (coding conventions, tools, libraries) applies to AllPeers as well. If you don’t have a Mozilla build already, you probably want to make one before continuing.

Still here? Good. Once you’ve got Mozilla built, you need to pull and build AllPeers. To understand how AllPeers fits inside your Mozilla source code tree, we highly recommend that you read Creating Custom Firefox Extensions with the Mozilla Build System. You can find instructions on how to get and build the AllPeers source code on the homepage of this site.

AllPeers is divided into a number of high-level modules. We make heavy use of XPCOM to encapsulate functionality into components and use these components across modules. Most of the code is written in C++, with some portions (especially those relating to the user interface) in JavaScript.

The major high-level modules are:

• Core functionality
• Resource framework
• Low-level networking
• Resource-level networking
• Application-level networking
• Data transfer
• User interface

Each of these is further divided into submodules, all of which are described in more detail is the following sections.

The logging subsystem provides extensive facilities for logging information to the file system. Four logging levels are available: marrow (areas that are well tested and don’t need logging other than in exceptional cases), bone (deep logging useful for debugging purposes), flesh (standard logging) and skin (always logged). Log files are written to the directory specified when starting the log service. A new log file is created whenever the current file exceeds the specified maximum size, and log files are cleaned up automatically when more than the specified number are present in the log directory.

Sometimes known as “things that could be in Mozilla but aren’t”. We created this module for core functionality that Mozilla doesn’t provide. Examples are various implementations of streams and enumerators, thread-safe versions of existing components and XPCOM implementations of native AllPeers types (date and binary). We hope that one day at least some of this code will find its way into Mozilla.

Almost every component is a wrapper for NSS (Netscape Security Services) API. Used by resource and network code for handling PKI. Also there are components for data symmetric encryption and signing/verifying. Hold services for managing user identity. User identity means presence of a private key generated by a user and signed certificate with paired public key.

The XML module consists mainly of code for managing our schemas, which use RELAX NG syntax. However, we use Mozilla’s schema interfaces (based on the W3C XML Schema Recommendation) to expose the schemas to the rest of AllPeers, so that W3C XML Schema can also be used theoretically for expressing AllPeers schemas. One major inconvenience is that Mozilla’s interfaces use UTF-16 whereas we use UTF-8 for all our interfaces. We don’t see any short-term solution to this so we simply convert the strings (using NS_ConvertUTF8toUTF16 and NS_ConvertUTF16toUTF8) when necessary.

This module also contains implementations of an XML reader, writer, SAX handler and in-memory document, based on GNOME libxml2. These are no longer widely used in AllPeers (only used for XML-based network communication), and we hope to remove them entirely in a future version so that we can eliminate the dependency on libxml2.

AllPeers uses a generic resource framework for managing data in its P2P network. This enables us to manage diverse types of data using a single general-purpose implementation. Specifically, the code for storing data in the persistent data store and distributing it to other users will work without modification on any type of resource. Every resource has a resource type which describes the data the resource may contain. A resource type is described using an XML schema; the schemas distributed as part of AllPeers use extended RELAX NG syntax. The main extension is to enable type inheritance since inheritance and polymorphism are used extensively.

Basic resource functionality includes the resource implementation itself in the apResource class. Resources are attached to a data source, where the actual data is kept, so the resource object itself has only a unique ID, the associated complex schema type and an optional metadata resource. The latter is used for managing data which should be stored locally but which is not included in the serialization of the resource when it is sent over the wire.

Resources consist of properties (which have a simple type) and children (which have a complex type). At the present time, all properties of all resource types must reside in the same namespace, so each property name must be unique to a given resource type. Children are resources in their own right, but they are anonymous (i.e. they have no identity independently of the parent resource). Non-anonymous resources are always derived from the Resource resource type and are associated with a resource description, which is the complex type plus extensions (a few extra attributes like a human-readable title, as well as schema describing the local metadata associated with the resource type). Anonymous resources are not derive from Resource and do not have a resource description. They are simply associated with a normal complex type.

Resource properties can use one of five datatypes: boolean, integer, decimal, date and binary. The last two can be accessed using XPCOM interfaces: apIDateTime and nsIStorageStream, respectively. All integers use 64 bits and all decimals are based on the native C double type.

Lists of resources use the special apResourceList class. A list can contain an explicit enumeration of the resources it contains, or it can be associated with a query that determines the contents based on certain criteria (the latter functionality is not fully implemented at this time). The apIResourceList interface provides special methods to make it easier to access the contents of the list.

Each resource type may be associated with a specific implementation of the apIResourceAdapter interface. Resource adapters can be used to implement functionality that may differ for various resource types, such an initializing a resource of that type.

Queries are built using apIResourceQuery and associated interfaces. Current query functionality supported is matching properties of resources and resource metadata, querying for lists that contain resources matching specific criteria, querying for items of lists that match specific criteria. Normally, all criteria must be matched for a given resource to be returned. The apIResourceQueryClause.addOrCondition() method can be used to specify disjunctive conditions.

AllPeers database functionality maps the resources used for native data management into SQL tables. The standard implementation uses SQLite (which is included with Mozilla), and we also have a MySQL-based implementation which is currently not available in source code form (though we may make it available in the future).

The database service enables persistent resources to be retrieved by ID or using a resource query. The database data source provides methods for making a resource persistent (i.e. storing it to the database) or unpersistent (i.e. deleting it from the database). Any changes to the properties of a persistent resource are automatically updated in the database. The data source also has interfaces for registering listeners that are called when specific changes occur to resources. These are very useful, for example, for keeping the user interface current when data changes.

One of the most important resource types is File (derived from Resource). This is the base type for all file system files. Specific types (derived from File) are provided for various types of media: Audio, Video, Image and Text. Files that don’t match any of these use the generic base type. We provide components for managing the link between a resource (which describes the properties of a file) and the actual physical file, as well as components for extracting metadata from specific types of media files (audio, video and images). Two open source libraries, FFMpeg and FreeImage are used to parse the various media formats.

This is an interesting module in that it lies at the intersection of the resource framework and the network functionality. Its purpose is to define the Peer resource type and provide functionality specific to that type. The Peer type is used to represent users of the AllPeers network. It is not intended to represent a complete user profile (i.e. with contact details, etc.) although we do plan to add this information in the future in another resource that could be linked to the peer. Since resources can only be shared with peers, functionality relating to sharing is also contained in this module. Mainly this module is intended as the top level user API and its implementation. Also contains services for creation of user’s (peer’s) identity, i.e. registration.

This module handles basic network functionality including name resolution (handled very differently from web DNS), managing and resolving hubs (relay servers) and managing proxy settings. It also provides a socket abstraction that enables usage of any BSD-style sockets in our framework (see TCP and MFP modules).

Maintains a centralized “network context”. This keeps track of the user’s identity, creates outgoing connections (using Mozilla interfaces), accepts incoming connections (using a generic data recognizer), manages the online status of the network and ties together all of this functionality. The network context also creates a thread for handling all network operations. For establishing outgoing connections and accepting incoming connections, the TCP and MFP plug-in modules are used.

Simplified implementation of the XMPP protocol for communication based on XML packets.

A wrapper (currently used only for XMPP) that provides a high-level of reliability for packet-based communication. Each packet (made up on name/value pairs) is recognized by its XML namespace. A generic framework is used for the creation of high-level (application-level in the AllPeers architecture) network services. Both client/server (e.g. interaction with the registration server) and peer-to-peer (e.g. chat) services are supported. There is also support for unidirectional XML packet channels on top of an existing XML session.

Simple low-level protocol for automatic determination of a user’s TCP connectivity status. This service enables the discovery of the user’s public TCP address and its accessibility from other machines over the public internet.

Full implementation of the STUN discovery protocol, used to determine client capabilities for UDP communication and NAT traversal.

Fully pluggable implementation of the TCP protocol. A generic mechanism is used to determine the optimal way to connect to a target peer, based on the connectivity status of both peers. If possible, a direct connection is established. If only one machine is directly accessible, it can request that the target peer connect to it using a callback mechanism. Other a third-party relay is used to ensure the connection. A highly secure SSL tunnel is used for communication over the established connection. The use of encryption ensures that a relay, if used, cannot eavesdrop on the relayed communication.

Fully pluggable implementation of the MFP protocol. MFP is a UDP-based protocol developed by Amicima (now part of Adobe). As such it offers a more lightweight approach for connecting peers, and one that facilitates NAT traversal to increase the likelihood that peers can be connected directly. MFP adds traffic shaping and reliability to UDP, among other features. As with TCP, all communication is encrypted and highly secure.

Components and services providing low-level functionality for sending single packets to multiple peers, sending datagrams and sending messages through offline storage (offline messaging). Offline messaging is used when the target peer is not available at the time a message is sent. In this case, the message is stored on a server and delivered the next time the peer comes online.

Simplified implementation of an HTTP client and server. This is used for communication through HTTP proxies, for data transfers and for other purposes. Generic HTTP request handlers can be created to process HTTP requests.

Provides low-level services for limiting upload speed to reduce bandwidth consumption. There are also components for measuring connection speed. A download throttle is also planned.

Provides a user interface and a network service for P2P instant messaging. The chat service uses the XML service framework and unidirectional XML channels to create a chat session. The implementation is currently limited to one-to-one chat sessions, although group chat functionality is planned. This module provides a good illustration of how to build a sophisticated P2P service using the AllPeers framework.

Implementation of an SMTP client and services for background transmission of emails. This includes a client/server service for sending emails, where recipients are peers on the AllPeers network. The server part runs on a server machine and is configured to use a particular SMTP server. The client service connects to the server and enables sending of both HTML and plain text emails to peers in the AllPeers network.

File resources, which are actually metadata for file on the user’s file system, are associated with the corresponding file data. These modules provide abstractions and concrete implementations for upload and download protocols for data interchange between peers. Additional data transfer protocols can be implemented using this framework.

Contains an abstraction for data transfer protocols, consisting of a few interfaces and services that must be implemented. Services are provided for download management, upload monitoring and data cache management. The data cache is an abstraction that encapsulates access to locally stored file data and is mostly implement in the File module. A specific protocol can implement its own data access mechanism (as is the case with our BitTorrent implementation).

This module also providers a lightweight plug-in protocol based on HTTP. This is a simple one-to-one data transfer protocol, which is fast but limited in capabilities compared to our BitTorrent protocol.

This is a complete implementation of BitTorrent client built on top of low-level Mozilla services and our network framework. It is registered as a plug-in service in AllPeers data transfer framework. It can be used for normal torrents as well as for our own data interchange (for which a sophisticated source propagation algorithm is used, rather than a centralized tracker). Both single-file and multiple-file torrents can be handled. Support for encrypted communication is included.

The AllPeers user interface is a combination of XUL and dynamically generated HTML. The workbench contains the code for generic user interface functionality, with certain specific user interface elements (e.g. forms and templates) located in the appropriate module. For example, forms for sharing are contains in the peer module and templates for displaying files as HTML are contained in the file module.

HTML generation is based on a view/viewer/provider architecture similar to that of Eclipse. Templates are used to describe the desired output, with placeholders for dynamic data. This data is retrieved from a provider component. This framework is likely to undergo major revision in the near future.

We will be adding more information to this website on a regular basis. You can already explore the rest of the site to find more details about developing on top of AllPeers. Another option is to visit us in our IRC channel #allpeers on the irc.mozilla.org server.