JSON JS module for Firefox 3.0/3.5 API compatibility

I’ve written a JSON module to wrap the incompatible Firefox 3.0 and 3.5 JSON APIs, presenting the 3.5 API on both versions of Firefox, which is useful for extensions that support them both. Import this module to parse and stringify JSON in both 3.0 and 3.5 without having to check the host application’s version each time.

Note: don’t import this into the global namespace! If you do, you’ll hork native Firefox 3.0 code that expects the 3.0 API. Instead, import it into your own object like this:

let MyExtension = {
JSON: null,
...
};
Components.utils.import("chrome://myextension/modules/JSON.js", MyExtension);
// Now MyExtension.JSON is an object implementing the Firefox 3.5 JSON API.

The module also works in Thunderbird 3.0, which uses Firefox 3.5’s API (although there’s no need to use it for an extension that only supports Thunderbird 3.0). I use it in Personas, which supports both Firefox 3.0 and 3.5 as well as Thunderbird 3.0.

The Firefox 3.5 JSON API is documented in the Mozilla Developer Center article Using JSON in Firefox. This JSON module is documented on Mozilla Labs’ JS Modules wiki page.

 

Merging with Mercurial 1.0 and Meld

In an earlier post, I described my experience merging changes with Mercurial 0.9.5 and Meld 1.1.5.1 on Ubuntu 8.04.

Things have changed in Ubuntu 8.10, which ships Mercurial 1.0.1. Now Meld displays your version of the file on the left, the other version on the right, and the base version (from which yours and the other are descended) in the middle.

Here it is handling the Merging conflicting changes example from the hg book:

Your job is to reconcile the conflicts between your version on the left and the other version on the right, turning your version on the left into the result of the reconciliation, which you then save to your working directory.

I understand why it can be useful to show the base version, and I think it’s ok to use the same pane to show both your version and the final result (although kdiff3 provides a fourth pane beneath these three for the latter, which seems better).

But I don’t understand why the base version is in the middle, given that one’s primary task in this situation is to compare the differences between the two versions on either side of the window, copying changes from one to the other as needed. Putting another file in the middle just gets in the way of that process (which is presumably why kdiff3 puts the base version on the left and your version in the middle according to the screenshot in the aforementioned example in the hg book).

In fact, I frequently find myself starting a merge by copying the other version of the file into the base version in its entirety. That destroys my ability to refer to the base version, but it places the other version and my version side by side, at which point it’s much easier to compare the two and edit my version accordingly:

But maybe I’m missing the point or misusing the tool, and there’s some valid reason for the way Mercurial uses Meld. Can someone tell me what it is?

 

A XULRunner-Based Unit Test Harness

Prologue

After finding a couple bugs in the Observers JS module recently, I looked around for a simple test harness to exercise it and other JS modules. Weave‘s (based on Mozilla’s xpcshell harness) seemed promising, although it has to hack around various limitations of xpcshell.

Atul Varma had suggested some time ago that a XULRunner-based harness might make sense, so I decided to see if I could modify Weave’s harness to run JS module tests using XULRunner instead of xpcshell.

Act 1

That proved doable, although some details were tricky. Sometimes I wondered whether I wouldn’t have been better off reusing MochiTest, although it’s heavier weight. Other times I wondered why I didn’t find this bug about XULRunner-based unit tests sooner, although it’s not clear that reusing the attachment in that bug would have been faster.

In any case, I got the harness working with XULRunner and wrote an initial set of tests of the Observers module. To run them, pull the jsmodules repository, change to its test/unit/ subdirectory, and run make with the location of your XULRunner executable (or set the XULRUNNER_BIN environment variable to its location):

hg clone https://hg.mozdev.org/jsmodules
cd jsmodules/test/unit/
make xulrunner_bin=/usr/bin/xulrunner

Since Firefox 3.0 comes with XULRunner, you can use any Firefox 3.0 (or newer) executable instead (except on Linux distros that ship Firefox as a XULRunner app–you have to use the XULRunner executable on those), like /Applications/Firefox.app/Contents/MacOS/firefox-bin on Mac and /c/Program Files/Mozilla Firefox/firefox.exe on Windows.

Epilogue

Once I got the harness working for jsmodules, I backported it to Weave, where it mostly works, except that IWeaveCrypto.generateKeypair (which calls PK11_GenerateKeyPair) fails, apparently because of SEC_ERROR_TOKEN_NOT_LOGGED_IN.

I have no idea why, but if you do, comment here or in bug 476539, since that’s the last blocker to Weave using this harness.

 

Observers JS Module Improvements

I’ve updated the Observers JS module with a number of changes. In particular, it’s now possible to specify a this object when adding an observer that is a method, so instead of doing this:

let MyObserver = {
onFoo: function(subject) {...},
onBar: function(subject) {...},
onBaz: function(subject) {...},
QueryInterface: XPCOMUtils.generateQI([Ci.nsIObserver]),
observe: function(subject, topic, data) {
switch(topic) {
case "foo":
this.onFoo(subject);
break;
case "bar":
this.onBar(subject);
break;
case "baz":
this.onBaz(subject);
break;
}
}
}

Observers.add("foo", MyObserver);
Observers.add("bar", MyObserver);
Observers.add("baz", MyObserver);

You can do this:

let MyObserver = {
onFoo: function(subject) {...},
onBar: function(subject) {...},
onBaz: function(subject) {...}
}

Observers.add("foo", MyObserver.onFoo, MyObserver);
Observers.add("bar", MyObserver.onBar, MyObserver);
Observers.add("baz", MyObserver.onBaz, MyObserver);

Other changes include a couple of bug fixes and some backwards-incompatible API improvements. See the documentation for all the details (including steps to update from the previous version of the module).

 

Snowl Roadmap

Based on Snowl’s potential futures, and in line with its principles, I’ve put together a roadmap for its next three releases (beyond the imminent 0.2 release).

Generally, I’m aiming for six week development cycles that are that ambiguous mixture of date-, feature set-, and quality-driven, so the goal is to hit the target date with high quality implementations of the specified set of features. But work may be pushed to later releases or the schedule extended as appropriate.

Each cycle has a theme on which I’ll focus my efforts, and I’ll publish regular preview releases, which will be numbered with consecutive “pre[n]” suffixes (f.e. 0.3pre1, 0.3pre2).

Most of the work on the roadmap is stuff I’m planning to do myself, although it also lists tasks that other developers have expressed an interest in tackling. And it’s ambitious, so I probably won’t get to all of these tasks (although I’ll probably do other things that come up along the way).

If you want to do something with Snowl, whether or not it’s on the roadmap, check out the source code and go for it! (And let us know what you’re doing so others don’t duplicate your work.)

Caveat: nothing is written in stone, it’s the best laid plans of mice and men, take it with a grain of salt, etc. Among other things, this roadmap doesn’t account for time I may spend working on other projects, nor am I any better than other hackers at estimating the time it takes to do work I haven’t already done. And the farther out the roadmap goes, the fuzzier its plans.

Also, your suggestions are welcome. Are you missing something from the roadmap? Is there something on it that shouldn’t be there? Let us know!

The latest version lives on the Mozilla wiki at https://wiki.mozilla.org/Labs/Snowl/Roadmap. Currently, it looks like this:

Priorities

As with Firefox development (f.e. with Firefox 3.1 features), Snowl tasks are prioritized. For Snowl, the fuzzy prioritization scheme is:

  • P1: really want
  • P2: want
  • P3: nice to have

Releases

These are the target release dates for the imminent 0.2 release and the three releases after that:

  • 0.2: Monday, January 12
  • 0.3: Monday, March 9
  • 0.4: Monday, April 20
  • 0.5: Tuesday, May 26

(Note: The 0.3 cycle is eight weeks long to accommodate twoish weeks of vacation I have planned in January and February.)

0.3: simpler and smarter

Feeds

  • P1 expose feeds from frecent (frequent + recent) websites [atul]
  • P1 pick a feed automatically from among feeds with equivalent content
  • P1 let users subscribe to feeds by entering the location of a website
  • P2 automatically subscribe to feeds in live bookmarks [atul]
  • P2 automatically discard old feed items and tweets
  • P2 automatically determine how often to refresh feeds
  • P2 let users specify how long to retain feed items and tweets

Twitter

  • P1 support replying to messages
  • P2 support direct messages (private messages to specific users)
  • P2 automatically subscribe to Twitter account in browser history
  • P3 shorten long URLs when composing messages

Architecture/Extensibility

  • P1 simplify composition of messaging service implementations from Source and Target prototypes

Misc

  • P1 split collections view into resizable accounts/people sections [alta88]
  • P1 support email protocols [alta88]
  • P2 integrate Snowl into Thunderbird
  • P2 organize river view into editions
  • P3 use JS templates to build stream, river views

0.4: people and extensibility

People

  • P1 add a person-centric view
  • P1 automatically associate identities that represent the same person
  • P1 let users associate identities that represent the same person

Extensibility

  • P1 optimize database schema for sparse/custom attributes (f.e. username for Twitter, website URI for feeds)
  • P2 move OPML import/export processing into messaging service connectors so they can provide custom attributes (f.e. username for Twitter)
  • P2 compose subscription dialog from messaging service connector-specific overlays
  • P3 factor out common functionality from Feed and Twitter implementations into Source and Target prototypes

Conversations

  • P2 derive conversations from message metadata
  • P2 display messages in the contexts of the conversations to which they belong

Misc

  • P2 add a wordle-like view of messages

0.5: broader horizons

Feed Conversations

  • P1 retrieve comments on feed items automatically
  • P2 enable users to post comments on feed items

People

  • P1 consume contact info from Portable Contacts service providers
  • P2 provide contact info to Portable Contacts consumers

Misc

  • P1 support several additional messaging services/social networks
  • P2 support categorization of accounts and messages via tagging
  • P2 synchronize subscriptions to a cloud datastore
  • P3 synchronize all message data to a cloud datastore
 

Snowl Principles

Over the course of working on Snowl, I’ve established a set of principles to guide its development. They’re aspirational, but they have many practical ramifications. And while the current implementation doesn’t embody them yet by a long shot, I expect it conform more and more closely to them over time.

The principles are:

  1. feeds are a transport format, not a feature;
  2. people and conversations are first-class objects;
  3. one app can satisfy a broad range of users;
  4. searching is better than pre-categorization;
  5. browser features are useful for messaging;
  6. a messaging app is a platform.

feeds are a transport format, not a feature

Feeds are currently presented in many application interfaces (including Firefox’s) as a user-facing feature of the web. But feeds are data formats defining generic mechanisms for syndicating content. They’re designed to be machine-readable, not human-consumable, and interfaces that expose them directly are often too technical, like the way websites publish both RSS and Atom feeds and Firefox asks users to pick between them (as aptly described by Atul Varma).

Treating feeds as a transport format means that rather than exposing them directly, Snowl should use them internally to give users a way to do the things they want to do, like find out when a website changes or keep up with what their friends are doing on the web.

So when a website provides RSS and Atom feeds of the same content, Snowl should pick a format instead of asking the user to do so. And when a website provides comment feeds, Snowl shouldn’t expose those feeds to users, it should just show the conversations embodied in them to the users following that site.

The same is true for feeds provided by social network APIs. For example, the Last.fm API exposes the listening habits of its users’ friends as individual feeds, but Snowl shouldn’t make users manage individual feed subscriptions when they’re really just interested in what their friends are listening to. Instead, it should present those individual feeds as Last.fm friend updates, separating them by friend–not feed–where appropriate.

people and conversations are first-class objects

There’s no unique identifier for people on the internet. We each have an ever growing number of email addresses, usernames, and nicks on various websites, social networks, and messaging services. But we think of each person’s various identifiers as representing the same being, and so should Snowl, so its users never have to figure out how they received a message (email, blog post, tweet, etc.) when looking for one from a particular person.

Also, context is crucial in conversations, and it’s rarely obvious which message contains a particular portion of an extended conversation, so Snowl should let users interact with conversations rather than just their constituent messages, searching and browsing them as easily as they search and browse individual messages, and displaying messages in context.

one app can satisfy a broad range of users

Most users don’t want to configure their experience, they just want things to work. And Snowl shouldn’t force them to make unnecessary decisions. In fact, Snowl should need no preferences, because it always knows exactly what its users want.

Realistically, that’s not possible, but the closer we can get, the better (while not trying to be too smart in ways that frustrate users). For example, a number of feed readers let users specify how often to update each feed. But this is information that it’s possible to derive based on how often a feed is updated. There’s no reason for Snowl to expose this question to users who shouldn’t need to answer it.

Some users, however, do want to configure their experience. Others have unusual needs. And some have great ideas about how to create better experiences.

Firefox has shown that it’s possible to build an application that satisfies all these users, and Snowl should follow in its pawsteps, using hidden preferences, subtle affordances, extensibility APIs, and other techniques to make it possible for those users to customize and modify Snowl to their hearts’ content, while keeping the interface simple for everyone else.

searching is better than pre-categorization

Traditional messaging interfaces use pre-categorization to help you find messages. But the evolution of web search engines has shown that it’s faster and simpler in many cases to simply search for what you want. Snowl should make it trivial to find any message quickly by searching for its content or metadata.

browser features are useful for messaging

Web browsers have developed a variety of features over the last dozenish years to make it easier for users to browse, find, save, and recall the web pages they visit. Snowl should take advantage of all these innovations to make accessing messages better, utilizing features like tabs for opening multiple messages, bookmarks to flag them for later reference, tagging for categorization, and the awesomebar for searching them.

a messaging app is a platform

Snowl was never intended to be just a feed and tweet reader. Feeds and Twitter are just two popular services with open APIs whose disparate kinds of messaging were good testbeds for early architectural and user experience decisions. Communication over the internet is evolving rapidly, and messaging apps have to evolve alongside it.

The best way to do that is to make Snowl a platform that others can extend to new messaging services and user experiences.

So Snowl comprises a datastore of subscriptions, messages, and people; a set of APIs for retrieving, storing, and sending messages; and a variety of interfaces for accessing them. And it should employ a flexible schema (like CouchDB‘s) to make it trivial to extend its data model; expose an API that makes it easy to add support for new messaging services; and provide useful primitives for building innovative new ways of looking at messages.

 

Snowl Futures

With Snowl 0.2 almost out the door, I’ve been thinking about potential directions for its future development, and I’ve identified three possibilities.

Message Reader

The first is the idea that started the project: a message reader integrated into the browser, which uses Firefox features like tabs, bookmarks, history, the awesomebar, the sidebar, etc. to present messages.

Internet users live in their browsers, and much of their messaging is web-centric (feed items, tweets, messages sent via social networks, etc.). A message reader built into Firefox can aggregate those messages, store them locally (where their recipients have control over them), and provide a unified interface to them no matter where they come from and what protocols delivered them.

And it can help users uncover unknown sources of useful messages, like feeds from websites they frequent and updates from social networks to which they belong.

Snowl’s biggest weakness relative to web-based alternatives (webmail, web feed readers, messaging apps built into social networks) is that it doesn’t persist messages to a remote datastore, so users can’t access them from anywhere. But it certainly could do so using Weave, CouchDB, or some other technology for synchronization to cloud storage.

Social Activity Tracker

The second potential direction for Snowl is for it to utilize feeds and social network APIs to keep users informed about what their friends are doing on the web. In other words, it could take up the mantle of The Coop, the idea for a Firefox-based social activity tracker that labs prototyped in 2007.

Just as with a message reader, a social activity tracker built into the browser would be in a unique position to expose useful updates to users across all of the sites they visit.

Snowl would need some modifications to behave in this way. In particular, it would need to get better at retrieving and aggregating lists of friends across multiple social networks. But its architecture is already well-suited to the task in a number of respects, since social activity is frequently exposed through messaging protocols, which Snowl has been designed to support.

Better Feed Reader for Thunderbird

The third potential is to integrate Snowl into Thunderbird as a better feed reader for that application.

Thunderbird’s current feed reading feature is based on the Forumzilla feed reading extension I originally started working on in 1999/2000, and it suffers from a number of limitations and archaicisms. For example, it confines its messages to a separate “server” within the Thunderbird folder pane, and it provides only the traditional three-pane interface for reading them.

Snowl integrated into Thunderbird could break feed messages out of their “separate folder” ghetto, integrating them into the stream of messages from other sources, and provide a variety of useful views on them. And it would bring Twitter messaging to Thunderbird, too. Plus, Snowl developers would gain exposure to some of the interesting and innovative work happening in Thunderbird these days, which could benefit Snowl on Firefox.

The biggest challenge would be integrating Snowl into the very different capabilities provided by Thunderbird. Recent improvements to Thunderbird’s extensibility and support for browser-like features such as tabs will make this easier, however.

I think all these directions have potential and are worth exploring. And while I’m going to remain focused primarily on Snowl as a message reader for now, I’ll be conducting experiments along the way to learn more about the other possibilities. I’d also love to see other folks tackling them, so if you’re interested in doing so, let’s talk!

 

Snowl 0.2rc1

The first release candidate of Snowl 0.2 is now available.  This version sports no new features relative to the last preview release, just a bunch of bug fixes.  Barring the discovery of unexpected issues, this is the build that will be released as Snowl 0.2.

Notable bug fixes include:

  • an updated visual design with platform-specific icons for feeds and people;
  • Firefox is more responsive when Snowl is refreshing sources and building views;
  • multiple Twitter accounts no longer occasionally get confused for each other;
  • views update more consistently when messages and sources are added or removed;
  • the river and stream views always show messages in the order they were received.

As of this release, I’ve also switched to hosting Snowl development on the same infrastructure (source code repository, bug tracker, discussion forums, etc.) as other labs projects to make it easier for participants in those other projects to contribute to Snowl.

Give the release candidate a try, and let me know what you think.

Install Snowl

Discussion Forum | Bug Reports | Report a Bug | Source Code

 

simpler, easier string bundle API

In the process of improving localizability in Snowl today, I put together a StringBundle JS module to make it easier to access string bundles from other JS modules (and JS XPCOM components and chrome JS, for that matter).

The module has a simple, easy-to-use API with a single get method that retrieves both plain and formatted strings:

let strings = new StringBundle("chrome://example/locale/strings.properties");
let foo = strings.get("foo");
let barFormatted = strings.get("bar", [arg1, arg2]);
for each (let string in strings.getAll())
dump(string.key + " = " + string.value + "n");

However, it also supports the API for the stringbundle XBL binding to make it easier to move from the binding to the module:

let strings = document.getElementById("myStringBundleElement");
new StringBundle("chrome://example/locale/strings.properties");
let foo = strings.getString("foo");
let barFormatted = strings.getFormattedString("bar", [arg1, arg2]);
let enumerator = strings.strings;
while (enumerator.hasMoreElements()) {
let string = enumerator.getNext().QueryInterface(Ci.nsIPropertyElement);
dump(string.key + " = " + string.value + "n");
}

The module is available with the other useful JS modules in the jsmodules project on mozdev.org. Documentation is embedded inside the module as well as on the project wiki.