A Mozilla App Outside Central

After forking gecko-dev again for an experiment, I wondered if there was a better way to create a Mozilla app outside the mozilla-central repository. After all, that repository, mirrored as mozilla/gecko-dev on GitHub, is huge. And forking the same repo twice to the same GitHub account (or organization) requires cumbersome workarounds (and workarounds for the workarounds in some cases).

Plus, Mozilla apps don’t necessarily need to modify Gecko. But even when they do, they could share a single fork, with branches for each app, if there was a way to create a Mozilla app that imported gecko-dev as a dependency without having to live inside a fork of gecko-dev itself.

So I created an example app that does just that. It happens to use a shallow Git submodule to import gecko-dev, but it could use a Git subtree or any other dependency management tool.

The build process touches only one file in the gecko-dev subdirectory: it creates a symlink to the parent (app) directory to work around the limitation that Mozilla app directories must be subdirectories of gecko-dev.

To try it for yourself, clone the mykmelez/mozilla-app repo and build it:

git clone --recursive --shallow-submodules https://github.com/mykmelez/mozilla-app
cd mozilla-app/
./mach build && ./mach run

You should see a window like this:

Mozilla app window

And your clone will contain the app’s files, the gecko-dev submodule directory, and an obj directory containing the built app:

> ls
LICENSE  app.mozbuild  components   gecko-dev  modules    moz.configure  obj
app      branding      confvars.sh  mach       moz.build  mozconfig

Caveats:

The app builds Gecko from source, so you still need Mozilla build prerequisites. On some platforms, you can install these via ./mach bootstrap.

I’ve only tested it on Mac, but it probably works on Linux, and it probably doesn’t work on Windows (because of the symlink).

Although the Mozilla application framework is mature and robust, it isn’t explicitly generalized for use beyond Firefox anymore, and it could change or break at any time.

Continuous Integration of Positron with TaskCluster

I recently enabled continuous integration of Positron using TaskCluster’s support for GitHub repositories. I’ve used Travis CI to integrate other GitHub projects, but Positron is too large for Travis to build without timing out, since it builds all of Gecko. So I needed an alternative.

Caveat: TaskCluster only supports GitHub repositories in the mozilla organization or another org that is explicitly configured for it, as described by Setting up Taskcluster with a Github Organization. So it won’t work with your personal fork.

Also, while the Positron repository is based on gecko-dev, I didn’t use GitHub’s “fork” feature to create it, since GitHub won’t let you fork a repo to another repo in the same organization. Instead, I created a new repository and then copied gecko-dev’s commits to it, as described in Copy/fork a git repo on github into same organization.

(I also had to push the copied commits to GitHub in batches, as described in Github remote push pack size exceeded. And I had to push the first batch via special syntax, per Pushing a large github repo fails with “unable to push to unqualified destination: master”.)

To enable CI, I first created this Docker image, based on Ubuntu 14.04 (Trusty), that includes the Mozilla build toolchain and dependencies installed by Gecko’s bootstrap.py script. The image also includes the taskcluster-vcs utility, which caches gecko-dev and clones from the cache so that TaskCluster pulls from GitHub are lighter-weight.

Here’s the Dockerfile for that image:

FROM ubuntu:trusty

RUN apt-get update -y && \
    apt-get install -y \
      curl \
      git \
      nodejs \
      nodejs-legacy \
      npm \
      python \
      wget

RUN npm install -g taskcluster-vcs

RUN wget -O bootstrap.py https://raw.githubusercontent.com/mozilla/positron/master/python/mozboot/bin/bootstrap.py && \
    python bootstrap.py --no-interactive --application-choice=browser

Then I replaced the upstream .taskcluster.yml file with one that clones and builds Positron when a developer submits a pull request or lands a change. .taskcluster.yml is like .travis.yml, but for TaskCluster. Here’s that file:

version: 0
metadata:
    name: "Positron"
    description: "continuous integration for Positron"
    owner: "{{ event.head.user.email }}"
    source: "{{ event.head.repo.url }}"
tasks:
    - provisionerId: "{{ taskcluster.docker.provisionerId }}"
      workerType: "{{ taskcluster.docker.workerType }}"
      extra:
          github:
              env: true
              events:
                  - pull_request.opened
                  - pull_request.synchronize
                  - pull_request.reopened
                  - push
      payload:
          maxRunTime: 7200 # seconds (i.e. two hours)
          image: "mykmelez/docker-build-positron:latest"
          command:
              - "/bin/bash"
              - "--login"
              - "-c"
              - "tc-vcs checkout repo https://github.com/mozilla/gecko-dev $GITHUB_HEAD_REPO_URL $GITHUB_HEAD_BRANCH &&
                 cd repo/ && SHELL=/bin/bash ./mach build"
      metadata:
          name: "checkoutandbuild"
          description: "check out and build the repository"
          owner: "{{ event.head.user.email }}"
          source: "{{ event.head.repo.url }}"

Finally, I added TaskClusterRobot to the list of collaborators in the repo’s Settings > Collaborators & teams list, giving it Write permissions, so it can annotate pull requests with icons representing the state of the job, just like Travis does:

Screen Shot of GitHub PR with CI annotations

Et voilà, continuous integration for Positron!

(Once Positron starts running Node and Electron tests, I’ll add tasks to run those tests and report their results.)

Project Positron

Along with several colleagues, I recently started working on Project Positron, an effort to build an Electron-compatible runtime on top of the Mozilla technology stack (Gecko and SpiderMonkey). Mozilla has long supported building applications on its stack, but the process is complex and cumbersome. Electron development, by comparison, is a dream. We aim to bring the same ease-of-use to Mozilla.

Positron development is proceeding along two tracks. In the SpiderNode repository, we’re working our way up from Node, shimming the V8 API so we can run Node on top of SpiderMonkey. Ehsan Akhgari details that effort in his Project SpiderNode post.

In the Positron repository, we’re working our way down from Electron, importing Electron (and Node) core modules, stubbing or implementing their native bindings, and making the minimal necessary changes (like basing the <webview> element on <iframe mozbrowser>) so we can run Electron apps on top of Gecko. Eventually we aim to join these tracks, even though we aren’t yet sure exactly where the last spike will be located.

It’s early days. As Ehsan noted, SpiderNode doesn’t yet link the Node executable successfully, since we haven’t shimmed all the V8 APIs it accesses. Meanwhile, Positron supports only a tiny subset of the Electron and Node APIs.

Nevertheless, we reached a milestone today: the tip of the Positron trunk now runs the Electron Quick Start app described in the Electron tutorial. That means it can open a BrowserWindow, hook up a DevTools window to it (with Firefox DevTools integration contributed by jryans), and handle basic application lifecycle events. We’ve imported that app into the Positron repository as our “hello world” example.

Clone and build Positron to see it in action!

 

New Year, New Blogware

Four score and many moons ago, I decided to move this blog from Blogger to WordPress. The transition took longer than expected, but it’s finally done.

If you’ve been following along at the old address, https://mykzilla.blogspot.com/, now’s the time to update your address book! If you’ve been going to https://mykzilla.org/, however, or you read the blog on Planet Mozilla, then there’s nothing to do, as that’s the new address, and Planet Mozilla has been updated to syndicate posts from it.

Introducing PluotSorbet

PluotSorbet is a J2ME-compatible virtual machine written in JavaScript. Its goal is to enable users you run J2ME apps (i.e. MIDlets) in web apps without a native plugin. It does this by interpreting Java bytecode and compiling it to JavaScript code. It also provides a virtual filesystem (via IndexedDB), network sockets (through the TCPSocket API), and other common J2ME APIs, like Contacts.

The project reuses as much existing code as possible, to minimize its surface area and maximize its compatibility with other J2ME implementations. It incorporates the PhoneME reference implementation, numerous tests from Mauve, and a variety of JavaScript libraries (including jsbn, Forge, and FileSaver.js). The virtual machine is originally based on node-jvm.

PluotSorbet makes it possible to bring J2ME apps to Firefox OS. J2ME may be a moribund platform, but it still has non-negligible market share, not to mention a number of useful apps. So it retains residual value, which PluotSorbet can extend to Firefox OS devices.

PluotSorbet is also still under development, with a variety of issues to address. To learn more about PluotSorbet, check out its README, clone its Git repository, peruse its issue tracker, and say hello to its developers in irc.mozilla.org#pluotsorbet!

simplify asynchronous method declarations with Task.async()

In Mozilla code, Task.spawn() is becoming a common way to implement asynchronous operations, especially methods like the greet method in this greeter object:

let greeter = {
message: "Hello, NAME!",
greet: function(name) {
return Task.spawn((function*() {
return yield sendGreeting(this.message.replace(/NAME/, name));
}).bind(this);
})
};

Task.spawn() makes the operation logic simple, but the wrapper function and bind() call required to start the task on method invocation and bind its this reference make the overall implementation complex.

Enter Task.async().

Like Task.spawn(), it creates a task, but it doesn’t immediately start it. Instead, it returns an “async function” whose invocation starts the task, and the async function binds the task to its own this reference at invocation time. That makes it simpler to declare the method:

let greeter = {
message: "Hello, NAME!",
greet: Task.async(function*(name) {
return yield sendGreeting(this.message.replace(/NAME/, name));
})
};

With identical semantics:

greeter.greet("Mitchell").then((reply) => { ... }); // behaves the same

(And it avoids a couple anti-patterns in the process.)

Task.async() is inspired by ECMAScript’s Async Functions strawman proposal and C#’s Async modifier and was implemented in bug 966182. It isn’t limited to use in method declarations, although it’s particularly helpful for them.

Use it to implement your next asynchronous operation!

qualifications for leadership

I’ve been surprised by the negative reaction to Brendan’s promotion by some of my fellow supporters of marriage equality. Perhaps I take it too much for granted that Mozillians recognize the diversity of their community in every possible respect, including politically and religiously, and that the only thing we share in common is our commitment to Mozilla’s mission and the principles for participation.

Those principles are reflected in our Community Participation Agreement, to which Brendan has always shown fealty (since long before it was formalized, in my 15-year experience with him), and which could not possibly be clearer about the welcoming nature of Mozilla to all constructive contributors.

I know that marriage equality has been a long, difficult, and painful battle, the kind that rubs nerves raw and makes it challenging to show any charity to its opponents. But they aren’t all bigots, and I take Brendan at his word and deed that he’s as committed as I am to the community’s inclusive ideals (and the organization’s employment policies).

As Andrew Sullivan eloquently states in his recent blog post on Religious Belief and Bigotry:

“Twenty years ago, I was confidently told by my leftist gay friends that Americans were all anti-gay bigots and would never, ever back marriage rights so I should stop trying to reason them out of their opposition. My friends were wrong. Americans are not all bigots. Not even close. They can be persuaded rather than attacked. And if we behave magnanimously and give maximal space for those who sincerely oppose us, then eventual persuasion will be more likely. And our victory more moral and more enduring.”

I’m chastened to admit that I substantially shared his friends’ opinion twenty years ago. But I’m happy to realize I was wrong. And perhaps Brendan will one day do the same. Either way, he qualifies to be a leader at any level in the Mozilla community (and organization), as do the many other Mozilla leaders whose beliefs undoubtedly differ sharply from my own.

from Webapp SDK to r2d2b2g, Firefox OS Simulator, and the App Manager

A little over a year ago, on August 31, 2012, I brainstormed the outline of a “Webapp SDK”:

Webapp SDK

That outline was the genesis for the r2d2b2g experiment, which built the Firefox OS Simulator, whose initial version hit the web on September 14, 2012 and which has gone through numerous iterations since then as we evaluated various features to enhance app development.

And that experiment spurred Mozilla’s Developer Tools group, particularly its nascent App Tools team, to build Firefox’s new App Manager, which landed last month and was introduced today on Hacks!

Despite the twisty passage from experiment to product, that initial outline bears a surprising resemblance to the App Manager feature set. The Manager checks off three of the four features on the outline’s primary list—”start Gaia in B2G,” “package app,” and “test app in Gaia/B2G”—plus a few on its secondary list, like “debug from Firefox” and “test on mobile device,” with “edit manifest in GUI” well underway over in bug 912912. And the Simulator continues to provide B2G/Gaia via an easy-to-install addon that integrates with the Manager.

Like any good product of a successful experiment, however, the Manager’s reach has exceeded its progenitor’s grasp! So it also gives you access to pre-installed apps, lets you take screenshots of device/Simulator screens, and will doubtless continue to sprout handy features to make app development great.

So kudos to the folks who built it, and long live the App Manager!

fixing this morning’s mach OS X psutil bustage

If mach is broken in your mozilla-central clone on Mac OS X this morning:

08-23 10:20 > ./mach build
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'TCPS_ESTABLISHED'

  File "/Users/myk/Mozilla/central/python/mozbuild/mozbuild/mach_commands.py", line 293, in build
    from mozbuild.controller.building import BuildMonitor
  File "/Users/myk/Mozilla/central/python/mozbuild/mozbuild/controller/building.py", line 22, in <module>
    import psutil
  File "/Users/myk/Mozilla/central/python/psutil/psutil/__init__.py", line 95, in <module>
    import psutil._psosx as _psplatform
  File "/Users/myk/Mozilla/central/python/psutil/psutil/_psosx.py", line 48, in <module>
    _TCP_STATES_TABLE = {_psutil_osx.TCPS_ESTABLISHED : CONN_ESTABLISHED,

Then you’ve been bit by the fix for bug 908296. To resolve the bustage, run this command in your Hg clone:

hg status -in python/psutil | xargs rm

Or, if you’ve cloned the Git mirror, run this instead:

git clean -xf python/psutil