Node.js and Docker realigned

sporkIt’s not really a surprise, but after just over six months since the “forking” of both Node.js and Docker, the two different projects have ended up back in some sort of alignment. For Node.js, it was the reunification with io.js under the Node.js Foundation, which was officially launched under the Linux Foundation’s umbrella. The Node.js and io.js technical development is now driven by a technical committee and hopefully this will all work out well for all.

The Docker situation is a little more complex. There’s no big group hug like with Node.js. Instead, there’s an official middle ground, the Open Container Project. The announcement of a vendor-neutral (how can it be vendor neutral when it’s founded by vendors?) project to come up with containerisation technology basically sees Docker throw its specs and CoreOS throw its specs for containers into the same ring and see what comes out.

OCP says it’ll try and come up with a future spec independent of what’s layered on top of it, not associated with any project or vendor and be portable. So no, there won’t be a standard command set or management layer, there shouldn’t be any lock-ins and there probably will emerge as standard with a scope so small that it’ll end up as a tiny checkbox on a requirements list.

On the plus side, with that out of the way, there’s room for people to get innovating with the rest of the containerisation stack, which is where all the vendors are heading right now. That list is long too – Amazon, Cisco, EMC, Fujitsu, Google, IBM, Red Hat, VMWare and more. With the essential core in neutral hands, the game always moves on. As for the spec itself? Keep your eyes on the OCP’s Github Repository where they say they’ll have something by end of July.

Let’s hope that OCP keeps to its goals better than that other OCP, you know, the one that was building Delta City in the soon-to-be ruins of Old Detroit. That just didn’t work out well at all.

Docker 1.5 and Node.js Foundations

sporkBack in December we saw two community splits, one in the Docker community and one in the Node.js community. It’s time to look back at both those splits.

Docker 1.5 just landed with IPV6 support, read only containers, a stats API and CLI commands for streaming results and the ability to specify what Dockerfile to use when building. Good updates.

Now, the other bits – There’s also an “Open Image Spec” which isn’t so much a spec as a formal declaration of whats currently implemented. Thats a documentation +1 but open it isn’t . It seems to be a response to Rocket, which is a specification by design. It’s a start but there’s still the question of how transparent and inclusive the development of that spec will be.

The openness question is, I guess, supposed to be addressed by the announcement of a new organisational structure for developing Docker, which is a good thing if it comes together and works, but Docker is still Docker’s ball.

Node.js owner Joyent has announced a the establishment of a Node.js Foundation. Billed as bringing “Neutral and Open Governance” to Node.js. The “Foundation” will be established by Joyent, IBM, PayPal, Microsoft, Fidelity and the Linux Foundation. One assumes the Linux Foundation will be providing the umbrella organisation for the Node.js Foundation to work under as they have for other organisations.

But, as Bradley Kuhn points out this isn’t really a foundation as an charity that works for the public good; in the US that’s a 501(c)(3). The Linux Foundation and the proposed Node.js Foundation are not that kind of foundation – they are trade associations run for benefit of their members, 501(c)(6) organisations. That means you don’t “magically get a neutral home and open governance”.

We don’t have enough information to see how the new organisation will be run but the announcement is just an announcement and the details may contain more than trace amounts of devil. For the Node.js community, this is just a milestone in a long road and a parallel road marked io.js is running alongside it for now.

Forking brilliant – Node/IO.js and Docker/Rocket

spork

What’s up with Node: So there’s been a fork in Node.js land with the appearance of IO.js. A group of core contributing developers have lost patience with Joyent, the developmental home of Node.js, and have set out to accelerate the development of the Async-JavaScript server side platform. This is the world of open source where people can vote with their time and effort.

It’s easy to see both sides of the fork. Joyent want steady, stable development as they move towards a foundationed, open-sourced release. That progress has been guided from within Joyent, as is their right, but it has ended up with a situation where old code, like an unsupported version of the V8 JavaScript engine, is still actively used.

The forkers wanted to move things foward faster. Some had been involved in a light fork, Node-forward, which was designed to make the enhancements and then offer pull requests to the Node project. But that wasn’t working for them. According to one of the better know users of Node, the fork has been a relatively polite affair in itself and most of the noise surrounding it has come from outside the Node developer community.

Which makes it all the more likely that this fork is going to be a good thing for the generality of the Node community. It’ll push both sides to compete on quality and progress and with commitments to compatibility from the forkers, the door is still open for changes to be backported. Of course, it could all go off the rails. Right now, we get to look forward to January 13, when IO.js will release its first alpha.

What’s up with Docker: Over with Docker, another case of long time contributers starting their own project has popped up. This time it’s all about containers. Containers in Linux let you run multiple systems off the same kernel. The problem was that LXC (Linux Containers) were hard work to set up and manage. Enter Docker in 2013 with an easy to configure and deploy solution to that problem. This was great stuff, bringing containers to more than just the pioneers who’d been harnessing them quietly.

It quickly started catching on and CoreOS contributed to the development by dotCloud, the original Docker company which eventually became Docker Inc, because they saw a use for a de facto standard container within CoreOS making app depolyment easy.

Time passed and as Docker Inc needed to grow it started a process of adding more and management elements to their Docker offering. Some of this was undermining CoreOS as they just needed a well matured container format to integrate with their server Linux. They weren’t happy with where development in Docker was heading and that it was bringing a big technical and architectural debt with it.

So CoreOS started building Rocket. Rocket isn’t a fork though; CoreOS started from scratch releasing a prototype to Github and specs for review. They started from scratch because one of their problems is what the see as the monolithic approach in Docker which they feel is counter a good security model. So rather than Docker tools talking to a single process and letting that do all the work, Rocket tools do the work themselves.

The company already is committed to Docker integrated into CoreOS and isn’t dropping it but it seems it wants to get building the foundations of a more secure container platform now, not wait till there’s an incident which blows out confidence. Rocket will notionally be done when it provides enough to create, package and run containers, containers defined by a specification which the Rocket developers created first. They hope that the spec will evolve and be implemented by others, including Docker.

Thoughts: These are two interestingly different splits. Both are powered by the force that powers most open source – enlightened self-interest. Both have the capacity to enhance the ecosystem that they are splitting from. And both are being created by developers who are already vested and have contributed, and probably will still contribute, in the the platforms they are splitting from. These have the potential to be sporks, splendid forks, if all parties are able to take as much as they give. Six months from now, both splits should have full releases and positions should be soldifying. How these things look a year from now is going to be very illustrative for open source in general. Just let me pop it in my diary now…

Developer Catchup: POODLE, Tails, Docker, Redis and more

developercatchupPOODLE yips: In what was a glorious nail in the coffin of SSLv3, the POODLE vulnerability(PDF) made sure no one would trust SSLv3 again. The simple fix is to turn off SSLv3 where its used. The bug itself is bad in terms of cryptography, in that it gives an attacker a route to completely decode a stream that has been encrypted, but in practice its not as bad because the attacker has to be a man in the middle to get started. So, using SSLv3 from the open Wi-Fi at the fast food cafe, a bad thing. More worthwhile reading includes Imperial Violet’s explanation and Zmap.io’s guide to disabling SSLv3 in servers.

Chasing Tails: The Tails Live Linux distro, which tries its level best to be an bootable anonymous secure distro, has had an update to Tails 1.2. In the wake of the POODLE hole, it’s switched over to Tor Browser, dropping the IceWeasel, and that change also happens to close its POODLE vulnerability. There’s also Tor and kernel updates and various other minor changes. If you use it, just upgrade.

Docker tightens security: Docker 1.3 has landed, or more accurately Docker Engine 1.3. Highlight is digital signature verification of repositories of images, albeit as a tech preview of the feature. A production option also lets you set SELinux and AppArmor profiles from the command line. Other goodies include the ability to inject a process into a running Docker app so you can wake up a shell when you need to debug something, create and start commands for containers (on top of existing the all in one run command) and most usefully to me at least, shared directories on Mac OS X. The more interesting (as in get the popcorn) move from Docker is its partnering with Microsoft with a long term goal of making Docker run on Windows containers, not just on an a VM with Linux inside. Big challenge there as Microsoft have to basically get cgroups and more onto Windows Server.

Redis Clustered: The Redis key/value cache and store has pushed a release candidate for Redis 3.0.0 out. This is a rather important release as @antirez explains in his blog, it’s the first version with Cluster support, a long in-development feature, which has reached “minimum viable product” level and is stable enough for testing.

Quickies: 6to5 – turns JavaScript ES6 code into plain ES5 code which could be well useful. Asciicinema – lets you record and playback terminal sessions (and could be even better with audio – hint). On the to read list – Building Web Apps with Go – MIT licensed book based around Heroku use but lots of interesting content. And Whiteout Mail has gone open source – it’s all about accessible secure mail and has been in the works since 2013.

Developer Catchup – Easier docker on Mac, versioning made hard, old school Unix on the Pi and new school packaging for Meteor

developercatchupLet’s go fly a Kitematic: There’s plenty of command line tools for Docker and command line driven ways to run it on Mac OS X. The latter’s harder because you need to run a VM and load it with an image and… well there’s boot2docker to help but… Enter Kitematic which takes the previous tools and rolls them with a neat UI and some extra neat tricks to make it a lot easier to start playing with the idea. Among those tricks are things like automatically creating an [appname].dev DNS entry so you can quickly connect to your new apps when they are up and running. If you like to run GUI tools alongside terminal sessions on your Mac, you might want to give Kitematic a go.

Versioning wars: Yes, people are arguing on the Internet and this time its over versioning. Some years back, Semantic Versioning appeared and set out some rules for when to bump the major, minor and patch numbers in a version number to embue it with some meaning. While this works for libraries where the consumer is often another program, it works less well with code to be consumed by people. The argument starts on Underscore’s Github where breaking changes as fixes were causing friction over what the version should actually be. This spilled out onto Hacker News which lead to the suggestion that “Semantic Versioning Isn’t” and back to HN where people continued to disagree. But it did get Fear-Driver Versioning (ferver) and the idea of romantic versioning a moment in the sun. From what I see, SemVer works but it does require discipline and transparency from the developers and the consumers of that code. Still… Developers eh?… because those bike sheds won’t pick what colour they are going to be by themselves.

Old School on a Pi: Want to run old school stylee? We’re talking Unix V5 here. Matt Hoskins has updated his 2005 presentation on how to do this (think PDP emulation and similar) so you can now do it all on a Raspberry Pi. Read on to learn the true old ways of Unix.

Meteor updates packaging: Meteor’s a supercool Node.js framework for building apps and its steadily approaching version 1.0. Version 0.9.0 has just been released and along side various additions, there’s now a new Isobuild packaging system because in the Twenty-Tens, no framework or language is complete without it’s own packaging system. IsoBuild does not make ISO images, by the way. The Iso is short for “isomorphic”. The devs explain that Isomorphic JavaScript is, in their model, JavaScript that runs on client or server and that there wasn’t a build system that worked with both. Isobuild lets you do that with Isopacks that deliver the right code for client or server automatically. They’ve already got 1800 Isopacks and have their eye on mobile apps too in their planning with something going on for Cordova. Watch out for it. 1.0 is not far away.

Catchup: New Pi?, MeArm & JavaScript, Docker goes big, harder Dart, Breach

CodescalingCatchup
Hot Pi: First up, a hot rumour via Hackaday is that there’s a Raspberry B+ with 4 USB ports, no composite, a 40 pin GPIO port and other changes… is this real? We shall see, but the Pi has been aching for an update and even if this isn’t it, there’s a gap to be filled.

MeARM by JavaScript: Two of my favourite projects, the low cost MeARM robot ARM and the JavaScript running Espruino board have been brought together for a fun little remote control project for the ARM.

Big Docks: Google’s Kubernates project has drawn in a whole load of contributors. Kubernates is a cluster manager for containers. Now, [according to the announcement], IBM, Red Hat, Microsoft, Docker (and CoreOS, Mesosphere and SaltStack) have all signed up to develop various aspects of the platform. Could be interesting… it’s already interesting that Docker has at once become the centre of and just a component of bigger effort all at the same time.

Dart hardens: Dart, Google’s replacement for JavaScript and poor relative of Go in terms of adoption, is now ECMA-408. Whether getting the ECMA badge will push Dart forward or will the cross-browser-centic JavaScript community continue not bothering to look… we shall see, though Dart seems to lack the features that engender excitement and adoption so…

HTML Breached: Ever wanted an all JavaScript browser? The Breach developers did and are creating just such a thing.Node.js eventloop and wrapping the Chromium App API so that the top side of the browser is all HTML/JS. There might be some interesting code to come from it but its still early days.

Developer Catchup: Docker 1.1.0, Rust 0.11.0, Python 2.7.8 and Dropbox Go Libraries

developercatchupDocker 1.1.0: The first post 1.0 update for Docker is in and Docker 1.1.0 now has a .dockerignore mechanism for ignoring file changes, containers that now pause when a commit it happening (rather than messing them up), container log tailing, the ability to feed tar archives to docker build and other changes which should make life a bit easier and more predictable.

Rust 0.11.0: The latest Rust announcement for version 0.11.0 is about smoothing out the type system to allow for dynamically sized types and refactoring the standard libraries to allow for that. It means that language embedded elements like ~ and @ have become library types called Box and Gc that should make the language easier to understand. It all brings Rust 1.0 closer – by the end of the year is the current hope.

Python Updates: The start of July saw a Python update for various security issues – Python 2.7.8 updated the OpenSSL library, fixed mimetypes and UNC paths regressions and blocked an arbitrary code execution hole in CGIHTTPServer. There were also a number of core and library fixes detailed in the release notes. There was no corresponding update for Python 3.x though the CGIHTTPServer issue is scheduled to be fixed in Python 3.4.2 according to the in progress changelog.

Dropbox Go: Dropbox, a big Python user, has also been working with Go and has been moving its infrastructure to Go based code. In the process, they’ve written a lot of libraries to support that work and now they are open sourcing those Go libraries with a 3 clause BSD license. There’s code for caching, an improved error interface, a programmatic SQL statement builder, a memcahce client library, connection management and a space efficient hash library. And they will be doing it the right way – they’ve committed to using the public versions of the libraries inhouse (rather than maintain their own branch). You’ll find the documentation for all the libraries over at Godoc.org.

Graduation Snippets – Docker 1.0, RHEL 7.0, Firefox 30.0

linuxsnippets150

Docker 1.0: The Docker container management platform has hit version 1.0 though the major work had been done by version 0.11 – this is the project’s graduation, acknowledging its ready for production. The actual packaging and management software is going to be referred to as Docker Engine now as the announcement is also the signal for Docker (the company) to roll out 1.0 of Docker Cloud, a platform for sharing Docker packaged apps. Actual changes in 1.0 are things like a new COPY command and an improved ADD command for developers and the ability to pause and resume running containers, added XFS support and performance improvements in container removal. Make a note too. Posts 2375 and 2376 are now officially the HTTP and HTTPS ports for Docker. Docker has changes how people think about package and run applications on Linux and all it would need is for major players to adopt Docker and … oh Google’s added App Engine support for Docker to go with its Compute Engine support and then there’s…

RHEL 7.0: Red Hat has rolled out version 7.0 of its enterprise Linux and 7.0 is looking like a cracking release. Top of their highlights, containers and Docker support, XFS as default file system and new caching file systems (Btrfs is still experimental), systemd and new management components and more capabilities to work with Windows domains. The release notes as with all Red Hat releases are comprehensive and cover things like the switch to GNOME 3 on the desktop (while retaining a classic shell). RHEL 7 is the commercially supported upstream for other distributions, most notably the CentOS distribution which is working on its CentOS 7 release – no dates on that yet but it is the first test of the new relationship between Red Hat and CentOS.

Firefox 30.0: Thirty… As Firefox versioning heads out of the twenties, the [release of Firefox 30.0] has brought sidebar button for toolbar, support for GStreamer 1.0, command-E find selected, various developer changes and five critical and two high security fixes. Now it is thirty, Firefox is well on its way to settling down to a boring life where change is mostly about moving the furniture about and keeping an eye on the neighbours. The place to look for excitement is Mozilla’s Servo browser, being developed in Mozilla’s Rust language and is developing steadily.

Docker 0.9, Vagrant 1.5 and Xen 4.4 – Virtually Snippets

snippets03
Docker 0.9 unloads: Docker bumps its version number to Docker 0.9 and as it approaches version 1.0 makes a big change. Docker’s been pretty tightly tied to Linux Containers (LXC) technology to run applications packaged with it but in 0.9 there’s now execution drivers so the option to plug in any one of a range of isolation systems is now available. “OpenVZ, systemd-nspawn, libvirt-lxc, libvirt-sandbox, qemu/kvm, BSD Jails, Solaris Zones, and even good old chroot” are on Docker’s planned list with more to come from various projects. There’s also a new libcontainer which lets Docker plug straight into the Linux kernel to control things – this Go library is likely to see a lot of use outside of Docker too as it wraps up container configuration into a neat JSON specified bundle. Next stop for Docker is a production quality 0.10 which will serve as a release candidate for 1.0. Its lively down at the docks.

Vagrant 1.5 roams out: The developer environment manager Vagrant has been updated too. The new Vagrant 1.5 has added a sharing system to make collaboration easier, versioning for boxes, rsync and smb sync’d folders and Hyper-V support. Simpler SSH authentircation setup, a reworked plugin manager and support for Funtoo, NetBSD and TinyCore Linux as guests wrap out the wedge of features in this release. Alongside the release is the announcement of Vagrant Cloud, a hosted box sharing service built to use Vagrant 1.5’s sharing functions.

Xen 4.4 meditates: Meanwhile, the other Linux virtualisation platform, Xen, has made the first release on its aspirational six month cycle (taking 8 months in this case). The announcement for Xen 4.4 highlights an improved libvirt/libxl interface for better integration with VM managers or cloud platforms, a more flexible event channel interface allowing for over tens of thousands of guests and a rapidly maturing ARM port now with a stable ABI going forwards. There’s also a ‘tech preview’ of nested virtualisation on Intel.

Notified by mqttwarn, better Docker images, emulating a ship computer and more – Snippets

snippets03

mqttwarn: Don’t want to run Node-RED but do want to route MQTT messages around? Jan-Piet Mens may have the application for you in the Python based mqttwarn, a pluggable framework which can subscribe to many MQTT topics and send them on to files, other MQTT systems, Twitter, SMTP, Redis, SQLite and Mac OS X notifications. He explains that the instigator for this was being introduced to Pushover, an iOS and Android notification app… which is now also supported by mqttwarn.

Docker image: The Phusion developers have launched baseimage-docker, a modified Ubunutu 12.04 LTS image which addresses problems they see when using Docker as more of a container for long running processes and systems – the correctness of that position is up for debate though. The mods include a corrected init, fixed apt issues, and running syslog-ng, cron and an ssh server and also includes “runit” to replace Upstart and a “setuser” command all in a 6MB of memory footprint.

SPARC in QEMU in Ship: Old, but gold, a short article on how a group of engineers replaced an old SPARCstation-5-like VME form factor system with a QEMU emulation of the hardware to keep a 3000 ton ship navigating.

Worth a read: Java 8 SE’s Date and Time quickly explained, one Java coder’s rationale and practices when using Java, Crunching NBA data with Postgresql and aggregates.