Developer Catchup: Synchronous Node, Serviced Polyfills, Sparks Sparked, Tangrams Mapped and SHAaaaaaa!

developercatchupNode.js synchronously: Node.js is sweet if you can adapt to the asynchronous model of start thing, say what you want to do when its done, do everything else anyway. Good for web request handling but bleh for trying to emulate a shellscript. Turns out that in Node.js 0.12 (coming soon? anyone? Bueller?) we get synchronous child processes to now you can run that curl or find or whatever and just wait till its returned with its results. The folks at Strongloop have written about these synchronous child process methods and how they make writing command line utilities in Node easier. Check it out Noders.

Serviced Polyfills: Polyfills fill gaps in browser functionality and standards compliance. The older the browser, the more Polyfill you need to fill the gaps and the newer, the less. But it gets hard working out how much Polyfill you are going to need. Fear not, as Samuel Giles at FT Labs has an answer, “Polyfills as a Service“. Add a simple script tag pointing at a source from the content delivery network to your pages and whatever browser views your page, it gets the polyfill it needs. This is because the system sniffs the browser agent and works out the best set of polyfill based on that. Neat idea, potentially very handy – and you can run your own private version if you need to.

Spark sparks: Apache Spark just got a 1.1 release. Spark is Hadoop data processing engine which can run on YARN-based Hadoop clusters or in standalone mode. Spark 1.1 improves the performance (and they already say they are up to 100 times faster than Hadoop MapReduce) and has SQL layer enhancements. 1.1 also adds more statistical functions, can take steaming data fromAmazon Kinesis and pull data from Apache Flume and more. If your into clusters and data crunching and haven’t looked at Spark, you might want to look into it.

Tangram Mapping: Do you want to render cool 2D and 3D maps? Check out Tangram, a Mapping Library then as it is building out from a WebGL implementation to other OpenGL platforms to make oodly cool dynamic map renders. Very slick.

SHAaaaaaa!: We mention the Google sunsetting of SHA-1 the other week. If you were unsure why this was important, can we send you off to Why Google is Hurrying the Web to Kill SHA-1 which explains why it all and includes a brief history of collision attacks in the wild.

Node-RED updated, Hadoop 2.3.0 out, NetBeans 8.0 RCs and Skrollr scrolls – Snippets

Node-RED updated: The most excellent graphical UI for connecting the Internet of Things (or just things in general), Node-RED has been updated to version 0.6. The announcement notes the process of separating the admin and server authentication to make deployment more robust has begun. Node-RED has nodes that accept HTTP connections and has a HTTP admin front end and previously these were all under one HTTP authentication mechanism – now the UI and nodes are more separate with the option to set a user/password for each. There’s some UI changes like a search filter for the palette of available nodes and easier flow importing by just dragging and dropping JSON onto the UI. In the node-red-nodes library, they’ve added Postgres, Amazon DynamoDB and Emoncms for more connections. There’s also fixes for the MQTT keepalive handing, an added socket timeout settings for TCP sockets and support for all 17 pins of WiringPi. More generally, there’s a range generating node now and the inject node can send empty payloads if needed. Finally, the MongoDB node now can send a user name and password – something I found I needed when writing this for MongoHQ.

Hadoop 2.3.0 released: In case you missed it, version 2.3.0 of the Apache Hadoop project got a release. The release notes list all the details. The short version is this is mostly about HDFS, the distributed file system and the changes include the ability to class the storage under HDFS so you can make tradeoffs between say spinning media, SSDs and memory, an ability to explicitly cache files or directories under HDFS (and local zero-copy reading from the cache) and the use of HDFS and YARN to simplify deploying MapReduce code. Hortonworks has a good writeup which also looks forward to Hadoop 2.4.0 with HDFS ACLs and rolling upgrades.

NetBeans 8 gets an RC: The NetBeans IDE has hit release candidate for 8.0. This is the version that will include JDK 8 support in the editor, Java SE Embedded and Java ME Embedded support, PrimeFaces code generators, AngularJS navigation and code completion, PHP 5.5 support and much more. There’s a summary in the announcement, a lot more detail in the New and Noteworthy wiki page and a pencilled in release date of mid-April.

Skrollr scrolls in: Recently spotted – Skrollr, a compact parallax scrolling and scrolling animation library for all your Webtml5.0 styled sites including the ability to “scale, skew and rotate the sh** out of any element”.

Facebook’s Conceal, Callback hell and a listening Pi – Snippets


Facebook’s Conceal revealed: Facebook have open sourced Conceal, a library for encrypting files on Android devices. The company uses the library for encrypting data that its apps store on SD cards. It uses pre-selected OpenSSL algorithms, picked for efficient memory management and speed, and gets the library down to 85KB by not trying to be a general purpose crypto kit. An interesting bit of pragmatism which means Facebook’s apps can happily encrypt on low-end Android devices, Conceal is available under a BSD licence with its source on GitHub.

Callback hell: Callbacks in Node.js can get pretty gnarly if you do everything with inline anonymous functions. This blog posting from Strongloop is a handy summary of some of the ways, from nesting, modularisation, async, promises and (soon to come to Node) ES6 generators. So callback, much techniques.

A Pi that listens: Meanwhile, a nice little Instructable covers converting an old bakelite Televox intercom into a voice controlled personal assistant by popping a Raspberry Pi inside.. and a sound board… and some software of course… It’ll probably be quite hard to find another fine bakelite intercom, but the rest of the projects a good starting point for assembling your own style of smart box…

Open source is a development process too

A recent report suggests that Microsoft may open source its M# language upon its as-yet-to-be determined release date. M# is a language being developed in tandem with Midori, a research operating system Microsoft is also developing and, apparently has been developing since 2008. In many ways this plan to open source on release is an excellent demonstration of a classic misunderstanding of open source and the audiences for open source. When Microsoft people talk about releasing M# and open sourcing it, it strikes me that the company still doesn’t get open source as a development process and still sees open source as a deliverable, as something you do to clear some check boxes or unlock a market.

The “open source as a deliverable” approach is a perfectly valid way of creating an open source software if you assume that for the thing to catch on it needs to be ready to roll. This can be because you perceive the people who will use the code as predominantly end users rather than developers because you are making software for them to use in the form of, say, an application. If you believe that at release, end users will begin using the application, and that as a secondary effect, developers with a specific interest or other motivation use the open source code as an opportunity to develop and contribute, and, all going well, the project will develop momentum, then yes, it is a valid way of open sourcing a project.

But that doesn’t hold true for development tools and languages; by definition your users will be developers, they are just not yet developing of your tool or your language. Every user is a potential developer for your project or a source of feedback on design. No user of tools is without an opinion on their tools or a desire to craft a better tool. If you go down the “open source deliverable” release path though you’ve already not made use of that resource and are sitting at version 1.0 having internalised all the development processes within your organisation. When the release does occur you’ll now have all your internal processes grinding against outside contributions.

It gets worse though as the longer you take to get to 1.0, the more established those internal processes are, the louder and harsher the grinding. And all of that is assuming developers even want to get on board the open sourced language or tool you’ve released – they can easily come to a conclusion that if you’ve made it to 1.0, plans for 2.0 are already nailed down and if developers want to do anything meaningful setting a direction for development, version 1.0 is more a flag for them to fork the code base and do their own thing.

For development tools and languages, open source isn’t just a way of delivering code to users and ticking a check box. At its most powerful, it’s the core of an open development process which creates an engaged ecosystem of developers and tunes your internal organisation to working with that ecosystem. “Release early…” is core to that process; developers understand that releasing an open source version 0.1 of something isn’t saying its is anywhere near complete but releasing a 0.1 does let people look in on what is being created and if your code, and associated documentation and writings, convey what you are planning in a compelling way, it will bring people onboard. And that will provide the seed for a meaningful open source development process.

Some might ask won’t opening the development process early like that disclose other proprietary plans that your organisation may have. The simple fact is that if your work is closely enmeshed with those plans such that it is visible in the code of your project, your project probably isn’t a good candidate for being open source anyway.

Be aware from the start that this doesn’t all work like magic and you will have to drive your vision forward with the project and actively engage with the outside developers and users, but that friction can be a powerful constructive friction in a process which can scale up over the time it takes to reach version 1.0. By the time you reach 1.0, you will already have a community of developers around your project… and that’s far more potent than just dropping an open source deliverable at release.

It’s probably too late for Microsoft’s M# to benefit from this approach, but if you are planning on creating a open source tool or language or something else aimed at a development audience, remember to open your development process too.

OIN and OpenStack, X and Security, Docker and Mac OS X – Snippets


  • Linux patent pool now covers clouds: By deftly expanding the list of packages it considers part of the Linux ecosystem to include OpenStack and Red Hat’s OpenShift Origin, the Open Invention Network in now including the cloud computing platforms as part of its protective cross-licence network. The change is set to take effect in March Companies can join OIN by dint of agreeing not to pursue patent litigation against other companies with respect to that package list. In return, they get a royalty free licence to the OIN’s patent pool. It will be interesting to see how well the OIN’s new safe harbour works for cloud providers.

  • X Security – It ain’t good: At the Chaos Computer Club’s 30th Congress, one presentation took on the issue of the security of X Window System in terms of its implementation in the code. Ilja van Sprundel has been working through the code over the past year and after finding 80 bugs in the client code, he’s gone on to just submit 120 bugs for the server side and he says he’s far from finished.

  • Docker on OS X: Mac OS X doesn’t have containers so running Docker natively is a no-no. The advice has been to setup a Linux VM with Vagrant, ssh into that and run Docker on there. But some people wanted a bit more simplicity and came up with Docker-osx which is a shell script which lets you run docker commands. It uses VirtualBox and Vagrant, automatically configuring the VM if needed. From then on, running “docker docker-command” sees the command automatically passed through to docker in the VM. There’s also two “new” docker commands, “halt” (to stop the VM) and “ssh” (to open a terminal session). A simple enough script but rather handy.

Ruby 2.1 rolls out a performance push

Ruby_logoRuby 2.1 has been released on Christmas day and is billed as offering “speedup without severe incompatibilities”. The performance boost is down to a new method cache in the VM and a new generational garbage collection system. The old method cache was cleared eacg time a new method was defined but now only that cache damage has been tracked down and reduced and a future of a more optimal larger cache has been opened up.

The generational GC was added as part of an optimisation of Ruby’s object management which also adds hooks for allocation and deallocation – there’s a deeper look into these changes in Koichi Sasada’s presentation(PDF) from RubyConf 2013. The GNU Multiple Precision Arithmetic Library is also now used to accelerate Bignum calculations.

Beyond the runtime changes, there are some language changes such as a new method for Array – to_h – to turn key/value pairs into a hash, a new syntax for rational numbers so that 1//2 is the same as Rational(1,2) and the addition of an explaining cause to Exceptions, but they seem to be few and far between. A full list is in the NEWS file which also notes any compatibility issues. Source for Ruby 2.1 is available now for download and will most probably be appearing in various binary forms over the coming weeks.

The performance improvements, although useful, will not help the perception that, outside of the CRUD web applications where Ruby and Rails made their mark, the native form of the language benchmarks as one of the slower languages out there. They will at least though help Ruby when it comes to a developer deciding which tradeoffs they want to make.

Enlightenment 0.18 lit, FreeNAS 9.2 released and Java 8 brews – Snippets


  • Enlightenment Updated: The Enlightenment/EFL window manager/libraries/desktop has been updated to version 0.18.0, just a year after the long silence that led up to the release of Englightenment 0.17.0. A full list of bug fixes and improvements is in the NEWS file for the release – compositing has been merged into the core, ten crashing bugs have been fixed and modules for music control, bluetooth, DBus application menus and compositing control have been added. Downloads are available from the project’s site.

  • FreeNAS 9.2 goes final: The network storage platform FreeNAS has been updated too with over 260 fixes and a rebasing on FreeBSD 9.2. The developers say it should sport improved performance, especially with encryption if appropriate hardware is available, and be more able to cope with higher loads. The release notes offer further details – Items I like on the list are full registration of all services through multicast DNS using Avahi, which should make a server much easier to just drop into a network, and the addition of a REST API for FreeNAS for remote control.

  • Java 8’s final draft: The final draft for the Java 8 specification is now available and this is going to be the reference document to the changes being made in Java 8, due in March 2014. Lambda expressions, new date and time APIs and type annotations are referenced with pointers out to the various JSRs to where Java will be next year.