Developer Catchup: Synchronous Node, Serviced Polyfills, Sparks Sparked, Tangrams Mapped and SHAaaaaaa!

developercatchupNode.js synchronously: Node.js is sweet if you can adapt to the asynchronous model of start thing, say what you want to do when its done, do everything else anyway. Good for web request handling but bleh for trying to emulate a shellscript. Turns out that in Node.js 0.12 (coming soon? anyone? Bueller?) we get synchronous child processes to now you can run that curl or find or whatever and just wait till its returned with its results. The folks at Strongloop have written about these synchronous child process methods and how they make writing command line utilities in Node easier. Check it out Noders.

Serviced Polyfills: Polyfills fill gaps in browser functionality and standards compliance. The older the browser, the more Polyfill you need to fill the gaps and the newer, the less. But it gets hard working out how much Polyfill you are going to need. Fear not, as Samuel Giles at FT Labs has an answer, “Polyfills as a Service“. Add a simple script tag pointing at a source from the content delivery network to your pages and whatever browser views your page, it gets the polyfill it needs. This is because the system sniffs the browser agent and works out the best set of polyfill based on that. Neat idea, potentially very handy – and you can run your own private version if you need to.

Spark sparks: Apache Spark just got a 1.1 release. Spark is Hadoop data processing engine which can run on YARN-based Hadoop clusters or in standalone mode. Spark 1.1 improves the performance (and they already say they are up to 100 times faster than Hadoop MapReduce) and has SQL layer enhancements. 1.1 also adds more statistical functions, can take steaming data fromAmazon Kinesis and pull data from Apache Flume and more. If your into clusters and data crunching and haven’t looked at Spark, you might want to look into it.

Tangram Mapping: Do you want to render cool 2D and 3D maps? Check out Tangram, a Mapping Library then as it is building out from a WebGL implementation to other OpenGL platforms to make oodly cool dynamic map renders. Very slick.

SHAaaaaaa!: We mention the Google sunsetting of SHA-1 the other week. If you were unsure why this was important, can we send you off to Why Google is Hurrying the Web to Kill SHA-1 which explains why it all and includes a brief history of collision attacks in the wild.

Node-RED updated, Hadoop 2.3.0 out, NetBeans 8.0 RCs and Skrollr scrolls – Snippets

Node-RED updated: The most excellent graphical UI for connecting the Internet of Things (or just things in general), Node-RED has been updated to version 0.6. The announcement notes the process of separating the admin and server authentication to make deployment more robust has begun. Node-RED has nodes that accept HTTP connections and has a HTTP admin front end and previously these were all under one HTTP authentication mechanism – now the UI and nodes are more separate with the option to set a user/password for each. There’s some UI changes like a search filter for the palette of available nodes and easier flow importing by just dragging and dropping JSON onto the UI. In the node-red-nodes library, they’ve added Postgres, Amazon DynamoDB and Emoncms for more connections. There’s also fixes for the MQTT keepalive handing, an added socket timeout settings for TCP sockets and support for all 17 pins of WiringPi. More generally, there’s a range generating node now and the inject node can send empty payloads if needed. Finally, the MongoDB node now can send a user name and password – something I found I needed when writing this for MongoHQ.

Hadoop 2.3.0 released: In case you missed it, version 2.3.0 of the Apache Hadoop project got a release. The release notes list all the details. The short version is this is mostly about HDFS, the distributed file system and the changes include the ability to class the storage under HDFS so you can make tradeoffs between say spinning media, SSDs and memory, an ability to explicitly cache files or directories under HDFS (and local zero-copy reading from the cache) and the use of HDFS and YARN to simplify deploying MapReduce code. Hortonworks has a good writeup which also looks forward to Hadoop 2.4.0 with HDFS ACLs and rolling upgrades.

NetBeans 8 gets an RC: The NetBeans IDE has hit release candidate for 8.0. This is the version that will include JDK 8 support in the editor, Java SE Embedded and Java ME Embedded support, PrimeFaces code generators, AngularJS navigation and code completion, PHP 5.5 support and much more. There’s a summary in the announcement, a lot more detail in the New and Noteworthy wiki page and a pencilled in release date of mid-April.

Skrollr scrolls in: Recently spotted – Skrollr, a compact parallax scrolling and scrolling animation library for all your Webtml5.0 styled sites including the ability to “scale, skew and rotate the sh** out of any element”.

Facebook’s Conceal, Callback hell and a listening Pi – Snippets


Facebook’s Conceal revealed: Facebook have open sourced Conceal, a library for encrypting files on Android devices. The company uses the library for encrypting data that its apps store on SD cards. It uses pre-selected OpenSSL algorithms, picked for efficient memory management and speed, and gets the library down to 85KB by not trying to be a general purpose crypto kit. An interesting bit of pragmatism which means Facebook’s apps can happily encrypt on low-end Android devices, Conceal is available under a BSD licence with its source on GitHub.

Callback hell: Callbacks in Node.js can get pretty gnarly if you do everything with inline anonymous functions. This blog posting from Strongloop is a handy summary of some of the ways, from nesting, modularisation, async, promises and (soon to come to Node) ES6 generators. So callback, much techniques.

A Pi that listens: Meanwhile, a nice little Instructable covers converting an old bakelite Televox intercom into a voice controlled personal assistant by popping a Raspberry Pi inside.. and a sound board… and some software of course… It’ll probably be quite hard to find another fine bakelite intercom, but the rest of the projects a good starting point for assembling your own style of smart box…

Open source is a development process too

A recent report suggests that Microsoft may open source its M# language upon its as-yet-to-be determined release date. M# is a language being developed in tandem with Midori, a research operating system Microsoft is also developing and, apparently has been developing since 2008. In many ways this plan to open source on release is an excellent demonstration of a classic misunderstanding of open source and the audiences for open source. When Microsoft people talk about releasing M# and open sourcing it, it strikes me that the company still doesn’t get open source as a development process and still sees open source as a deliverable, as something you do to clear some check boxes or unlock a market.

The “open source as a deliverable” approach is a perfectly valid way of creating an open source software if you assume that for the thing to catch on it needs to be ready to roll. This can be because you perceive the people who will use the code as predominantly end users rather than developers because you are making software for them to use in the form of, say, an application. If you believe that at release, end users will begin using the application, and that as a secondary effect, developers with a specific interest or other motivation use the open source code as an opportunity to develop and contribute, and, all going well, the project will develop momentum, then yes, it is a valid way of open sourcing a project.

But that doesn’t hold true for development tools and languages; by definition your users will be developers, they are just not yet developing of your tool or your language. Every user is a potential developer for your project or a source of feedback on design. No user of tools is without an opinion on their tools or a desire to craft a better tool. If you go down the “open source deliverable” release path though you’ve already not made use of that resource and are sitting at version 1.0 having internalised all the development processes within your organisation. When the release does occur you’ll now have all your internal processes grinding against outside contributions.

It gets worse though as the longer you take to get to 1.0, the more established those internal processes are, the louder and harsher the grinding. And all of that is assuming developers even want to get on board the open sourced language or tool you’ve released – they can easily come to a conclusion that if you’ve made it to 1.0, plans for 2.0 are already nailed down and if developers want to do anything meaningful setting a direction for development, version 1.0 is more a flag for them to fork the code base and do their own thing.

For development tools and languages, open source isn’t just a way of delivering code to users and ticking a check box. At its most powerful, it’s the core of an open development process which creates an engaged ecosystem of developers and tunes your internal organisation to working with that ecosystem. “Release early…” is core to that process; developers understand that releasing an open source version 0.1 of something isn’t saying its is anywhere near complete but releasing a 0.1 does let people look in on what is being created and if your code, and associated documentation and writings, convey what you are planning in a compelling way, it will bring people onboard. And that will provide the seed for a meaningful open source development process.

Some might ask won’t opening the development process early like that disclose other proprietary plans that your organisation may have. The simple fact is that if your work is closely enmeshed with those plans such that it is visible in the code of your project, your project probably isn’t a good candidate for being open source anyway.

Be aware from the start that this doesn’t all work like magic and you will have to drive your vision forward with the project and actively engage with the outside developers and users, but that friction can be a powerful constructive friction in a process which can scale up over the time it takes to reach version 1.0. By the time you reach 1.0, you will already have a community of developers around your project… and that’s far more potent than just dropping an open source deliverable at release.

It’s probably too late for Microsoft’s M# to benefit from this approach, but if you are planning on creating a open source tool or language or something else aimed at a development audience, remember to open your development process too.

OIN and OpenStack, X and Security, Docker and Mac OS X – Snippets


  • Linux patent pool now covers clouds: By deftly expanding the list of packages it considers part of the Linux ecosystem to include OpenStack and Red Hat’s OpenShift Origin, the Open Invention Network in now including the cloud computing platforms as part of its protective cross-licence network. The change is set to take effect in March Companies can join OIN by dint of agreeing not to pursue patent litigation against other companies with respect to that package list. In return, they get a royalty free licence to the OIN’s patent pool. It will be interesting to see how well the OIN’s new safe harbour works for cloud providers.

  • X Security – It ain’t good: At the Chaos Computer Club’s 30th Congress, one presentation took on the issue of the security of X Window System in terms of its implementation in the code. Ilja van Sprundel has been working through the code over the past year and after finding 80 bugs in the client code, he’s gone on to just submit 120 bugs for the server side and he says he’s far from finished.

  • Docker on OS X: Mac OS X doesn’t have containers so running Docker natively is a no-no. The advice has been to setup a Linux VM with Vagrant, ssh into that and run Docker on there. But some people wanted a bit more simplicity and came up with Docker-osx which is a shell script which lets you run docker commands. It uses VirtualBox and Vagrant, automatically configuring the VM if needed. From then on, running “docker docker-command” sees the command automatically passed through to docker in the VM. There’s also two “new” docker commands, “halt” (to stop the VM) and “ssh” (to open a terminal session). A simple enough script but rather handy.

Ruby 2.1 rolls out a performance push

Ruby_logoRuby 2.1 has been released on Christmas day and is billed as offering “speedup without severe incompatibilities”. The performance boost is down to a new method cache in the VM and a new generational garbage collection system. The old method cache was cleared eacg time a new method was defined but now only that cache damage has been tracked down and reduced and a future of a more optimal larger cache has been opened up.

The generational GC was added as part of an optimisation of Ruby’s object management which also adds hooks for allocation and deallocation – there’s a deeper look into these changes in Koichi Sasada’s presentation(PDF) from RubyConf 2013. The GNU Multiple Precision Arithmetic Library is also now used to accelerate Bignum calculations.

Beyond the runtime changes, there are some language changes such as a new method for Array – to_h – to turn key/value pairs into a hash, a new syntax for rational numbers so that 1//2 is the same as Rational(1,2) and the addition of an explaining cause to Exceptions, but they seem to be few and far between. A full list is in the NEWS file which also notes any compatibility issues. Source for Ruby 2.1 is available now for download and will most probably be appearing in various binary forms over the coming weeks.

The performance improvements, although useful, will not help the perception that, outside of the CRUD web applications where Ruby and Rails made their mark, the native form of the language benchmarks as one of the slower languages out there. They will at least though help Ruby when it comes to a developer deciding which tradeoffs they want to make.

Enlightenment 0.18 lit, FreeNAS 9.2 released and Java 8 brews – Snippets


  • Enlightenment Updated: The Enlightenment/EFL window manager/libraries/desktop has been updated to version 0.18.0, just a year after the long silence that led up to the release of Englightenment 0.17.0. A full list of bug fixes and improvements is in the NEWS file for the release – compositing has been merged into the core, ten crashing bugs have been fixed and modules for music control, bluetooth, DBus application menus and compositing control have been added. Downloads are available from the project’s site.

  • FreeNAS 9.2 goes final: The network storage platform FreeNAS has been updated too with over 260 fixes and a rebasing on FreeBSD 9.2. The developers say it should sport improved performance, especially with encryption if appropriate hardware is available, and be more able to cope with higher loads. The release notes offer further details – Items I like on the list are full registration of all services through multicast DNS using Avahi, which should make a server much easier to just drop into a network, and the addition of a REST API for FreeNAS for remote control.

  • Java 8’s final draft: The final draft for the Java 8 specification is now available and this is going to be the reference document to the changes being made in Java 8, due in March 2014. Lambda expressions, new date and time APIs and type annotations are referenced with pointers out to the various JSRs to where Java will be next year.

Node-RED’s cool GUI for the Internet of Things

Node-RED and a quick IRC bot flow
Node-RED and a quick IRC bot flow
The latest version, 0.5.0 of IBM’s Apache licensed, incredibly useful and very cool Node-RED has landed but before going further, I suspect a lot of readers will want to know what Node-RED is.

There’s usually a lot of connecting of things involved with making the Internet of Things do something useful. Whether it be detecting messages on Twitter, listening to IRC, watching a Websocket or grabbing a web page, each source then needs to be processed and if required make something happen. Now, you can write a lot of code to do that or you can check out Node-RED. Billing itself as a “Visual tool for connecting the Internet of Things”, Node-RED is built atop of Node.js and offers a graphical world of IoT building blocks in the browser for you to wire up as needed and test.

Those building blocks, nodes in Node-RED, start with the simple, inject to send something, function to process it using JavaScript and debug to see what is being sent. They then build up to more powerful capabilities such as Http requests, MQTT subscription and publication, WebSocket listening and writing, tcp and udp comms and sentiment analysis. To connect up to social networks, nodes for Twitter and IRC listening and publishing are included. There’s also file storage, logical switches, data manipulation and delay nodes and, for the brave, even a node which will exec a process on the system.

Each node can be placed, configured and its outputs connected to other nodes inputs in the Node-RED GUI, and with the click of the Deploy button, put into action. New nodes can be created and plugged into the system too and there’s a repository of user created extra nodes available.

But, you say, where’s the Internet of Things in this? Well, Node-RED is able to work with Arduinos (connected via USB to host computers over serial or Firmata protocols) or run directly on Raspberry Pi (with GPIO and wiring-Pi modules) and the BeagleBone Black (with BoneScript access). And, obviously, you can write your own plugin nodes to connect up whatever hardware or devices you need to access.

Node-RED is a very capable tool and worth adding to your toolkit. For example, while writing this I was also prototyping an IRC bot which did basic sentiment analysis and commented in the channel. Why not give it a go over the coming holidays? You can download Node-RED from the website or you can find it on GitHub. Documentation including a quick tutorial on creating flows, along with instructions on writing function nodes, creating new nodes, embedding Node-RED into existing applications and running it with Arduino, Raspberry Pi and BeagleBone Black.

The latest version, which we mentioned at the start, has new visuals for showing the deployment state of nodes and handles the idea of “unknown nodes” visually when importing a flow from someone who’s used nodes you haven’t got yet. There’s also new user and direct message tracking in the Twitter node, session aware TCP and WebSocket nodes, enhancements to the MQTT node for authentication and client ID, an “otherwise” option in the Switch node, selectable data delimiters for the serial node and a HTTP Request node that follows 301s. The contributed nodes now include a Snapchat node and a Phillips Hue mode.

The Node-RED developers are now looking at making new nodes installable with npm, Node.js’s package manager, and tackling the separation of the administration UI from the runtime so it’s more easily deployed into future production scenarios.

Fedora 20, Meteor 0.7.0 and hacked Linux servers examined – Snippets


  • Fedora 20 arrives: The latest Fedora has arrived, making it into 2013 and looking pretty good. There’s oodles of changes too. Desktop users will find GNOME 3.10 is the default desktop but there’s also Cinnamon 2.0 and Enlightment available, along with the latest KDE 4.11, MATE and others. Under the hood, system administrators will find syslog gone, replaced by journald, and experimental SSD caches, while developers are getting a GUI on Fedora’s DevAssistant, updated Perl, boost, glibc and Ruby 2 with Rails 4. The full release notes will guide you around. We’ve been tracking 20 since alpha, running it on machines here and its been working well – the one thing we haven’t checked out is Fedora 20 on ARM given ARM is now a primary architecture for the distribution. Download Fedora 20 in all its forms from the project’s download page.

  • Meteor gets update smarts: The latest release of the platform for web applications has moved to a smarter way of working out database changes. Meteor 0.7 changes how changes in the database are discovered, away from polling the db and creating a diff and to a technique called oplog tailing – consuming the underlying MongoDB operations log and using it to reduce the queries that have to go to the database. There are caveats, most notably, in production you’ll need a MongoDB server configured as a replica. More details on this and other changes in 0.7.0 are in the release notes.

  • Hacking Linux Servers: Ars Technica has an article on how a security researcher documented the exploitation of a Linux server with PHP holes and a perlbot. It’s a reminder that attack tools for taking on Linux servers are no longer obscure or complex things and even a script kiddie can do real damage. Old holes do persist in the wild and every old, fixed hole is ready for exploiting. Now, more than ever, keeping your servers up to date with security fixes is essential.

Firefox 26, Netflix’s Suro, Vagrants and Dockers and Websockets for all – Snippets


  • Firefox 26 digs in: Today we’ll see the release of Firefox 26, latest in the overly regular Firefox release cycle. From the (currently beta) release notes, we can see the big changes. All but the Flash plug-in are now click-to-play by default, Windows users can update their Firefox without having to write into the Firefox folders, the password manager can handle password fields generated by scripts and on Linux, if the installed gstreamer can handle h264, so can Firefox. A couple of fixes, some developer enhancements and thats about it. There’s also a Firefox for Android update due today. The release notes note some performance improvements, the same password manager enhancement and some fixes. The developer page for Firefox 26 covers changes of interest to developers in more detail. Firefox 26 will be turning up in updates and for download later today.

  • Netfix’s Suro goes open: From the people who brought you a cloud full of monkeys… Netflix’s latest open source release is Suro, an application monitoring system used by the video stream vendor to track the behaviour of their Amazon AWS deployed applications. Originally based on Apache Chukwa and adapted to fit Netflix’s demands, Suro pulls the company’s monitoring data from the various app clusters and pushes it to S3 (for Hadoop based analytics), to Apache Kafka (and on to Storm, Amazon ElasticSearch and Druid and to other event processors. There’s a lot more detail in the announcement including in production stats and how the pipeline is used to analyse errors.

  • Vagrant meets Docker: The latest update to Vagrant, version 1.4 has been announced and the big improvement in system that has traditionally been used to create automatically reproducible development environment is the addition of Docker support. The Docker provisioner can install Docker and then lets Vagrant cirtual machine pull and configure Docker containers within it. There’s also some enhancements to the scriptability of Vagrant itself, the ability to require a particular version of Vagrant and support for standalone file sync plugins.

  • websocketd: And finally, have you wanted to make a shell script or other app into a WebSocket server but lacked a library or access to the code to do it? Websocketd might be the answer as it turns anything with console I/O into a WebSocket server in a style rather reminiscent of CGI. Remember, most command line applications are not suitable for being exposed to the raw web, but the app could get you out of a hole when prototyping.

And, for reference, everything mentioned today is open source software.