Blogging to Nowhere
Having managed to produce five blog posts last week, I’ve taken a few days break from blogging A) because I’ve been busy with other stuff and B) because I ran out of stuff to talk about! However, now I’ve thought of something to write about…
Wow, that title is a little overwhelming. Basically this post is about what I’m going to term the Open Cloud, the problems with it and how treating The Web differently might help. I suppose I should start by defining exactly what I mean by the Open Cloud…
The Open Cloud (A.K.A Free as in AGPL)
Like it or not, cloud computing is here to stay despite what Richard Stallman might have to say and for most of us it’s going to be great. Now, I respect RMS as much as your next Free Software supporter, but I fear he’s way off the mark here. I share his concerns over companies that don’t really care about your data or your privacy. However, Cloud Computing itself isn’t inherently bad – how can it be, its just a technology and technology is only as moral or immoral as those who wield it. So, what do privacy respecting, Freedom loving, FOSS advocates do. We do what we do best – start coding! We start building an Open Cloud.
“But Surely we already have an Open Cloud?”
Yes we do, but shouldn’t confuse this Open Cloud with the Open Web. The modern web, for the most part is built on standards which are completely open, from TCP/IP right up to CSS, HTML, RSS and other glorious acronyms. However, most of the applications that live in the cloud most certainly aren’t Free or Open. They are as closed as your average binary blob, more in fact because you don’t even have access to the running binary code (oh, and APIs don’t count guys). When I say Open Cloud, I’m referring to FOSS applications running in the cloud, preferably under AGPL (other licences are available), which is specifically designed for this.
“Hang on, aren’t there loads of FOSS web-apps out there?”
Well, yes there are, this blog is running on one of them, WordPress. Then there’s Drupal, Joomla, Roundcube, Davical, etc. The problem with most all of these is that they miss out one of the prime benefits of Cloud Computing in that they all need hosting by individual users (OK, so WordPress was a bad example as there is a hosted option). What I’m really talking about is hosted applications, that people can just use, but can also be installed on your own server. These can be really small things and would preferably start with replacements for popular closed web services.
“Hang on, don’t we have a few of those already?”
Right again, but the emphasis should be on A FEW. We don’t have nearly enough of them. So far we have identi.ca/StatusNet, Libre.fm and Libravatar and that’s pretty much it (please let me know of any more), although I guess Diaspora is coming. Ideally, we would have a replacement for every proprietary web app and more, but we just don’t seem to be getting there.
So what’s going wrong…
Basically, web apps are different to desktop apps. Apart from using different technologies, they require hosting to actually run them. If you want to run your shiny new FOSS web app that you’ve just come up with as a hosted service, you yourself have to find this hosting. For small FOSS developers who might otherwise build these services this could initially put them off. However, it you’re just starting out then your hosting costs wouldn’t be huge, but as your service grows this is likely to be an increasing problem. I don’t think it’s any surprise that identi.ca (which is probably the most successful free web service) is backed by a commercial company.
So what do we do about it…
HTML5 (or just HTML as it should now be known) gives the FOSS community an amazing opportunity. The technologies available in the HTML5 specification should allow us to build web apps that are lighter on the server. Specifically, I’m talking about the offline web app functionality and local storage. This basically gives us the ability to build cool web apps that are really light on server side processing and storage. Much as I think JavaScript has its problems, it really has come of age with HTML5. HTML5 gives us the opportunity to build FOSS applications that are not only available to people running FOSS platforms, but available to anyone with a web browser (a whole new demographic of users) and hopefully without killing all our servers.
One of the projects driving this (in the FOSS world) is UNHOSTED. Primarily this is a JavaScript framework in which applications would be composed of static HTML, CSS and JavaScript (i.e. no server side scripts). The system proposes other nodes which are solely for storage, thus providing a separation between the storage of data and its processing by applications. The data would even be client side encrypted (think Firefox Sync or SpiderOak), but only with JavaScript so I’m not sure how secure this is. That’s all cool, but currently it lacks one of the major advantages of cloud computing, which is the ability to do processing on both the client and the sever. Perhaps it will evolve into something which will allow this processing in a distributed, decentralised, privacy respecting way.
I certainly think there are use cases for wholly client side JavaScript apps and that the FOSS community could tackle building some of them, whilst we are searching for solutions to the wider scalability problems of server side code and storage. However, there’s no use writing apps that everyone can use, if nobody knows about them.
An Application Distribution Platform?
The web already is an application distribution platform. It has been for ages, but up until now those applications haven’t been very discoverable. This is starting to change. I guess most people have heard about Google’s Chrome Webstore. This is their attempt to bring the App Store/Package Manager model to the web, basically its just a catalogue of web apps and some scripts that create short-cuts to them. Whether you think this equivalent to installing an application or not (I actually don’t think it matters) the idea is great, at last there will be a place to go to find web apps that you want to use. Mozilla are also jumping in with their Open Web Apps project. This is more of a specification for doing this across multiple sites and is the tech that I’m backing (personally I’d like to see web app stores that automatically discover new apps and add them to their catalogues). If one of these app stores makes its way into a future version of Firefox then that would be the perfect platform to push FOSS web apps to the masses.
Conclusion…
I don’t know whether the ability to make almost entirely serverless web apps will help the state of the Open Cloud. I really hope it does, but JavaScript still has a lot of limitations that native app developers don’t have to put up with. That said, there are a lot of advantages, such as users never having to update an app (they always have the latest version). I also don’t see web apps overtaking native apps any time soon. More likely we’ll see a continued fusion between the two, especially on mobile platforms, so that eventually the boundary between them is indistinguishable. I hope that the app store model will increase the visibility of FOSS web apps and encourage developers to get coding, but we’re only at the very start of the story on this, so only time will tell.
If you read my previous post regarding the site overhaul that I’m currently doing you will have seen me mention that I’m now using the Piwik Open Source Analytics Package in place of Google Analytics. Well I’ve had it running for a few days and have played around with it a bit, so I thought I’d review it. I’m going to start with my reasons for moving from GA and then move along and score it on several different criteria:
- Installation and Setup
- Site integration
- User interface
- Extensibility (API availability)
- Overall impressions (documentation, community, etc.)
The philosophical argument
As well as the obvious benefit (from a Freedom perspective) of using one less proprietary web service, there is also another reason that I switched away from Google Analytics. Basically, this was privacy. For a while I’ve been using technologies to limit the amount of data which leaks from my browser as I navigate the web, in order to reduce the amount of profiling of my web activities. This isn’t because I have anything to hide. I just don’t like the idea of large companies building up a huge database on me, without my permission. The upshot of this is that I found myself in the slightly hypocritical situation of blocking GA in my own browser, but using it to track others on my site.
The solution was obvious, remove GA from my site. However, I didn’t want to lose the valuable information that it provides me with. Also, I don’t have a problem with site owners collecting data that can help them, just with them sharing it with 3rd parties such as Google, who then build it into their larger profiling efforts. A quick search turned up Piwik which aims to provide a full featured GA replacement that you can run on your own server. Because site owners run their own instances, they remain in charge of their tracking of users, retain ownership of the data and best of all don’t give any data to Google.
With the aim of responsible and unobtrusive tracking in mind I’ve added a page to my site to allow users to Opt-out of the Piwik tracking by means of a cookie. The link is also accessible from the sidebar under the copyright notice. I’m afraid some of the text on that page is pretty difficult to see with my current theme, but I’m working on this. For now just uncheck the check box to opt-out.
Right, on to the main event, the actual review…
1. Installation and Setup
There’s actually not much to say here, which is because installation was ridiculously easy! I just downloaded the zip to my sever (with wget) and unzipped it into my server root directory. This produced a directory called ‘piwik’ and a ‘How to Install Piwik.html’ file, which if you point your browser at it will redirect you to the installation instructions. The rest of the installation was fairly simple, following the instructions I pointed my browser at the ‘/piwik/’ directory of my site and was greeted by the installer. Following this was really easy, you’ll need to create a MySQL database when prompted for the database info, but that’s about as hard as it gets. Towards the end you’ll be prompted to setup your site with Piwik which involves entering a few details about the site, then you’ll be provided with a snippet of JavaScript to add to your site template. Which leads me neatly into the next section…
2. Site Integration
I didn’t copy and paste the JavaScript into my template, instead opting to install the WP-Piwik addon for WordPress. This made the set up easy and also gave me a widget on my WordPress admin dashboard which gives me a nice overview of my site visits. As I already said I was also able to add a widget to the site to enable visitors to opt-out of tracking. This was also simple, just involving a copy and paste of a couple of lines of HTML from one of the settings pages into a WordPress page. Easy!
You can also integrate Piwik widgets with your site, by following the instructions in the documentation, this is a neat feature, especially if you have a custom start page set in your web browser (something which I have yet to get around to making).
I also investigated the campaigns functionality in order to track entries to my site from the RSS feed. This is really simple to use, all you have to do is append the query string ‘?piwik_campaign=NAME’, where NAME is the name of your campaign to the end of a URL, to have it show up under that campaign. I found that I could integrate this with WordPress pretty well by adding the following snippet of code to the functions.php file of my theme:
1 2 3 4 5 6 7 | function piwik_track_feed($url) { $url .= "?piwik_campaign=RSS"; return $url; } add_filter("the_permalink_rss", "piwik_track_feed"); |
If you now check the URLs in your RSS feeds, they will all have the query string added and clicks will be attributed to the ‘RSS’ campaign in Piwik.
3. User Interface
The Piwik user interface is really nice. I’ve included some screenshots below, so that you can make up your own mind. It’s pretty similar to the GA user interface, only cleaner and all the AJAX stuff makes it feel really responsive. I also love the real time tracking widget, which is something GA totally lacks. The only bad thing about the UI is the requirement of Flash for the graphs. I hate Flash and it doesn’t have a reliable 64-bit Linux version, which means I only have it installed on my netbook. Oh, and before you ask, I tried it with Gnash and it didn’t work!
-
- Piwik Locations Page
-
- Piwik Search Engine Pages
-
- Main Piwik Dashboard
4. Extensibility
By extensibility, I was primarily interested in API access. There’s certainly no shortage of this with two APIs listed on the documentation page. One API is for performing tracking, which I didn’t need given my usage of the WordPress plugin. I looked instead at the analytics API, which allows you to access all the data through simple HTTP requests. I was able to write a simple Python script to email me my main statistics once a day, in about an hour (including working out how the Python email and smtplib modules work!). Performing an Piwik API call in Python is as simple as:
1 2 3 | def getVisits(idSite, period, date): url = "%s/index.php?module=API&method=VisitsSummary.get&idSite=%d&perio$ return json.load(urllib2.urlopen(url)) |
Of course, as it’s Python its ridiculously simple!
Of course, if you find something that you can’t do with the API (which is unlikely, because it seems to cover everything), the you can access the data in the database – because it’s in YOUR database. You can also back-up and secure your data exactly how you want to. This is something that GA just can’t compete with!
5. Overall Impressions
My impressions of Piwik as a project have been really good. The documentation is excellent and there seems to be a good community behind it. As a product its a pleasure to use, really easy to install and just works. The reliance on flash for the graphs is a bit disappointing, but perhaps this will change in the future as HTML5 matures. Here are the obligatory scores:
- Installation and Setup – 5/5
- Site Integration – 4/5
- User Interface – 3/5
- Extensibility – 5/5
Overall Score: 4/5
Verdict: If your currently using Google Analytics, stop it! (and use this instead)
[EDIT: This series has been renamed so as not to conflict with Doug Hellman's excellent Python Module of the Week series. See UPMotW #1 for details]
I’m going to try something different today. This is going to be the first in a series (hopefully once weekly) of Python Modules of the Week. This has probably been done to death elsewhere, but I’m using it as an opportunity to learn some more Python and to cement some of these modules in my mind.
Argparse is a Python module for processing command line arguments. It’s new in version 2.7 and is designed to replace the Optparse module. I’m not sure what the rationale for the change was as the two modules look very similar to me. However, it suffices to say that you should use argparse in all new code.
The main class of argparse is called ArgumentParser, it allows you to add arguments to your program and parse the provided arguments at runtime:
1 2 3 4 5 6 | parser = argparse.ArgumentParser() parser.add_argument("-v", "--verbose") # a flag type argument parser.add_argument("value", type=int) # an integer positional argument ... args = parser.parse_args() # parse the args from sys.argv print(args.value) # arguments accessed as data members |
You can also add meta-data such as help and descriptions to your arguments via keyword arguments to many of the methods. Take a look at the module documentation for examples. Once nice thing is that when you provide this data, argparse automatically generates the -h and –help flags and populates them with nicely formatted help output using the info you provide.
The argument parsing provided by argparse allows you to create some very complex command line interfaces to your scripts. I especially like the sub-commands framework, which allows each sub-command to exist as its own separate entity with its own options and help information. To create an argument parser with a sub-command is just a few lines:
1 2 3 | parser = argparse.ArgumentParser() subcommands = parser.add_subparsers() mysubcommand = subcommands.add_parser('mysubcommand') |
Of course,each sub-command is an ArgumentParser in its own right, so you can add all the options that you normally would, including command line flags, positional arguments and help information. Also each sub-command will be listed in the help and will also have its own help page which can be accessed via:
1 | ./pyprog.py subcommand --help |
All in all, this is a very useful Python module to know about. Its ease of use and the number of things it ‘just takes care of’ make it a pleasure to use. Hopefully it’ll have you writing nice command line interfaces to all those hacked together scripts that are currently controlled by commenting bits of them in and out!
I’ve been thinking more recently about automating routine computing tasks. Using Free Software this is ridiculously easy, usually a combination of Python and Cron does the trick. More difficult I have found is actually working out what to automate. It’s very easy to get into the habit of doing something and then not realise that you do it every day, or every hour when in front of the computer.
One of my major successes in this area has been in terms of feed reading. A while ago I setup rss2email on my server to check my feeds and email the stories to me. Previous to this I actually found it really hard to ever get around to reading news feeds. I think this was mainly because checking news feeds suffers from the ‘extra inbox problem’, its just more stuff to check. Having news stories emailed to me fixes this and I can also read them easily on my phone. Because I use IMAP email, my read/unread status is also synchronised on all my computers.
Now I’m kind of at a brick wall. I don’t know what else I can set up to reduce the number of routine tasks I do. The purpose of this post is to crowd source that problem. I want everyone reading this to get back to me with the things they’ve set up to help them. You can get in touch either through the comments on this post, or via identi.ca. If I get enough responses I’ll write a follow up post detailing some of the best ideas.
This morning TVNZ ran a story on their ‘Breakfast’ TV show about how the signs of the Zodiac are out of date due to the wobbling of the Earth on its axis (a phenomenon known as ‘precession’ – incorrectly referred to in the article as ‘procession’).
This story is based as they rightly say on actual scientific evidence. However, what they forgot to mention is that there is no credible scientific evidence for the efficacy of Astrology. I’ll quote Ivan Kelly (an outspoken critic of Astrology) on the subject:
[Astrology] has no relevance to understanding ourselves or our place in the cosmos. Modern advocates of astrology cannot account for the underlying basis of astrological associations with terrestrial affairs, have no plausible explanation for its claims, and have not contributed anything of cognitive value to any field of the social sciences.
So, it is irrelevant as to the actual number of star signs in existence because they have no bearing on anything whatsoever. The tacit acceptance of the truth of Astrology by TVNZ undermines their credibility as a news organisation (it should be noted that its not just TVNZ running this story, as a quick Google Search will show). The story also contained a couple sentences stressing the reality of the phenomenon of precession which, although true, has the effect of subliminally linking this very real phenomenon with the pseudo-scientific practice of Astrology.
The professor currently proposing this (Professor Parke Kunkle) comes from the Minneapolis Community and Technical College, which I’m not sure is a hugely authoritative institution on Astronomy. I don’t even know if Professor Kunkle is actually an Astronomer or whether he has been incorrectly labelled as such by the media (he may be an Astrologer rather than an Astronomer). Given that I can’t find any staff information on the MCTC website I’ll have to leave this open (of course he could be a well meaning skeptic pointing out that Astrology is wrong! It would be typical media incompetency for them to get the wrong end of the stick).
I’ve also put in a complaint to TVNZ on the subject via their feedback form, I’d encourage you to do the same:
I take issue with your recent report on ‘Breakfast’ regarding a 13th sign of the Zodiac in which you tacitly accept Astrology as fact and link it to a real scientific phenomenon without any clarification of its status of unsubstantiated pseudo-science. I refer you to my blog posting on the subject for more of my thoughts: http://blog.webworxshop.com/2011/01/18/note-to-tvnz-astrology-is-bs
Of course, I’ll report back if I get a response. I’ll just leave you with a quote from a great Astronomer, Johannes Kepler:
God provides for every animal his means of sustenance – for an astronomer he has provided astrology.
