rubyonrails-ug.de Planet

DevHouseFriday is back! (Update)

2010-09-06T21:25:09+00:00

As mentioned a few days ago the next DevHouseFriday is coming up. We have our first speakers! Galaxy Cats, 9elements and SalesKing are going to talk about the future of web technologies! We’re searching for a few more speakers. If you want to talk about something you’re thinking it is the future of web technologies, just drop us a line to team@railslove.com.

Stay tuned! More updates and a full session table are comming this week.

P.S. und alles findet natürlich im sehr gemütlichen Kölner Coworking Space “Gasmotoren Fabrik” statt.

My nodeJS deployment approach

2010-09-04T13:05:14+00:00

A while back I said I wasn’t really satisfied with how I was deploying node applications, in this case Codeshelver. After dealing with the problem for some more time I’ve now arrived at a more general solution that I’d like to share with you. Having a strong Ruby background some of this will seem familiar to you Ruby developers: I’m deploying to a nginx server, the node process gets monitored with monit and the basic deployment stack includes Vlad as a deployment tool, a ndistro file that contains the app dependencies (think of it as a Gemfile) and a self-written start/stop script for my node applications (with a little Express flavor).

Rakefile

This is a standard Rakefile that just loads Vlad and sets it to use git for source control.

require 'rake'
 
begin
  require 'vlad'
  Vlad.load(:app => nil, :type => nil, :scm => 'git')
rescue LoadError
  puts "Could not load Vlad - please run 'gem install vlad vlad-git'"
end

deploy.rb

This is Vlad’s config file that contains some configuration as well as some recipes I wrote for ndistro and the deployment process:

set :user, "deploy"
set :application, "myapp"
set :domain, "myapp.com"
set :deploy_to, "/var/www/#{application}"
set :repository, "git@github.com:myuser/#{application}.git"
 
namespace :vlad do
  # overrides
  set :shared_paths, { }
  set :mkdirs, []
 
  namespace :ndistro do
    desc "Install ndistro"
    remote_task :setup, :roles => :web do
      set :use_sudo, true
      run "cd /usr/local/bin && curl http://github.com/visionmedia/ndistro/raw/master/install | sh"
    end
 
    desc "Install dependencies"
    remote_task :install_deps, :roles => :web do
      %w(bin lib modules).each do |dir|
        shared_dir_path = File.join(shared_path, dir)
        release_dir_path = File.join(current_release, dir)
        run "mkdir -p #{shared_dir_path}"
        run "ln -s #{shared_dir_path} #{release_dir_path}"
      end
      run "cd #{current_release} && ndistro"
    end
 
    desc "Clean dependencies"
    remote_task :clean_deps, :roles => :web do
      %w(bin lib modules).each do |dir|
        shared_dir_path = Filec.join(shared_path, dir)
        run "rm -r #{shared_dir_path}"
      end
    end
  end
 
  desc 'Start the app'
  remote_task :start_app, :roles => :app do
    run "/etc/init.d/express_app #{application} start"
  end
 
  desc 'Stop the app'
  remote_task :stop_app, :roles => :app do
    run "/etc/init.d/express_app #{application} stop"
  end
 
  desc 'Restart the app'
  remote_task :restart_app, :roles => :app do
    %w(stop_app start_app).each { |task| Rake::Task["vlad:#{task}"].invoke }
  end
 
  desc "Full deployment cycle: Update, ndistro:install_deps, restart, cleanup"
  remote_task :deploy, :roles => :app do
    %w(update ndistro:install_deps restart_app cleanup).each do |task|
      Rake::Task["vlad:#{task}"].invoke
    end
  end
end

.ndistro

The ndistro file contains the module and node dependencies of your app. This one is pretty simple – here you can read up more on ndistro and there also is a short screencast about it.

module senchalabs connect
module visionmedia express
module visionmedia expresso
module visionmedia haml.js
module visionmedia sass.js

express_app

This is the script I use to start/stop the server, I keep it under /etc/init.d/ so that I can use it like any other system script. You use it like this: /etc/init.d/express_app {app_name} {start|stop}

#!/bin/bash
user=deploy
dir=/var/www/$1/current
srv_file=$dir/server.js
pid_file=$dir/log/server.pid
log_file=$dir/log/server.log
 
case $2 in
  start)
    sudo -u "$user" EXPRESS_ENV=production node "$srv_file" >> "$log_file" 2>&1 &
    echo `ps ax | grep $srv_file | grep -v grep | awk '{print $1}'` > "$pid_file"
    chown "$user":"$user" "$log_file"
    chown "$user":"$user" "$pid_file" ;;
  stop)
    kill `cat "$pid_file"`
    rm "$pid_file" ;;
  *)  
    echo "usage: /etc/init.d/express_app {app_name} {start|stop}" ;;
esac
exit 0

monit

This is pretty simple as it just keeps an eye on the server pid file and utilizes the start/stop script to keep the app running.

check process myapp with pidfile /var/www/myapp/current/log/server.pid
  start program = "/etc/init.d/express_app myapp start"
  stop program  = "/etc/init.d/express_app myapp stop"

nginx.conf

Last but not least here’s an excerpt from the nginx config file:

http {
  upstream myapp_cluster {
    server localhost:3000;
  }
 
  server {
    listen 80;
    server_name myapp.com;
    root /var/www/myapp/current;
 
    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_pass http://myapp_cluster/;
      proxy_redirect off;
    }
  }
}

I hope this helps you figuring out your deployment and maybe you got some tips for me on how to enhance this. Of course there are many other valid approaches like using upstart or with an Apache as webserver.

Why I Love and Hate Distributed Systems

2010-09-03T00:00:00+00:00

Let me go ahead and say it: I love distributed systems. Why? Simply because they bend my brain. Yesterday I tweeted "Distributed databases are my happy place." One response I got was along the lines of: "then you're probably not running a distributed database in production." Busted! But does it matter? We all love distributed stuff, we love thinking about scaling. They seem like problems everyone wants to have and solve.

But the truth is, I don't, and I can assure you, you don't want to either, sometimes I doubt my brain is even capable properly solving these problems, but that doesn't prevent me from trying. I prefer to work on as small a scale as possible, you could even say I hate distributed systems. Scaling and distribution is a problem most of us don't have, and are probably better of not having.

Truth be told, I'm not highly interested in running highly distributed systems in production, quite the opposite. I prefer maxing out what I have as far up as possible. Sometimes I do take the plunge and just try something new in production, but I'm happy prepared to replace it with something different, even something simpler, if that seems like the better option in the end. Everyone should experiment at some point, but not all the time.

But why then do I love distributed systems? Simply because they make me think about how they could be put to use, what algorithms and the problems involved are, and what implications they would have on a production system, both from an operations and developer perspective. That's where the value is for me, it allows me to simply make informed decisions when the time comes.

Take Riak, for example, on which I gave a shortish talk at yesterday's meet-up of the local Ruby brigade. Riak's distribution model is based on Amazon's Dynamo implementation, with some neat features sprinkled on top. Riak is built by a bunch of really, really smart guys at Basho, whose work I have nothing but respect for, but who also are sane and open enough to tell people when their database may or may not be a good fit (something a certain other database is severly lacking).

Riak is exciting for me because it was the first database that really made me dive into Amazon's Dynamo, and once I started grokking it, it blew my mind. If you haven't read it, please do. It blew my mind simply because it introduced me to a whole new thinking, to heavily distributed storage, with all the potential hot spots, downsides and business use cases for specifics parts of it thrown in. The same is true for Google's BigTable. The technologies involved with both are true mind-benders.

And there's my bottom line. Distributed systems aren't necessarily awesome just because they allow scaling to infinite heights (exaggeration intended), but because they broaden your personal horizon. It's like learning new programming languages. It's about getting new ideas in your head, ideas outside of your everyday working realm. Ideas you can maybe even take back to what you're working on and start applying them where it makes sense, and only if it makes sense. Learning about distributed systems is not just about learning how to use them, but when. Knowing is half the battle.

While you're at it, check out Evan Weaver's "Distributed Systems Primer", a collection of papers on distributed systems, or the papers collection over at NoSQL Summer. Get ready to have your mind blown in whole new ways. Say what you will, that stuff is just fascinating. It appeals to the distributed database lover in me.

Am Montag wieder ein Webmontag

2010-08-30T11:02:56+00:00

Am Montag, dem 13.09.2010 ist es wieder so weit: der 23. Kölner Webmontag läd wieder ein. Da in dieser Woche auch die Internetwoche Köln stattfindet (13.09 - 19.09.), passt der Webmontag hier ganz gut in die Reihe.

Wir treffen uns wie gewohnt um 19:00 Uhr im Spielplatz und freuen uns auf zahlreiche Besucher!

Mehr Informationen findet ihr unter:

Der Webmontag im Wiki
Webmontag auf Upcoming
Webmotag auf Venteria
webmontag.de
Internetwoche Köln

DevHouseFriday is back!

2010-08-27T14:31:50+00:00

Während der Internetwoche Köln gibt es wieder ein DevHouseFriday. Nach der Sommerpause dieses Jahr, treffen wir uns am 17.09.2010 um 19:00 wieder. Dieses mal im Coworking Cologne Space - der Gasmotorenfabrik.
Unser Thema dieses mal ist: The future of Web Technologies..

Alle Hacker und Thinker aus Köln und Umgebung sind herzlich wilkommen!
Mehr Informationen gibt es unter:

Wir freuen uns sehr auf euren Besuch!

Celebrating Whyday!

2010-08-19T09:12:30+00:00

why the lucky stiff was the persona of an anonymous and prolific writer, cartoonist, musician, artist, and computer programmer.(wikipedia) He was one of the key figures in the Ruby community and helped to build what the Ruby community is today. He gave the Ruby community the spirit of freedom, whimsy and most notably creativity.
On August 19, 2009 he withdrew from the online community. He shut down all his online presence including his Github, Twitter Accounts, the sites he maintained, etc. Since then his work has been collected at the whymirror on Github.

The Ruby community wishes him well and celebrates Whyday today to remember Why’s contributions to the community and culture.

The Ruby hackers here at cowoco join the celebration. So put away your best practices and create something wonderful! Head over to the whyday page to get some ideas: Try that wild idea you’ve been sitting on because it’s too crazy, teach programming to a child, see how far you can push some weird corner of ruby, …

Happy whyday!

ART && CODE Symposium: Hackety Hack, why the lucky stiff from STUDIO for Creative Inquiry on Vimeo.

One of why’s talks: ART && CODE Symposium: Hackety Hack, why the lucky stiff from STUDIO for Creative Inquiry on Vimeo.

via Coworking Cologne

What we are really doing after our working hours

2010-08-19T00:50:04+00:00

This is a very short post about what is railslove doing after working hours. Yes, we’re drinking beer, playing urban terror or somking cigars and thinking about “how we do spend our next million”….

Yeah.. and what is railslove really doing? The short answer is: coding, thinking about really hot applications and having a lot of fun to work with our team.

But now we’re trying something new: Railslove Research (short: RR). So we started a small independent blog, where we want to find out, how to do apis more usable and flexible for developers. Just visit our blog: http://arrrpi.posterous.com (and of course we’re building a sexy application around that…)

AWS User Group Köln Meetup am 18.08, 19:00 Uhr

2010-08-18T13:28:39+00:00

Heute (18.08.) trifft sich die AWS User Group Köln im Coworking Space der Gasmotorenfabrik. Schwerpunktthema während der Veranstaltung werden die Cloud Services von Amazon sein. Startzeit ist 19:00 Uhr.

Es wird sich zu der Nutzung von Amazons Web Services wie EC2, S3 und anderen gemeinsam ausgetauscht. Nach zwei, drei kurzen Vorträgen zu Beginn des Treffens ist dann im Anschluss viel Platz zur regen Diskussion und fürs Networking.

Mehr Informationen findet man bei Xing

Via Coworking Cologne

An Inconvenient Caveat about MongoDB's Replica Sets (updated)

2010-08-12T00:00:00+00:00

Update: Read the comments and below. The issue is not as bad as it used to be in the documentation and the original design, thankfully.

A lot has happened since I've first written about MongoDB back in February. Replica Pairs are going to be deprecated, being replaced by Replica Sets, a working Auto-Sharding implementation, including rebalancing shards, and lots more, all neatly wrapped into the 1.6 release.

The initial draft on how they'd turn out sounded good, but something struck me as odd, and it is once again one of these things that tend to be overlooked in all the excitement about the new features. Before we dive any deeper, make sure you've read the documentation, or check out this rather short introduction on setting up a Replica Set, I won't go into much detail on Replica Sets in general, I just want to point out one major issue I've found with them. Part of the documentation sheds some light on the inner workings of Replica Sets. It's not exhaustive, but to me more interesting than the rest of the documentation.

One part struck me as odd, the paragraph on resyncing data from a new primary (as in master). It's two parts actually, but they pretty much describe the same caveat:

When a secondary connects to a new primary, it must resynchronize its position. It is possible the secondary has operations that were never committed at the primary. In this case, we roll those operations back.

Also:

When we become primary, we assume we have the latest data. Any data newer than the new primary's will be discarded.

Did you notice something? MongoDB rolls operations back that were never committed to the primary, discarding the updated data, which is just a fancy term for silently deleting data without further notice. Imagine a situation where you just threw a bunch of new or updated data at your current master, and the data has not yet fully replicated to all slaves, when suddenly your master crashes. According to the protocol the node with the most recent opslog entries takes over the primary's role automatically.

When the old master comes back up, it needs to resynchronize the changes from the current master, before it can play any role in the set again, no matter if it becomes the new primary, or sticks to being a secondary, leaving the new master in place. During that resync it discards data that has not been synchronized to the new master yet. If the opslog on the new master was behind a couple of dozen entries before the old one went down, all that data is lost. I repeat: lost. Think about that.

There's ways to reduce the pain, and I appreciate that they're mentioned appropriately in the documentation. You can tell MongoDB to consider a write successful when it replicated to a certain number of secondaries. But you have to wait until that happened, polling getLastError() for the state of the last operation. Or you could set maxLag accordingly, so that the master will fail or block a write until the secondaries catch up with the replication, though I couldn't for the life of me figure out (using the Googles) where and how to set it.

But I don't approve of this behavior as a default, and the fact that you need to go through the internals to find out about it. Everything else suggests that there's no point of failure in a MongoDB setup using sharding and Replica Sets, even comparing it to the Dynamo way of guaranteeing consistency, which it simply isn't when the client has to poll for a successful write.

It's one of those things that make me reconsider my (already improved) opinions on MongoDB all over again, just when I started to warm up with it. Yes, it's wicked fast, but I simply disagree with their take on durability and consistency. The tradeoff (as in: losing data) is simply too big for me. You could argue that these situations will be quite rare, and I would not disagree with you, but I'm not fond of potentially losing data when they do happen. If this works for you, cool! Just thought you should know.

Update: There's been some helpful comments by the MongoDB folks, and there's good news. Data is not silently discarded in 1.6 anymore, apparently it's stored in some flat file, fixed with this issue, though it's hard for me to say from the commits what exactly happens. The documentation does not at all reflect these changes, but improvements are on the way. I'm still not happy about some of the design decision, but they're rooted in the way MongoDB currently works, and changing that is unlikely to happen, but at least losing data doesn't seem to be an option anymore. If making a bit of a fool out of myself helped to improve on the documentation front, so be it. I can live with that.

Ferret 0.11.8

2010-08-06T11:22:32+00:00

Just in case you didn’t notice - Ferret development has continued during the last year.

Unfortunately no new gem has been released for a while, so in case you want to give it a try here’s how to build and install your very own Ferret 0.11.8.1:

For me the new version fixed some rare crashes occuring when running Ferret along with acts_as_ferret under Ruby Enterprise Edition 1.8.7. Reading the commit messages it also looks like this version is ready to compile and run with Ruby 1.9.

If you’re feeling lazy you can also download the Ferret 0.11.8.1 gem from this blog.

GitHub, I like…

2010-07-30T16:20:53+00:00

Toying around with GreaseMonkey I wrote a little userscript that integrates the Facebook Like button into GitHub repository pages (click here to install it). Nothing special, but maybe you like it too

Next up: Stop messing around with GreaseMonkey and learn how to write Safari Extensions.

10 Annoying Things About CouchDB

2010-07-26T00:00:00+00:00

Hi, I'm Mathias, and I'm a CouchDB user. I've been using it for almost a year now, and we have a project using it in production, with a side of Redis. I think it's an awesome database, some of its features are simply unrivaled. Offline replication, CouchApps, to name a few. CouchDB just hit version 1.0. It's been a long time coming, with CouchDB having probably one of the longest histories in the non-relational database space. I've heard about it first back in September 2008, when Jan Lehnardt talked about it at a local co-working space. I still blame him for getting me all excited about this whole NoSQL thing. Fun fact: I bookmarked the CouchDB website back in February 2008.

The features being added to it with every release are nothing short of exciting. CouchDB 0.11 got filtered replication, support for URL rewriting and vhosts, amongst other things. But there's still some things that annoy me, that somewhat bug me in my daily work with it.

The following things are not incredible pet-peeves I have with CouchDB. I think CouchDB is pretty awesome, and I really like using it. However, it doesn't come without the occasional odditity that will leave you scratching your head. These probably aren't the only things to be aware of, they're just the most annoying to me. Your mileage may vary. They may or may not be annoying to you, but they're things that are good to know working with CouchDB. Whether CouchDB should or should not have what I'm listing here is a whole different story. It's my wishlist of improvements, if you will.

It's also stuff you're buying into when you move off the beaten path of relational databases. As always, some of these are not hard to find out, some of them do only get really annoying once you're moving into production, or when you get a deeper knowledge of the tool at hand. Nothing specific to CouchDB here, but some of the issues listed below stem from actively using it. Take them with a grain of salt. While they may seem annoying at first, they're things you can live with. Believe me, you can.

Views are updated on read access

You can dump in as many documents as you want, and you can create as many map/reduce views as you want. The truth is, they'll only come all together to slow down your application when you're querying the view. Assume you have a good stash of documents in your database, and you decide you need a new view on your data. Throw in the JavaScript functions and go ahead and query the view. Calling it a slow-down may be a stretch at times though, it really depends on how often your data is updated.

CouchDB will notice that the B-tree for the view doesn't exist yet, so it goes ahead and builds it on the first read. Depending on how many documents you have in your database, that can take a while, putting a good work load on your database.

On every subsequent read, CouchDB will check if documents have changed since the last write, and throw the changed documents at the map and reduce function. So if you only query some views from time to time, but have lots of changes in between, expect some delays on the next read. A way around this would of course be to keep your views warm by reading them regularly, e.g. through a cron job.

When you add new views, be sure to pre-warm them before you first access them in your application. One way would be to add the views at a time where you database isn't accessed as much. It doesn't block all access to the documents, but it sure has a certain impact on your database's performance, and of course the first requests that may time out because CouchDB is building the requested views in the background.

When it comes to just updating a view, and it might take too long, you can set the parameter stale=ok. That way, even if the view data needs to be updated, CouchDB won't update it and just return the last known state of the view's B-tree.

That's all fun and giggles, but when on earth are you supposed to actually update your view? Always reading stale data is not great? I've gotten some odd suggestions when I complained about this elsewhere, but in the end I just want to tell the database that I'm okay with stale data, but that it should update the view in the background.

No automatic compaction

As your database grows and data gets updated, CouchDB leaves old and stale data untouched, appending new data (inserted and updated documents are considered new data) to the end of its database files, a fact that's also true for view files. That has the neat advantage that you can still access old revisions of your documents, but it will also leave your database files growing constantly. Now, depending on the number of documents and updates on them, that might not be a big deal, but it's a good idea to start regular compaction earlier than later.

Riak's Bitcask file backend has a neat way of automatically compacting its files. It appends data in a similar manner as CouchDB, but can determine if a node in the cluster can run compaction on its data, and do so automatically, without much need for human intervention. It'd be nice to have something similar as part of CouchDB without having to run cron jobs to do that.

The append-only mechanism makes CouchDB bullet-proof, no doubt, you'll always have consistent data files on your hard disk, backups are as simple as copying the files elsewhere, or take an EBS volume snapshot at any time. But that level of data consistency comes with a price, and that's an ever-growing data file.

No partial updates

Whenever you update a document in CouchDB, you update it as a whole, there's nothing in between. That kind of makes sense with the way CouchDB works, but as a user it annoys me from time to time. It seems so pointless fetching and sending a whole document when I'm just updating one attribute. There's a neat RFC for the PATCH command in HTTP making the rounds, I'd love to see that end up in CouchDB at some point. No idea how likely that is, the makers of CouchDB have a weird aversion to using diffs to update data.

Note that I'm not talking about the MongoDB way of setting attributes atomically. I don't need that, because it simply doesn't scale well, especially not with the CouchDB storage model, and you're not updating data in-place like MongoDB. It's more about just being able to send a diff or a minor update than a whole document.

You can somewhat fake this using update handlers (look at the view called "in-place") from CouchDB 0.10 on. It's pretty neat, but it's just not the same.

No built-in way to scale up

CouchDB's replication is unrivaled, no doubt. Being able to replicate any database with any other database at any point in time makes CouchDB unique, some say it's the killer feature, and I concur. There's lot of argueing whether or not that defines CouchDB as being distributed. In the most traditional sense, at least to me, it sure does, but I'm not here to nitpick about that. It's easy to scale out by adding more nodes and setting them up to constantly replicate with each other, make anyone a master or slave as you like. But there's no way to distribute write and read access across a cluster of nodes.

CouchDB-lounge has been the traditional way to approaching that, but I never really liked it, because it added more components to the infrastructure. Something like that should really be built in. The good news is that Cloudant is planning on open-sourcing their clustering solution Open Cloudant, which will then hopefully become part of CouchDB. A quorum based system for CouchDB would be neat, and it doesn't seem too far away.

Pagination is awkward

CouchDB's B-tree is a leaky abstraction, that's the conclusion I came to at some point. It has a pretty big impact on your application's code, and that's not necessarily a bad thing. Suddenly you deal with things like conflicts, or simply updating views on reads. But no other part of your web application will make that as obvious as pagination, a pretty common and natural part of a web application.

The path of least resistence to get pagination is to use the skip and limit parameters, but it's not recommended, as you'll still be walking the whole B-tree to determine the number of documents that must be skipped before it can collect the ones you're interested in.

The recommended way to do pagination is a bit awkward if you ask me. There's a good explanation in the CouchDB book, so I'll spare you repeating it here. But be sure to read it, because understanding that takes you half way to understanding the B-tree. It may be awkward, and very different from what you're used to, but that's how the B-tree works. It's not always unicorns and rainbows, sometimes it kinda gets in your way. Trade-offs, meh.

The simpler alternative would of course be to just use endless pagination, where you let the users just click a more button instead of clicking through the pages, because you know the last document displayed in your list, and the key that was used to fetch it. You simply use that key and the last document's id to step directly into the B-tree where you left off. You need to remember to fetch one additional document, as CouchDB will return the last document too, or you can just skip one document, which is acceptable, as skipping just one leaf in a tree is an operation of predictable performance.

Range queries are awkward

To do a range, you have to specify a start and an end key. That's the simple part. It starts getting awkward when your keys get slightly more complex, e.g. when your map function emits arrays. Assume you want to fetch all elements where the first part of the array matches a particular key, and the second part doesn't matter, e.g. when you emitted a timestamp as the second part to keep a natural (in terms of last update for example) order.

Assume your keys look like this: ['123', '2010/07/21'], that's the key format SimplyStored uses to manage associations between documents. To get the range that only matches the first part of the key, your startkey has to look like this: ['123']. This will match all documents having the above key. If you don't specify an endkey, CouchDB will simply return all documents following that key, so you need to specify an endkey. The recommended way to do that is to use the following format: ['123', {}]. That way you'll get all documents matching the first part of the key, because {} is considered to be greater than any string you may have emitted. See the CouchDB wiki on more details on this technique called view collation.

Obviously it's not impossible to do range queries in CouchDB, but it's slightly awkward. It all goes downhill as soon as you want to fetch only a particular subrange of the original one, using startkeydocid or endkeydocid, say for pagination. With the above ranges, they simply don't work. Both need a startkey and endkey that is an exact match. The whole point of the above range query is not to care about the exact start and end key, isn't it?

No CommonJS available in MapReduce functions

With CouchDB 0.11, CommonJS and all its awesomeness became available in view functions. I was pretty excited about it, and I still am. However, map and reduce functions were left out in the cold. Every time I have to write the same piece of JavaScript in a map or reduce function that I've used elsewhere already, I get bitter about that. Sometimes it's just very basic stuff that I could easily solve by throwing an existing library at it, but instead I'm cluttering my view code with it over and over again. And yes, there's the !code placeholder, but it's not about throwing an undebuggable mess of code into my view function, it's about not repeating myself. !code doesn't really solve that problem good enough for me.

Word is that it's got something to do with determining whether files have updated or not, but hey CouchDB, why don't you let me worry about that and let me tell you when I think a file I've included through CommonJS has been updated? I would very much appreciate that.

No link-walking between documents

With CouchDB 0.11, map functions got a way to emit other documents using {_id: doc.other_id}, but that still doesn't allow full access to e.g. attributes of said documents. Sometimes that'd just be handy to have. Sure, you could use embedded documents, but in that case it'd just be a dumb workaround, where I could just have a way to fetch a document by its identifier and throw some of its attributes at the map function.

Say what you will though, just being able to emit other documents is still pretty cool. Makes querying and fetching associated documents a bit easier.

All reads go to disk

CouchDB doesn't cache anything. It does delay commits if you want it to, so that it doesn't hit the disk on every document update, but it sure as heck doesn't cache anything in memory. This is both curse and blessing. It keeps the memory footprint of CouchDB incredibly small, no doubt. Considering they're targeting mobile devices it makes a lot of sense, plus, accessing flash-based storage is a lot cheaper than spinning disks.

But, on the other hand, when I have the memory available, why not use it? I know caching is a hard problem to solve. CouchDB is also made for high concurrency, no doubt, but my disks aren't necessarily. Sure, I could buy faster disks, but if you really think about it, memory is the new disk, plus, tell Amazon to offer faster network storage for EC2, please do, maybe that'd already help. CouchDB somewhat relies on the file system cache doing its magic to speed up things, but I really don't want to rely on magic. You could put an HTTP-level reverse proxy like Varnish in front of CouchDB though, that'd be a feasable option, but that adds another layer to your infrastructure.

In all seriousness, I'd love to see some caching introduced in CouchDB. I won't say it's an easy feature to implement, because it sure isn't, but it doesn't need to be something fancy, I just would like to see CouchDB use some of my memory for data that's read more often than it's written. But until then, Varnish to the rescue!

Error messages are not helping

I'm just gonna post the following snippet from my CouchDB log file, and leave you to it. You tell me how useful it is. Suffice it to say, I just wish CouchDB would not dump all that Erlang trace into my log, but maybe a useful error message for a change. It works in some cases, but a lot of times, when the problem usually is as simple as a permissions problem, you're left scratching your head.

{<0.84.0>,supervisor_report,
 [{supervisor,{local,couch_secondary_services}},
  {errorContext,start_error},
  {reason,
      {'EXIT',
          {undef,
              [{couch_auth_cache,start_link,[]},
               {supervisor,do_start_child,2},
               {supervisor,start_children,3},
               {supervisor,init_children,2},
               {gen_server,init_it,6},
               {proc_lib,init_p_do_apply,3}]}}},
  {offender,
      [{pid,undefined},
       {name,auth_cache},
       {mfa,{couch_auth_cache,start_link,[]}},
       {restart_type,permanent},
       {shutdown,brutal_kill},
       {child_type,worker}]}]}}

The End

There you go, some annoying things about CouchDB. They're annoying, but I still like CouchDB a lot. It's stuff I can live it, it's stuff I can work around, it's stuff that doesn't have as big an effect in production as it may seem. The bottom line is, as always, evaluate your tools. The above list is not to be taken as a list of arguments purely against using CouchDB. Consider them a list of things you need to be aware of, that may or may not be acceptable compared to what you gain.

In the end, and any way you look at it, CouchDB still kicks butt.

Loading GreaseMonkey userscripts from a server

2010-07-21T07:11:34+00:00

Codeshelver relies heavily on the GreaseMonkey/GreaseKit userscript to customize the GitHub dashboard and repository pages. As I have got some more ideas for extending these functionalities (i.e. adding a list of recently shelved repositories to the dashboard) I was searching for a way to load the userscript from my server so that the users do not have to update the userscript manually everytime I add new features.

It turns out to be fairly simple, because you just have to split your userscript into two parts: The first one is the userscript that the user adds to his browser and which just loads the second part, namely a javascript file that gets integrated into the webpage and which contains the real functionality.

myapp.user.js

This is the userscript that you offer to download/add to the browser: It just integrates the real userscript from the server, so that the users do not have to update their userscripts and it also disables the userscript functionality in case your server is not available. The latter can be seen as a bug or a feature (depends on your use case), but for codeshelver.com this is a feature, because if the server is down you are not able to shelve repositories anyway

(function() {
  //
  var script = document.createElement('script');
  script.src = 'http://myapp.com/userscript.js';
  script.type = 'text/javascript';
  document.getElementsByTagName('head')[0].appendChild(script);
})();

userscript.js

Here goes the real functionality

var Userscript = {
  init: function() {
    alert("Hello :)");
  }
};
Userscript.init();

As I said this is pretty straight forward, so go ahead and enhance the web!

Codeshelver: Tagging GitHub repositories

2010-07-18T12:37:28+00:00

Last week I polished my latest spare time project so that I could release it into the wild: Codeshelver lets you clean up your GitHub watchlist by tagging repositories you would like to remember. If you are a GitHub fanboy like me you probably know the problem: Almost daily an interesting project appears in your timeline and you want to remember it, because you might need something like that in the future. You hit the watch button and from now on you are notified about every single commit, issue and wiki change of that project- This is great for code you are using on a daily basis but doesn’t make sense for repositories you would just like to remember, because it practically spams your dashboard timeline with things you are not really interested in.

I had the idea for this project on my mind for some time, but didn’t get around to realizing it because I lacked the motivation to build it with the common stack of technologies. A few weeks ago I started to play with NodeJS and tried to come up with something that was beyond the Hello World examples so I stumbled upon ExpressJS, which could be described as the Sinatra of NodeJS. And so I had my use case for trying out some new technologies to which I also added GreaseMonkey and CouchDB to make it more buzzwordy.

All in all it was fairly easy to develop the app as I just had to knit some good libraries together like express, node-couchdb and nodejs-autorestart which I’m currently using to make the deployment a little easier. Talking about deployment: I had some problems figuring out how to do it the best way, everyone seems to be using Ubuntu’s Upstart and Monit for deploying NodeJS apps. I’d prefer a more generic solution, but I haven’t found a really satisfying way yet, so I’m just killing and restarting the processes.

There are some features in the pipeline that I’d like to add once some more people are using the service and I gather more data, which would offer the possibility to evaluate things like trending and interesting repositories. Now that ExpressJS 1.0 is around the corner I’ll migrate the app and will most likely use CoffeeScript for that, because it looks very promising and easy to integrate – it will also give me one more buzzword up my sleeve

Please go ahead and try Codeshelver, let me know what you think and leave comments and feedback here!

Installing a current CouchDB (and Erlang) from source on Ubuntu 8.04 LTS

2010-07-14T20:13:30+00:00

Two weeks ago I had quite a hard time getting CouchDB up and running on my VPS, but with some help from Jonathan and Lars it finally worked. My main problem was that Erlang wasn’t configured with SSL support and that the CouchDB version I was using (0.11.0) did not play nice with the latest Erland (R14A), so that I had to get CouchDB from the SVN repository. Now that CouchDB 1.0 is here, I thought it might be a good opportunity to summarize the steps I had to take to install the latest Erlang and CouchDB 1.0 on my Ubuntu Hardy system…

UPDATE: Here is a gist that looks like there is a more hassle-free way to do the installation, please try it first and see the rest of this tutorial as a reference point if the gist does not work out for you.

I usually keep my source installs at /usr/local/src/, your milage may vary, adjust the steps accordingly.

Installing the dependencies

First of we got to install the dependencies for Erland and CouchDB:

$ sudo apt-get install -y automake autoconf libtool help2man
$ sudo apt-get install -y build-essential erlang libicu-dev
$ sudo apt-get install -y libmozjs-dev libcurl4-openssl-dev
$ sudo apt-get install -y openssl libssl-dev

Installing Erlang

Download and unpack the current Erlang version, in my case it’s R14A:

$ cd /usr/local/src
$ wget http://www.erlang.org/download/otp_src_R14A.tar.gz
$ tar xzf otp_src_R14A.tar.gz
$ cd otp_src_R14A

Maybe you would like to see the config options and need to checkout which openssl should be used:

$ ./configure -h
$ which openssl
/usr/bin/openssl
$ ./configure --with-ssl=/usr/include/openssl/
$ make
$ sudo make install

Once you installed Erlang, check whether its cryptography library (OpenSSL) are functioning correctly.

$ erl
Erlang R14A (erts-5.8) …
 
Eshell V5.8  (abort with ^G)
1> crypto:start().
ok
2> q().
ok

Installing CouchDB

Download and unpack the current CouchDB version, in my case it’s 1.0:

$ cd /usr/local/src
$ wget http://apache.linux-mirror.org/couchdb/1.0.0/apache-couchdb-1.0.0.tar.gz
$ tar xzf apache-couchdb-1.0.0.tar.gz
$ cd apache-couchdb-1.0.0

Here you might like to see the config options to and export some of the variables:

$ ./configure -h
$ export ERL=/usr/local/bin/erl
$ export ERLC=/usr/local/bin/erlc
$ ./configure --with-erlang=/usr/local/lib/erlang/usr/include/ 
$ make
$ sudo make install

For security reasons it makes sense to have a separate couchdb user that runs the process, go ahead and create it:

$ sudo adduser --system \
        --home /usr/local/var/lib/couchdb --no-create-home \
        --shell /bin/bash \
        --group --gecos "CouchDB" couchdb

You will also have to give this user access to some directories:

$ sudo chown -R couchdb:couchdb /usr/local/etc/couchdb
$ sudo chown -R couchdb:couchdb /usr/local/var/lib/couchdb
$ sudo chown -R couchdb:couchdb /usr/local/var/log/couchdb
$ sudo chown -R couchdb:couchdb /usr/local/var/run/couchdb

… and change the permissions for the directories:

$ chmod -R 0770 /usr/local/etc/couchdb
$ chmod -R 0770 /usr/local/var/lib/couchdb
$ chmod -R 0770 /usr/local/var/log/couchdb
$ chmod -R 0770 /usr/local/var/run/couchdb

After that, verify that it finally works:

$ couchdb
Apache CouchDB is running as process …, time to relax.
$ curl -X GET http://127.0.0.1:5984/
{"couchdb":"Welcome","version":"1.0.0"}
$ /usr/local/etc/init.d/couchdb start
 * Starting database server couchdb
   ...done.

Caveats

If CouchDB does not work and it does not write into the log it is likely a permission problem, see if the couchdb directories belong to the couchdb user and that the permissions are set correctly.

Another thing that occured to me was an error that resulted from still having leftovers from previous installations at /usr/lib/couchdb/erlang/lib. Older version, especially of mochiweb-r97 or couchdb can lead to “Bad Request” errors, so clean up the erlang lib directory.

Relational Data, Document Databases and Schema Design

2010-07-05T00:00:00+00:00

By now it should be obvious that I'm quite fond of alternatives data stores (call them NoSQL if you must). I've given quite a few talks on the subjects recently, and had the honor of being a guest on the (German) heise Developer Podcast on NoSQL.

There's some comments and questions that pop up every time alternative databases are being talked about, especially by people deeply rooted in relational thinking. I've been there, and I know it requires some rethinking, and also am quite aware that there are some controversial things that basically are the exact opposite of everything you learned in university.

I'd like to address a couple of those with some commentary and my personal experience (Disclaimer: my experience is not the universal truth, it's simply that: my experience, your mileage may vary). When I speak of things done in practice, I'm talking about how I witnessed things getting done in Real Life™, and how I've done them myself, both good and bad. I'm focussing on document databases, but in general everything below holds true for any other kind of non-relational database.

It's easy to say that all the nice features document databases offer are just aiming for one thing, to scale up. While that may or may not be true, it just doesn't matter for a lot of people. Scaling is awesome, and it's a problem everyone wants to solve, but in reality it's not the main issue, at least not for most people. Also, it's not an impossible thing to do even with MySQL, I've had my fun doing so, and it sure was an experience, but it can be done.

It's about getting stuff done. There's a lot more to alternative databases in general, and document databases in particular, that I like, not just the ability to scale up. They simply can make my life easier, if I let them. If I can gain productivity while still being aware of the potential risks and pitfalls, it's a big win in my book.

What you'll find, when you really think about it, is that everything below holds true no matter what database you're using. Depending on your use case, it can even apply to relational databases.

Relational Databases are all about the Data

Yes, they are. They are about trying to fit your data into a constrained schema, constrained in length, type, and other things if you see fit. They're about building relationships between your data in a strongly coupled way, think foreign key constraints. Whenever you need to add data, you need to migrate your schema. That's what they do. They're good at enforcing a set of ground rules on your data.

See where I'm going with this? Even though relational databases tried to be a perfect fit for data, they ended up being a pain once that data needed to evolve. If you haven't felt that pain yet, good for you. I certainly have. Tabular data sounds nice in theory, and is pretty easy to handle in Excel, but in practice, it causes some pain. A lot of that pain stemmed from people using MySQL, yes, but take that argument to the guy who wrote it and sold it to people as the nicest and simplest SQL database out there.

It's easy to get your data into a schema once, but it gets a lot harder to change the schema and the data into a different schema at a later point in time. While data sticks around, the schema evolves constantly. Something relational databases aren't very good at supporting.

Relational Databases Enforce Data Consistency

They sure do, that's what they were built for. Constraints, foreign keys, all the magic tricks. Take Rails as a counter-example. It fostered the idea that all that stuff is supposed to be part of the application, not the database. Does it have trade-offs? Sure, but it's part of your application. In practice, that was correct, for the most part, although I can hear a thousand Postgres users scream. There's always an area that requires constraints on the database level, otherwise they wouldn't have been created in the first place.

But most web applications can live fine without it, they benefit from being free about their data, to shape it in whichever way they like, adding consistency on the application level. The consistency suddenly lies in your hands, a responsibility not everyone is comfortable with. You're suddenly forced to think more about edge cases. But you sure as hell don't have to live without consistent data, quite the opposite. The difference is that you're taking care of the consistency yourself, in terms of your use case, not using a generic one-fits-all solution.

Relationships between data aren't always strict. They can be loosely linked, what's the point of enforcing consistency when you don't care if a piece of data still exists or not? You handle it gracefully in your application code if you do.

SQL is a Standard

The basics of SQL are similar, if not the same, but under the hood, there's subtle differences. Why? Because under the hood, every relational database works differently. Which is exactly what document databases acknowledge. Every database is different, trying to put a common language on top will only get you so far. If you want to get the best out of it, you're going to specialize.

Thinking in Map/Reduce as CouchDB or Riak force you to is no piece of cake. It takes a while to get used to the ideas around it and what implications it has for you and your data. It's worth it either way, but sometimes SQL is just a must, no question. Business reporting can be a big issue, if your company relies on supporting standard tools, you're out of luck.

While standards are important, in the end it's important what you need to do with your data. If a standard gets in your way, how is that helpful? Don't expect a standard query language for document databases any time soon. They all solve different types of problems in different ways, and they don't intend to hide that from you with a standard query language. If on the other hand, all you need is a dynamic language for doing ad-hoc queries, check out MongoDB.

Normalized Data is a Myth

I learned a lot in uni about all the different kinds of normalization. It just sounded so nice in theory. Model your data upfront, then normalize the hell out of it, until it's as DRY as the desert.

So far so good. I noticed one thing in practice: Normalized data almost never worked out. Why? Because you need to duplicate data, even in e-commerce applications, an area that's traditionally mentioned as an example where relational databases are going strong.

Denormalizing data is simply a natural step. Going back to the e-commerce example, you need to store a lot of things separately when someone places an order: Shipping and billing address, payment data used, product price and taxes, and so on. Should you do it all over the place? Of course not, not even in a document database. Even they encourage storing similar data to a certain extent, and with some of them, it's simply a must. But you're free to make these decisions on your own. They're not implying you need to stop normalizing, it still makes sense, even in a document database.

Schemaless is not Schemaless

But there's one important thing denormalization is not about, something that's being brought up quite frequently and misunderstood easily. Denormalization doesn't mean you're not thinking about any kind of schema. While the word schemaless is brought up regularly, schemaless is simply not schemaless.

Of course you'll end up with having documents of the same type, with a similar set of attributes. Some tools, for instance MongoDB, even encourage (if not force) you to store different types of documents in different collections. But here's the kicker, I deliberately used the word similar. They don't need to be all the same across all documents. One document can have a specific attribute, the other doesn't. If it doesn't, just assume it's empty, it's that easy. If it needs to be filled at some point, write data lazily, so that your schema eventually is complete again. It's evolving naturally, which does sound easy, but in practice requires more logic in your application to catch these corner cases.

So instead of running migrations that add new tables and columns, and in the end pushing around your data, you migrate the data on the next access, whether that's a read or a write is up to your particular use case. In the end you simply migrate data, not your schema. The schema will evolve eventually, but first and foremost, it's about the data, not the constraints they live in. The funny thing: In larger projects, I ended up doing the same thing with a relational database. It's just easier to do and gentler on the load than running a huge batch job on a production database.

No Joins, No Dice

No document database supports joins, simple like that. If you need joins, you have two options: Use a database that supports joins, or adapt your documents so that they remove the need for joins.

Documents have one powerful advantage: It's easy to embed other documents. If there's data you'd usually fetch using a join, and that'd be suitable for embedding (and therefore oftentimes: denormalizing), there's your second option. Going back to the e-commerce example: Whereas in a relational database you'd need a lot of extra tables to keep that data around (unless you're serializing it into single column), in a document database you just add it as embedded data to the order document. You have all the important data one in place, and you're able to fetch it in one go. Someone said that relational databases are a perfect fit for e-commerce. Funny, I've worked on a market platform, and I've found that to be a ludicrous statement. I'd have benefited from a loser data storage several times, joins be damned.

It's not always viable, sure, and it'd be foolish to stick with a document database if that's an important criterion for your particular use case, then no dice. It's relational data storage or bust.

Of course there's secret option number three, which is to just ignore the problem until it's a problem, just by going with a document database and see how you go, but obviously that doesn't come without risks. It's worth noticing though that Riak supports links between documents, and even fetching linked documents together with the parent in one request. In CouchDB on the other hand, you can emit linked documents in views. You can't be fully selective about the document data you're interested in, but if all you want is fetch linked documents, there is one or two ways to do that. Also, graph databases have made it their main focus to make traversal of associated documents an incredibly cheap operation. Something your relational database is pretty bad at.

Documents killed my Model

There's this myth that you just stop thinking about how to model your data with document databases or key-value storage. That myth is downright wrong. Just because you're using schemaless storage doesn't mean you stop thinking about your data, quite the opposite, you think even more about it, and in different ways, because you simply have more options to model and store it. Embedding documents is a nice luxury to have, but isn't always the right way to go, just like normalizing the crap out of a schema isn't always the way to go.

It's a matter of discipline, but so is relational modelling. You can make a mess of a document database just like you can make a mess of a relational database. When you migrate data on the fly in a document database, there's more responsibility in your hands, and it requires good care with regards to testing. The same is true for keeping track of data consistency. It's been moved from the database into your application's code. Is that a bad thing? No, it's a sign of the times. You're in charge of your data, it's not your database's task anymore to ensure it's correct and valid, it's yours. With great power comes great responsibility, but I sure like that fact about document databases. It's something I've been missing a lot when working with relational databases: The freedom to do whatever the heck I want with my data.

Read vs. Write Patterns

I just like including this simply because it always holds true, no matter what kind of database you're using. If you're not thinking about how you're going to access your data with both reads and writes, you should do something about that. In the end, your schema should reflect your business use case, but what good is that when it's awkward to access the data, when it takes joins across several tables to fetch the data you're interested in?

If you need to denormalize to improve read access, go for it, but be aware of the consequences. A schema is easy to build up, migrating on the go, but if document databases force you to do one thing, and one thing only, it's to think about how you're reading and writing your data. It's safe to say that you're not going to figure it all out upfront, but you're encouraged to put as much effort into it as you can. When you find out you're wrong down the line, you might be surprised to find that they make it even easier to change paths.

Do your Homework

Someone recently wrote a blog post on why he went back to MySQL from MongoDB, and one of his reasons was that it doesn't support transactions. While this is a stupid argument to bring up in hindsight, it makes one thing clear: You need to do research yourself, noone's going to do it for you. If you don't want to live up to that, use the tools you're familiar with, no harm done.

It should be pretty clear up front what your business use case requires, and what tools may or may not support you in fulfilling these requirements. Not all tool providers are upfront about all the downsides, but hey, neither was MySQL. Read up, try and learn. That's the only thing you can do, and noone will do it for you. Nothing has changed here, it's simply becoming more obvious, because you suddenly have a lot more options to work with.

Polyglot Data Storage

Which brings me to the most important part of them all: Document databases (and alternative, non-relational data stores in general) are not here to replace relational databases. They're living alongside of them, with both sides hopefully somewhat learning from each other. Your projects won't be about just one database any more, it's not unlikely you're going to end up using two or more, for different use cases.

Polyglot persistence is the future. If there's one thing I'm certain of, this is it. Don't let anyone fool you into thinking that their database is the only one you'll need, they all have their place. The hard part is to figure out what place that is. Again, that's up to you to find out. People ask me for particular use cases for non-relational databases, but honestly, there is no real distinction. Without knowing the tools, you'll never find out what the use cases are. Other people can just give you ideas, or talk about how they're using the tools, they can't draw the line for you.

Back to the Future

You shouldn't think of it as something totally new, document databases just don't hide these things from you. Lots of the things I mentioned here are things you should be doing anyway, no matter if you're using a relational or a non-relational data store. They should be common sense really. We're not trying to repeat what went wrong in history, we're learning from it.

If there's one thing you should do, it's to start playing with one of the new tools immediately. I shouldn't even be telling you this, since you should hone your craft all the time, and that includes playing the field and broadening your personal and professional horizon. Only then will you be able to judge what use case is a good fit for e.g. a document database. I'd highly suggest starting to play with e.g. CouchDB, MongoDB, Riak or Redis.

Vim - A Never-ending Love Story

2010-06-29T00:00:00+00:00

About eighteen months ago I wrote about going back to Vim as my daily text editor. It was a bust, and I went back to TextMate after about a week.

Suddenly it's the year 2010, and I'm typing this in Vim. What happened? My itch was re-scratched if you will. I was wary of some of TextMate's perceived shortcomings, and honestly I missed having a command and insert mode. It may sound stupid, but I really prefer that way of working with text and code. TextMate is still a nice editor, but seeing its development coming to a perceived halt made me realize that Vim is simply forever, not being developed by just one guy, but a community.

It's also worth mentioning that I simply started from scratch. Last time I built upon a configuration that grew over the years, and that included things about whose purpose I just had no idea. I watched the Smash Into Vim PeepCode too, and started with the clean slate configuration set that comes with it. If you're thinking of getting (back) into Vim, it's highly recommended, it's sure to wet your appetite. There's also a collection of screencasts and a free book on Vim 7 available on the interwebs. I have some useful links in my bookmark collection too.

There's been a lot of developments around scripts for Vim that bring TextMate-like functionality, or that support things like Cucumber, smart quotes and auto-closing braces, or even the most awesome Git integration you'll find. But the nicest of them all is Pathogen, a script that allows you to keep all your other scripts in separate places, not losing overview of what's installed where, and in which version.

Coming from TextMate, you're gonna miss the "Go To File" dialog, I'm sure. Check out Command-T, which does exactly that, only with path-matching sprinkled on top. It's not as fast unfortunately, but a lot faster to use than the annoying fuzzy thing I used the last time I tried to live on Vim. There's also PeepOpen, but it always opens files in new tabs, and that can get quite annoying, as new Vim tabs are quite different from Vim buffers. For project views I use NERDtree, though LustyExplorer also seems acceptable.

As I said, I started from scratch, with a clean slate. So the decent thing to do was to put all my Vim configuration files on GitHub. They include all the scripts I'm using, and my configuration, all neatly separated into different bundles thanks to Pathogen. There's a couple of things that are still a bit wonky. Lusty Juggler doesn't work as advertised all the time, though it's a neat tool, allowing you to quickly select one of a list of the latest open buffers. RubyTest is quite weird, and I'm thinking of dumping it completely, and simply rolling my own commands to run tests based on it. The rails.vim script package does include some support to run tests too, but not to execute a single test case.

In general, I haven't found anything that works in TextMate that you can't somehow get to work in Vim. Yes, I've used the word somehow. It's not easy as pie all of the time, and it can be different, heck it's a different editor. But I willingly accept that, because as a text editor, I find Vim to be a lot better than TextMate.

I've been back on Vim for a month now, and I'm not looking back at all. It's like coming back to an old friend and learning what awesome things he's been up to. It's pretty much as exciting as playing with new technologies at the moment. Learning new things can be pretty exciting, even if it's just another text editor. But it's not all fun and giggles. I have some annoyances still, but no editor is perfect. I'm more willing to accept Vim's for the increased text surgeon skills than TextMate's, to be frank. TextMate is still a nice editor, don't get me wrong, my heart just always belonged to Vim.

Honestly, I'm more willing to invest my learning time in an editor that I know I can use everywhere than one I can only use on the Mac with a running user interface. I'm using Vim on every server I'm managing, so why not on my local machine? Vim makes me think about how I can edit text in the most efficient way possible, and I like that very much. It even made me map my caps-lock key to control, finally!

Update: Was just tipped off that PeepOpen can be made to behave properly and open files in the current MacVim tab. When you set your MacVim options like in the picture below (notice the part "Open files from applications"), it works a treat. Thanks, Mutwin!

Multitouch Inspector for iPad

2010-06-24T20:23:48+00:00

During the last couple weeks I’ve been playing around with the iPad and Mobile Safari. I built a little tool to familiarize myself with the Multitouch JavaScript API provided by Mobile Safari as well as web applications for the iPad in general. I named the result Multitouch Inspector because that’s what it does: Inspect the TouchEvents fired by the JavaScript API. ;-) Today I decided to rewrite the tool to drop the dependency on Prototype.js and I published it on GitHub.

Using Multitouch Events from JavaScript

There are four events that are related to touch: TouchStart, TouchMove, TouchEnd and TouchCancel. I’m not sure about what situation would trigger TouchCancel so I decided to skip it for now. Working with the touch events is straight forward:

   1  document.addEventListener('touchstart', function(event) {
   2    /* Do whatever you'd like to do with the event */
   3  }, false);

Of course you can add the listener to any element just like you would with Click or MouseOver events. The function gets passed an object of type TouchEvent. There are several properties on this object, but the most interesting ones are touches, changedTouches and targetTouches. They all are of type TouchList and contain several Touch objects. The touches property lists all touches currently on the screen. The changedTouches list contains the touches that changed and caused the event to fire. The touches in the targetTouches list are those that are currently within the target element.

Every Touch has an identifier property as well as pageX and pageY properties. As you might have guessed already, the pageX and pageY properties include the touch’s position on the screen. The identifier property provides an unique integer for this touch. It stays the the same for this touch as long it is on the screen. This is particularly useful as removing one finger will trigger a TouchEnd event that implies that all fingers were removed, immediately followed by a TouchStart event including the remaining fingers. Luckily the identifier property stays the same for those fingers that weren’t removed from the screen.

Offline Application Caching

In order to use the application without having an active internet connection or simply while the development server isn’t running I’m using HTML5 Offline Application Caching. It works by defining a manifest file and referencing it in the html-tag:

   1  <!DOCTYPE html>
   2  <html manifest="application.manifest">

The manifest file itself looks like this and defines what files are required to view the application while offline:

   1  CACHE MANIFEST
   2  
   3  # c94640e9114e05f16e189605e5b65ba2357117712c949cae92cc29bc1bbd3c47
   4  /images/background.png
   5  /images/icon.png
   6  /index.html
   7  /javascripts/application.js
   8  /stylesheets/application.css

You might wonder about the random string at the top. As the browser will reload everything when it can’t find one file in it’s cache, I’m using this string to force a reload during development. I built a small Sinatra app (see the listing below this paragraph) that generates the manifest and resets this string for every request. As a result, the browser reloads everything while online but falls back to the cached files when offline.

   1  require 'sinatra'
   2  require 'digest/sha2'
   3  require 'pathname'
   4  
   5  # ...
   6  
   7  get '/application.manifest' do
   8    content_type 'text/cache-manifest'
   9  
  10    manifest = "CACHE MANIFEST\n\n"
  11    manifest << "# " << Digest::SHA2.hexdigest(Time.now.to_s + Time.now.usec.to_s) << "\n"
  12  
  13    root = Pathname.new(settings.public)
  14    Pathname.glob(File.join(root, "**", "*")).each do |p|
  15      manifest << "/" << p.relative_path_from(root) << "\n" if p.file?
  16    end
  17  
  18    manifest
  19  end

Issues

There are some issues that I couldn’t resolve while building the application. The startup image will only work while in portrait orientation and its dimension has to be exactly 1004×768. Currently there isn’t a way to define a startup image for horizontal orientation.

While the tracking of the touches works on the iPhone (with iOS 4.0) as well for some reason it isn’t possible to press the buttons in the toolbar. At the moment I have no explanation for this rather strange behavior, but I might take another look at it in the future.

Screenshot & Demo

Now that you have some insights on the internals of the application, here’s a screenshot as well as a link on it to a demo:

Fork it on GitHub

I published the source code on GitHub. Feel free to fork it!

Presentation Fu

2010-06-24T00:00:00+00:00

I've attended my fair share of conferences this month alone, plus a Seedcamp, and I can safely say that in any way, I learned a lot about how to build slides, how to keep the audience engaged and things one just shouldn't do in a talk or in slides. While I certainly don't claim to be an expert on the topic now, I just wanted to put all of my impressions and lessons learned into a post.

I'm definitely not the first person to write about this kind of stuff, a year ago Geoffrey Grosenbach wrote on presenting, and just recently John Nunemaker wrote a post on improving your presentations for less then $50. Both are well worth reading, but they don't cover everything I find annoying in presentations, so there you go.

Slides

Keep them small

Seven bullet points per slide is bullshit, that's way too much. One phrase per slide is a decent rule, though I'm not dogmatic about it. One phrase and a couple of short bullet points (not more than four) work from time to time, but not all the time. I usually go for a bigger slide set these days, with less content on each slide.

I can run through 80 slides in 45 minutes. I know that sounds like a lot, and I certainly go through them fast, but I'd rather give people something to think about than bore them to death. Slides with too much text on it also have the negative effect of distracting the audience. They shouldn't read the slide text, they should be listening to what you have to say. Even if you do talk slow, less text on slides is always a good idea. The people should listen to you, not try to understand what your slides are saying.

What I usually do is just crank out slides with any text that I'd like to say, and then I go through them one or two times to refine and shorten the prases I used to be no more than four or five words for the most part. I also throw out slides when I realize they're disrupting the flow or contain things I'm likely to talk about when I'm on a different slide.

Use a large font

Just do it. Not only does it make your slides more readable for everyone in the audience, it forces you to keep the information on a single slide short. My headlines are usually 60pt, my subheadings and bullet points around 45pt. The bigger the better.

While we're talking about fonts, avoid italic. It's a lot harder to read, especially when you mix it with a regular font. If you need to emphasize something, just make it bold. Italic fonts disrupt your slides' flow.

Avoid full sentences

Except when you're quoting someone. Short phrases or even just a single word are much easier to grasp for the audience, and they give you a better sense of flow.

Dark text on a bright background

A dark background only works for Steve Jobs, because his team does everything they can to adjust the lighting on location for his talk. You on the other end, have to assume the worst. If there's just a little too much light coming into the room, your slides will be unreadable, when you use a dark background. I've even seen slides where people chose a dark background and just a slightly dark font.

You have no influence on the lighting in the room, and you'll pretty much just embarrass yourself when your slides are unreadable. There's just no excuse why you shouldn't just use a light background and a dark font.

Avoid dark photos

Photos are at a similar risk. The more contrast you have in photos you're using in your preso, the less likely people will be able to see them. I tend to not use a lot of photos in my slides anyway, but I just hate having to say: "Geee, that's a bit hard to see, isn't it?"

Slides are for the people attending the talk

Your slide set should not be focussed on being fully understandable by people who have not attended your talk. You end up with so called slideuments, presentations that read like a document. You're talking for the people attending your talk, they probably paid to hear you speak, so focus your energy on giving them a good talk. If you want the rest of the world to know about details of your preso, write a blog post or put it into the presenter notes.

Video killed the conference star

I've seen video in presentations quite a few times, and honestly, it bores me to death, especially when there's a voiceover on the video. If you must include video, at least talk yourself, taking the audience through whatever happens on the screen, especially because you don't know how the audio is going to be at the venue. I'm well aware that live demos are a finnicky thing, but so is video. Not always do you have the luxury of using your own computer to do the presentation.

Avoid long code snippets

Code is simply hard to grasp within just a couple of seconds, and it's awkward trying to explain larger chunks of it. Use short snippets instead. If you must include some longer examples, split it up in smaller bits, explaining them one by one. I tend to avoid overly complex code snippets. Trying to explain them properly just takes too much time.

Avoid flashy animations

They simply take up valueable time and distract the audience. Even though they're nice to look at in theory, in practice they're the bane of a well-built presentation. This is true for both transitions between slides and elements of a single slide appearing later. Just make them appear, not sparkle or fade in.

The Talk

Practice, practice, practice

I find practicing a talk by speaking to myself awkward, not because it's embarrassing, but simply because of the butterflies in my stomach I always end up saying different things in the actual talk. Now, that's not to say you shouldn't think about what you want to say. I tend to go through my slides several times, going through the things I associate with every single one of them, giving me a rough idea and a line of thought on what I want to say. This definitely is a lot easier to do when it's a topic you've talked about before, but in general the above has worked much better for me.

Drink, drink, drink

It's a simple fact that talking a lot lets your mouth run dry. I need about half a liter of water to get through a talk. Or at least I make sure I have that amount ready. Before you run dry and faint in the midst of your talk, drink, it's not a shameful thing to do, it simply keeps you going. Shame on conference organizers not thinking about having drinks ready for their speakers. When in doubt, scout the talks before you and make sure you have a bottle ready should it not being taken care of.

Look at the audience, not the big screen

It should be so obvious, yet I've just seen people do it again at Cloud Expo. One of the guy's slides had 14 bullet points on it, and the font probably was too small for him to be able to read it from the laptop screen. Another reason why I keep my slides short, they're purpose is to keep me in a flow, to give me short reminders of what I want to talk about.

Don't read your presenter notes

If you need presenter notes to run your talk, you need to practice more. They're surely useful for people just looking at your slides, but if it takes full sentences to keep your talk running, you'll end up wasting a lot of time trying to read what your notes say. Talking freely is a challenge, but the earlier you take it on, the faster you'll get used to it. I've seen people use index cards with their presenter notes on them, handwritten, trying to decipher what they've written on them.

If you know what you're talking about (at least the slightest bit), you'll be fine without them, trust me.

Two's not a company

Having more than one speaker is awkward, especially when one of them is just standing there for most of the time, waiting for his turn. Have one up in front at any one time, bring in the next person when it's his turn. Simple like that.

Don't ask questions

The audience simply won't answer. If you ask anything, make the audience raise their hands on a topic, but don't expect anyone to answer a specific question. That's your task. Involving the audience sounds like a good idea, but they're lazy, they want to learn something.

Jokes, tiny bits and stories

Stories and jokes can really lighten up a presentation. Sure, you shouldn't tell jokes all the time, but something sarcastic thrown in from time to time sure can help to wake up the audience. Stories are even better, people love benefitting from real life experiences in any way. If it has a happy ending, even better.

Talking slowly is for wimps

The rule of spending two minutes on a slide is bullshit. It would only mean you'd have seven bullet points on a particular slide. You shouldn't rush through anything, and I certainly try to avoid doing that, and it definitely depends on the topic you're talking about, but when I talk about technical things I expect the audience to be curious about it and try to keep up. If they can't, they can always come back to my slides or ask questions. But as always, it depends.

Talking fast is for the impatient

If it's on more generic things that involve higher level topics, or some sort of longer-running workshop, it's only appropriate to walk the people through it and take your time doing so. Usually in these situations it's a lot easier to focus on a single topic. It just depends on how broad your talks topic is.

Take tiny breaks

Should you realize you're sort of losing track, simply bring yourself back on the rails. Take a tiny break or just stop talking. You don't need to apologize for that. It's easy to start blabbering on about a certain topic which you didn't even intend to cover in your talk. On the other hand, that's what makes every talk unique, and is exactly why shorter phrases on slides are so much better. They keep your brain engaged, making up associations with certain things as you go, and they help keeping a talk interesting.

Avoid longer breaks though as people end up being bored, and you're losing precious time. Longer breaks are usually a sign that you're not as prepared as you should be. If you need to switch in between e.g. slides and a live demo, make sure that everything is prepared before the talk.

Talking in front of others is a challenge, no doubt about it, but there's really no point trying to avoid it, because the only way to improve your skills is to simply talk in front of people. This is my view of the talking world. I constantly try to improve on my slides and think about what I'm doing wrong during talks to improve on that. I'll never loose the excitement right before a talk, and that's a good thing. When it becomes routine, you tend to bore people instead of engaging them. It's about constantly improving yourself to simply become better at talking in front of others.

This is my view of giving presentations. Feel free to throw in your ideas, or even to disagree. These guidelines probably aren't for everyone, and they might even change for me within just a couple of months, but most of them simply make sense to me. I do need to get me a good remote though, since with my larger slide sets, I find myself hitting the space bar a lot.

June - A Month of Conferences

2010-06-22T00:00:00+00:00

June was an exhausting month for me. I spoke at four different conferences, two of which were not in Berlin. I finished the last talk today, so time to reciprocate on conferences and talks. In all I had good fun. It was a lot of work to get the presentations done (around 400 single slides altogether), but in all I would dare say that it was all more than good practice to work on my presentation skills and to loose a bit of the fear of talking in front of people. But I'll follow up on that stuff in particular in a later post.

RailsWayCon in Berlin

I have to admit that I didn't see much of the conference, I mainly hung around, talked to people, and gave a talk on Redis and how to use it with Ruby. Like last year the conference was mingled in with the International PHP Conference and the German Webinale, a somewhat web-related conference. I made a pretty comprehensive set of slides for Redis, available for your viewing pleasure.

Berlin Buzzwords in Berlin

Hadoop, Lucene, NoSQL, Berlin Buzzwords had it all. I spent most of my time in the talks on the topics around NoSQL, having been given the honor of opening the track with a general introduction on the topic. I can't remember having given a talk in front of this many people. The room took about 250, and it seemed pretty full. Not tooting my own horn here, I've never been more anxious before a talk of how it would go. Obviously there were heaps of people in the room who have only heard of the term, and people who work with or on the tools on a daily basis. Feedback was quite positive, so I guess it turned out pretty okay. Rusty Klophaus wrote two very good recaps of the whole event, read on about day one and day two.

The slide set for my talk has some 120 slides in all, trying to give a no-fuss overview of the NoSQL ecosystem and the ideas and inspirations. There's some historical references in the talk, because in general the technologies aren't revolutionary, they use ideas that've been around for a while and combine them with some newer ones. Do check out the slides for some more details on that.

MongoUK in London

10gen is running MongoDB related conferences in a couple of cities, one of them in London, where I was asked to speak on something related to MongoDB. Since I'm all about diversity, that's pretty much what I ended up talking about, with a hint of MongoDB sprinkled on top of it. Document databases, the web, the universe, all the philosophical foundation knowledge you could ask for. I talked about CouchDB, Riak, and about what makes MongoDB stand out from the rest.

Most enjoyable about MongoUK was to hear about real life experiences of MongoDB users, what kind of problems they had and such. Also, I finally got to see some of London and meet friends, but I'll write more about that (and coffee) on my personal blog. Again, the slide set is available for your document database comparison pleasure.

Cloud Expo Europe in Prague

Just two 36 hours after I got back from London I jumped on the train to Prague to speak about MongoDB at Cloud Expo Europe. Cloud is something I can get on board with (hint: Scalarium), so why the hell not? It turned out to be a pretty enterprisey conference, but still, got some new food for thought on cloud computing in general.

I already gave a talk on MongoDB at Berlin's Ruby brigade, but I built a different slide set this time, improving on the details I found to be a bit confusing at first. Do check out the slides, if you don't know anything about MongoDB yet, it should give you a good idea.

Showing off

As you'll surely notice, my slides are all websites, and not on Slideshare. Two months ago I looked into Scott Chacon's Showoff, a tool to build web-based presentations that simply run as tiny JavaScript apps in the browser. I very much like that idea, because even though Keynote is still the king of the crop, it's still awful. Using Markdown, CSs and JavaScript appeals much more to the geek in my. It's so easy to crank out slides as simple text, and worry about the styling later. Plus, I can easily keep my slides in Git, and who doesn't enjoy that? I'd very much recommend giving it a go. If you want to look at some sources, all my talks and their sources are available on the GitHubs, MongoDB, Redis, NoSQL, document databases and again MongoDB.

It's a pleasure to build slides with Showoff, and it has helped me focus my slides on very short phrases and as few bullet points as possible. Sure, it's not Keynote and doesn't have all the fancy features, but I noticed that it forced me to focus more, and that keeping slides short helped me stay focussed, but again, more on that in a follow-up post.

Feel free to use my slides as inspiration to play with Showoff, there's surprisingly little magic involved. Also, if you think I should speak at a conference you know of or that you're organising, do get in touch.

Was macht eigentlich… ?

2010-06-10T20:48:36+00:00

Half a year ago I promised to let you know about my experiences becoming a freelancer. Not having written more than a few project announcements since then is actually a good thing, because it means I’m too busy to blog Nevertheless: Here is an update of what I have been doing the last months and about what I plan to do for the rest of the year

After spending the first months completely involved in some iPhone projects and consulting and not doing anything web development related I decided to not take on new iPhone/iPad client projects. There are some reasons for that – first and foremost: Right now the web and mobile market are moving really fast and I just can’t (and don’t want to) find the time to stay up to date with both worlds (it’s appeasing that I’m not the only one who thinks like that ) There is so much exciting stuff happening in the Rails, JavaScript and HTML5 realm that I don’t want to miss and I really missed working in teams, because most of the iPhone gigs were just me on the development side. Another point is that I don’t like the lock in way Apple seems to be pursuing, but this is more of a personal issue. I’d really like to get the personal impression from those of you who are also doing iOS development, maybe it’s just me but I talked to some people who seamed to have similar opinions.

So what happened after making this decision? Something that seems logical looking at it retrospectively: I joined the guys over at Railslove and worked with them on a World Cup prediction game (I’m sorry but I just don’t know how to translate “Tippspiel” – btw: why not join the Coworking Cup? Invitation token is “weworkunited”). We had a very productive time and lots of fun together so we concluded it might be a good idea to work on some more projects and this assumption did not fail to deliver. We are working on some pretty exciting projects and if you are interested in joining Railslove just contact them.

Okay, that’s the past and the present – what about the future? I don’t know, but if it turns out to be as exciting, informative and funny as the first six month it will be awesome! Two weeks ago I’ve been at the RailsWayCon (you might want to read Jan’s recap) and met a lot of the nice folks from the European Ruby/Rails Community. I’m also really looking forward for some more conferences and events this year, like RailsCamp, JSConf EU and RailsRumble to name a few – I hope we’ll see each other there!

Ruby on Rails 3 Security Updated

2010-06-08T12:13:13+00:00

I hold a talk about Rails 3 Security at the RailsWayCon10. It is about the new Cross-Site Scription protection in Rails 3, what is going to change in ActiveRecord and other Rails Security topics. You can find the presentation at Slideshare.

Ruby on Rails Security Updated (Rails 3) at RailsWayCon

View more presentations from heikowebers.

Upcoming Conference Talks

2010-06-01T00:00:00+00:00

June is a busy month when it comes to me speaking at conferences, you should drop by at least one of my talks, if you're in the area.

This week is RailsWayCon in Berlin, and I'll be talking about Redis. Don't tell anyone, but my slides are already up, in case you can't make it.

Next up is Berlin Buzzwords, on June 7th and 8th in Berlin, where I'll be giving a general introduction on NoSQL databases, their history and the current state of affairs. The conference is all about scale, search and data, a definite must.

On June 18th I'll be speaking at MongoUK, about MongoDB and document databases in general. If you want to come, use the coupon code paper_planes for 20% off the full ticket price.

Then, last but no least, Ill be talking about MongoDB in general at Cloud Expo Europe in Prague.

Hope to see you at one of these conferences. Happy to chat about anything from databases to clouds, and to give you a tour of Scalarium.

Conference Talks You Should Be Watching

2010-05-24T00:00:00+00:00

Thanks to the guys from Confreaks, there's a whole bunch of video material from conferences for the rest of us to enjoy. I've been watching quite a few recently, so here's some recommendations of talks I found interesting.

BERT and Ernie: Scaling your Ruby site with Erlang

Talk on the BERT protocol created by Tom Preston-Werner of GitHub.
Bits, Bytes, and BLOBs

Everything on bit shifting, binary protocols and whathaveyou. I'm a big fan of text-based protocols, but it's nice to get some perspective on the world of binary things as well.
NoSQL - Death to Relational Databases

The title is to be taken with a grain of salt obviously, but nonetheless, nice introduction to the most important players of the NoSQL world.
SOLID Ruby

Jim Weirich on applying the SOLID principles to Ruby coding, though it's more of an introduction to the principles in general with a hint of Ruby.
Clojure for Ruby Programmers

The title says it all.
Rippin' off Python

Nice talk on how Python solves some problems compared to Ruby.
Cooking with Chef - Your Servers will thank you

Great introduction to the essence of Chef and how easy it is to write your own cookbooks and recipes. I agree with lots of the things James talks about here, and the their way of using Chef comes pretty close to how we're using it on Scalarium.

The gist of all of them for me: They're a lot less about Ruby or even Rails. Personally I'm pretty bored of hearing yet another talk on these topics, or stuff like Cucumber. Talks that expand my horizon beyond what people do with Rails have become a lot more interesting to me.

ActiveRecord's Callbacks Ruined My Life

2010-05-07T00:00:00+00:00

Recently I've been having a foul taste in my mouth, or just a bad feeling, if you will. Whenever I started adding validations and callbacks to a model in a Rails application. It just felt wrong. It felt like I'm adding code that shouldn't be there, that makes everything a lot more complicated, and turns explicit into implicit code. Code that is only being run depending on the persistence state of an object. Code that is being hard to test, because you need to save an object to test parts of your business logic. Don't get me started on observers, I never was a fan of them. Putting stuff that should be run when an object was saved elsewhere is the worst kind of hiding business logic.

The two are somewhat bound together: Create an object, run a callback and validations. Update an object, check if an attribute is set, run a different callback. I started to loathe the lack of implicitness that callbacks gave my code, and I started to loathe that libraries everywhere copied that behaviour from ActiveRecord, and that you involve testing your database in your unit tests, though implicitly, and that noone had a better answer for how that stuff should work. Sure, it's not big news that involving your database in your unit tests is not a recommended practice, hence the overuse of mocking and stubbing fostered by RSpec, but it took a while for me to really feel the pain.

Ingredients: REST, Rails, Resources

The root problem (in a very philosophical way and in my very own opinion) lies in Rails' understanding of RESTful resources. The path to a resource is supposed to be thin, and a resource is mostly represented by something that's stored in a database. So far, so good. That thin layer however, the controller, implies that running any logic on a resource somehow ends up in simply updating the resource, setting some attribute signaling that an action needs to be taken. That's how tools like resource_controller, make_resourceful and inherited_resources came about, to get rid of the thin-layered controller tier, the boiler plate code. And I loved them, because I hated writing the same code over and over again.

But over time I realized that controllers aren't the problem, it's the way Rails treats resources that lead us to this mess where everything's declared in a model's class body, where you have to look at callbacks to find out what's going on when you create a new object, and where you clutter your models with code that just doesn't need to be there (sending mails, for example, or doing custom validations that are bound to specific state). It all started to lack explicitness for me. There's no clear path of entry, no list of method calls you can follow along.

I'm not sure how other people see this, obviously a lot of people don't realize it's a problem, or they're simply fine with how things work, I was for a long time too, almost five years now, to be exact. I know I'm not alone in this though, talking to others made me realize they're on a similar page, but given that it's the way Rails works, what can you do?

Validations

What about validations? I can live with validations being part of the model, but they're still part of the problem generated by callbacks, simply because validations are callbacks themselves. I always found myself ending up writing custom validations in before_validation et. al. hooks, and it just felt wrong, because the list of callbacks grew noticably. Validations are not always the same depending on your object's state. They can even be quite useless when you're handling a lot of backend code that's asynchronous and decoupled from the web interface.

What do we do to work around it? We resort to using save(false), whose lack of explicitness I cannot even begin to fully explain in a peaceful manner. Validations tend to depend on the object's state, at least more complex ones. They're not universal, they can be different when creating and saving objects. Yes, I know that in ActiveRecord you can say :on => :create, but for me that just emphasizes my point. Persistent state determines callbacks and therefore business logic, decreasing expressiveness.

Forms? Presenters?

I've looked into how Django is dealing with this, and it sort of struck a chord with me. Their approach is to define form objects that do the validations for you. They're somewhat decoupled from your business logic and are your intermediate layer between the web interface and the business model. Good lord, did I just recommend we all use presenter objects, even though Jay Fields said they don't work? I think I did, Bob!

It's not the way, but it's one way, and there's been work on bringing this to Rails, have a look at Bureaucrat for example. Again, it's one way to try solving the problem, it doesn't come without any downsides too. Tying the idea of explicit services to form data may or may not make unit testing harder, or at least mix two different concerns (forms and creating/manipulating objects). What I like about it though is that it decouples validation from the business logic, and ties it to the layer it's really necessary, the view.

Not without irony, I've started using something similar to handle embedded objects on a project using CouchDB. I wanted to extract out the stuff handling with embedded objects, because I was simply to lazy to add support for that to CouchPotato. The result felt amazingly good, almost like the above forms mixed with a *gasp* service.

Services?

I'm still seeking for the answer to that one. Pat Maddox and James Golick suggest services as one way to make callbacks explicit, be sure to read through the comments on James' post. Now, I don't know about your history, but coming from a Java background that struck a bad cord with me. I laughed off the idea, to be frank.

One thing still bugs me about the approach James is laying out in his post, it's the extensive use of mocking. It sure does make for fast unit tests, and I'm starting to see the sense in it when using a service model, but I've grown fond of the fakes approach, which is why we wrote Rocking Chair, and why people came up with things like SuperModel, in-memory database implementations you can easily switch on and off, not exactly decoupling your unit tests from the persistence layer, but decoupling them from having a database running, making it easier to e.g. split batches of tests across several cores and processors, and simply making them faster.

However, the more I thought about it, the more sense the services approach made. You don't have to necessarily make it a separate class, depending on how much you like having all of your business logic in the model. I'm not sure which approach I'd prefer, because honestly, I haven't tried it yet, but I'm willing to give in. I'm willing to try things that let me ditch callbacks and validations, because I simply loathe them, they just don't feel right to me anymore when used in the way ActiveRecord does.

With Rails 3 for example, you can put your validations anywhere, they're not just bound to ActiveRecord anymore, they leave a lot more room for you to use them in different ways. I'm not suggesting you should get rid of validations, they're just a part of the callbacks madness, and they tend to not always be the same depending on what you're doing in a particular use case. Using save(false) just started to feel like a bad workaround to me to avoid having callbacks run. I want to save an object no matter what, and run whatever code I require in that course of action, in an explicit fashion.

Are Fine-grained Resources the Future?

The question that's left for me, and this is where I'm starting to gasp question Rails' approach to resources. I'm starting to think they're wrong, and that for the service or the forms model to work, the idea of a single resource you update whenever you want your model to do something needs to change. Having very specific services and their methods would mean a switch to more lightweight and finer-grained resources, specific resources where you e.g. accept an invitation not by PUTting to /invitation/12345, but instead to /invitation/12345/accept.

But, and here's the kicker, to me this leaves the question of how RESTful that approach is. This is where I'd like to get input. How do people using services already solve this problem? Have lots of tiny methods in your controllers? Maybe it's time to work more with Sinatra to get different ideas in my head, to rethink resources. I'm still not certain what a good and RESTful way to approach this would be. Whatever it is, it'd leave my business logic in a way that's a lot more approachable both for people reading it, and for my tests. That's something pretty dear to me, and more important than fully conforming to a REST model. There must be something better, because the current status quo of handling business logic with Rails' resources just doesn't work for me anymore, it doesn't make me a happy programmer.

The answers have yet to be found. My intention is not to rant, but rather to start a discussion around it. In my opinion the way Rails recommends handling resources doesn't have a future, if only for me, and even if that means using other frameworks in the future. I don't think we wasted time with Rails' current way, instead we're going to see an evolution, and I find that pretty exciting, because the future has yet to be written in this regard, and we can be a part of that.

Coworking Cologne offen ab 03.05.2010

2010-04-16T13:16:56+00:00

Es freut uns zu verkünden, dass die Türen vom Coworking Cologne Space offen sind. Wir sitzen schon seit einigen Wochen hier im Space und arbeiten zusammen mit anderen Coworkern am Aufbau vom Space hier in Köln-Deutz.

Wir freuen uns auch auf zahlreichen Besuch ab Anfang Mai.

Testing CSV Download With Cucumber

2010-03-17T17:18:11+00:00

Thanks to Upstream for their blog post on testing pdfs with cucumber.

Testing CSV download is exactly the same apart from that we do not need to turn the PDF to text, simply because…it isn’t a PDF…

(From RedWritesHere)

“Digitaler Schimpfweltmeister”: DISSER für’s iPhone

2010-03-07T13:42:46+00:00

Als Stephan Griesenbrock und Daniel Knobloch mir Ende letzten Jahres die Idee für ihre DISSER App für’s iPhone (und iPod Touch) vorstellten war mir schon klar, dass die App ein sehr großes Potential hat: Ein schlüßiges Konzept um eine witzige und bislang so nicht umgesetzte Idee, eine gut zu erreichende Zielgruppe und angebrachte Sharing-Funktionen für soziale Netzwerke, so dass auch der Grundstein für eine virale Verbreitung gelegt ist. Aber damit, dass die App direkt so einschlagen würde, hat von uns dreien selbst bei dem guten Feedback das wir von Freunden bekommen haben keiner gerechnet.

Wir schreiben heute Tag 4 nach Livegang im App Store und der Disser hat sich bis ganz nach oben an Position 1 der meistverkauften Apps in Deutschland gepöbelt. Der helle Wahnsinn, vor allem weil es so schnell ging – was aber wiederum der sehr Marketingarbeit der zwei Jungs aus Köln zu verdanken ist: Rund um den Disser wurde vor dem Livegang ein umfassendes Gesamtpaket inklusive Website, Projektblog, Facebook Fan-Page, Marketing-Video und Pressematerial geschnürt, so dass die App nicht nur auf dem iPhone stattfindet. Schnell trudelten die ersten sehr guten Rezensionen und Video-Reviews ein: So schreibt beispielsweise iPhone-Junkie Carsten Scheibe im Stern.de Blog: “Der Disser ist eine überaus professionell umgesetzte Fun-App, die auf jedes iPhone und auf jeden iPod touch gehört.”

Auch das Feedback der Nutzer im App Store und auf der Website sind super: Viele wollen den Wortschatz des Dissers erweitert sehen und schlagen selbst neue Wörter vor, so dass sich ein sehr kuriose und amüsante Liste von Vorschlägen ansammelt, die wir möglichst schnell mit einem neuen Update herausgeben werden. Wir sind gespannt auf die nächsten Tage, freuen uns über weitere Reviews und Vorschläge und bedanken uns bei allen Nutzern für’s Herunterladen der App – sowas macht echt Spaß!

SimplyStored and CouchDB

2010-03-05T09:10:39+00:00

Yesterday I gave a presentation about CouchDB and SimplyStored, our convenience Ruby library, at the Ruby User Group Berlin.

There is a recording of the presentation at ustream.tv.

Mathias and I wrote SimplyStored in order to easily interact with Ruby objects serialized in CouchDB. We use CouchDB as the main data store for Scalarium and so far it has been great. But it is a bit cumbersome to write all those map and reduce functions yourself.

SimplyStored generates the JavaScript map&reduce functions for handling associations or dynamic finders for you.

SimplyStored offers:

Models
Associations
Callbacks
Validations
Dynamic finder
S3 attachments
Paranoid delete

    class User
      include SimplyStored::Couch

      property :login
      property :age
      property :accepted_terms_of_service, :type => :boolean
      property :last_login, :type => Time
    end

    user = User.new(:login => 'Bert', 
                    :age => 12, 
                    :accepted_terms_of_service => true, 
                    :last_login = Time.now)
    user.save

    User.find_by_age(12).login
    # => 'Bert'

    User.all
    # => [user]

    class Post
      include SimplyStored::Couch

      property :title
      property :body

      belongs_to :user
    end

    class User
      has_many :posts
    end

    post = Post.create(:title => 'My first post', 
                       :body => 'SimplyStored is so nice!', 
                       :user => user)

    user.posts
    # => [post]

    Post.find_all_by_title_and_user_id('My first post', user.id).first.body
    # => 'SimplyStored is so nice!'

    post.destroy

    user.posts(:force_reload => true)
    # => []

The code is on github and OpenSource: SimplyStored example code

Another thing I talked about is RockingChair. RockingChair is an in-memory CouchDB implementation that understands all of SimplyStored's functionality. We use it to speed up our tests and be able to run them in parallel.

Ruby on CouchDB - SimplyStored and RockingChair

View more presentations from Jonathan Weiss.

Frozen Rails 2010

2010-03-05T08:54:23+00:00

Although Berlin looks a bit like Antarctica right now, I'm really looking forward to travel to Helsinki, Finland for Frozen Rails on May 7th.

Forzen Rails is organized by the guys from the HHLinuxCliub and Kisko Labs. I've met them in Amsterdam back in October and we had a great time. I'm sure Frozen Rails will be too.

They have a great line-up organized with speakers like Yehuda Katz, Chris Wanstrath, Joseph Wilk, Jarkko Laine, Mike Dirolf, or Nizar Jouini. The ticket prices are reasonable, starting with 99 Euro for the early bird tickets. They even have a discount scheme that will give you a free ticket if you get 5 friends to sign-up.

Regional conferences like this are always a lot of fun as you meet a lot of local developers and the organizers are very passionate.

I hope to meet you at Frozen Rails 2010 in Helsinki.

Duplicate entries in resultset of a hibernate query

2010-02-25T10:43:49+00:00

If you have duplicate entries in a resultset of a hibernate query, then this could be caused by a eager fetchmode. A result is to add a ResulTransformer to your query.

Example:

Criteria criteria = getSession().createCriteria(Order.class);
criteria.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY);

More Infos here: https://www.hibernate.org/117.html#A12

Notes on MongoDB

2010-02-25T00:00:00+00:00

For an article in a German magazine I've been researching MongoDB over the last week or so. While I didn't need a lot of the information I came across I collected some nicely distilled notes on some of its inner workings. You won't find information on how to get data out of or into MongoDB. The notes deal with the way MongoDB treats and handles your data, a high-low-level view if you will. I tried to keep them as objective as possible, but I added some commentary below.

Most of this is distilled knowledge I gathered from the MongoDB documentation, credit for making such a good resource available for us to read goes to the Mongo team. I added some of my own conclusion where it made sense. They're doing a great job documenting it, and I can highly recommend spending time to go through as much of it as possible to get a good overview of the whys and hows of MongoDB. Also, thanks to Mathias Stearn for hooking me up with some more details on future plans and inner workings in general. If you want to know more about its inner workings, there's a webcast coming up where they're gonna explain how it works.

Basics

Name stems from humongous, though (fun fact) mongo has some unfortunate meanings in other languages than English (German for example)
Written in C++.
Lots of language drivers available, pushed and backed by the MongoDB team. Good momentum here.
According to The Changelog Show ([1]) MongoDB was originally part of a cloud web development platform, and at some point was extracted from the rest, open sourced and turned into what it is today.

Collections

Data in MongoDB is stored in collections, which in turn is stored in databases. Collections are a way of storing related data (think relational tables, but sans the schema). Collections contain documents which have in turn keys, another name for attributes.
Data is limited to around 2 GB on 32-bit systems, because MongoDB uses memory-mapped files, as they're tied to the available memory addressing. (see [2])
Documents in collections usually have a similar data structure, but any arbitrary kind of document could be stored, similarity is recommended for index efficiency. Document's can have a maximum size of 4MB.
Collections can be namespaced, i.e. logically nested: db.blog.posts, but the collection is still flat as far as MongoDB is concerned, purely an organizational means. Indexes created on a namespaced collection only seem to apply to the namespace they were created on though.
A collection is physically created as soon as the first document is created in it.
Default limit on number of namespaces per database is 24000 (includes all collections as they're practically the top level namespace in a database), which also includes indexes, so with the maximum of 40 indexes applied to each collection you could have 585 collections in a database. The default can be changed of course, but requires repairing the database if changed on an active instance.
While you can put all your data into one single collection, from a performance point of view, it seems to make sense to separate them into different collections, because it allows MongoDB to keep its indexes clean, as they won't index attributes for totally unrelated documents.

Capped Collections

Capped collections are fixed-size collections that automatically remove aged entries by LRU. Sounds fancier than it probably is, I'm thinking that documents are just appended at the last writing index, which is reset to 0 when limit of the collection is reached. Preferrable for insert-only use cases, updates of existing documents fail when the data size is larger than before the update. This makes sense because moving an object would destroy the natural insertion order. Limited to ~1GB on 32-bit systems, sky's the limit on 64-bit.
Capped collections seem like a good tool for logging data, well knowing that old data is purged automatically, being replaced with new data when the limit is reached. Documents can't be deleted, only the entire collection can be dropped. Capped collections have no indexes on the _id by default, ensuring good write performance. Indexes generally not recommended to ensure high write performance. No index on _id means that walking the collection is preferred over looking up by a key.
Documents fetched from a capped collection are returned in the order of their insertion, newest first, think log tailing.

Data Format

Data is stored and queried in BSON, think binary-serialized JSON-like data. Features are a superset of JSON, adding support for regular expressions, date, binary data, and their own object id type. All strings are stored in UTF-8 in BSON, sorting on the other hand does not (yet), it uses strcmp, so the order might be different from what you'd expect. There's a sort of specification for BSON, if you're into that kind of stuff: [3] and [4]
Documents are not identified by a simple ID, but by an object identifier type, optimized for storage and indexing. Uses machine identifier, timestamp and process id to be reasonably unique. That's the default, and the user is free to assign any value he wishes as a document's ID.
MongoDB has a "standard" way of storing references to other documents using the DBRef type, but it doesn't seem to have any advantages (e.g. fetch associated objects with parent) just yet. Some language drivers can take the DBRef object and dereference it.
Binary data is serialized in little-endian.
Being a binary format, MongoDB doesn't have to parse documents like with JSON, they're a valid in-memory presentation already when coming across the wire.

References

Documents can embed a tree of associated data, e.g. tags, comments and the like instead of storing them in different MongoDB documents. This is not specific to MongoDB, but document databases in general (see [5]), but when using find you can dereference nested objects with the dot, e.g. blog.posts.comments.body, and index them with the same notation.
It's mostly left to the language drivers to implement automatic dereferencing of associated documents.
It's possible to reference documents in other databases.

Indexes

Every document gets a default index on the _id attribute, which also enforces uniqueness. It's recommended to index any attribute that's being queried or sorted on.
Indexes can be set on any attribute or embedded attributes and documents. Indexes can also be created on multiple attributes, additionally specifying a sort order.
If an array attribute is indexed, MongoDB will indexed all the values in it (Multikeys).
Unique keys are possible, missing attributes are set to null to ensure a document with the same missing attribute can only be stored once.
If it can, MongoDB will only update indexes on keys that changed when updating a document, only if the document hasn't changed in size so much that it must be moved.
MongoDB up to 1.2 creates and updates synchronously, 1.3 has support to update indexes in the background

Updates

Updates to documents are in-place, allowing for partial updates and atomic operations on attributes (set for all attributes, incr, decr on numbers, push, pop, pull et. al on arrays), also known as modifier operations. If an object grows out of the space originally allocated for it, it'll be moved, which is obviously a lot slower than updating in-place, since indexes need to be updated as well. MongoDB tries to adapt by allocating based on update history (see [6]). Writes are lazy.
Not using any modifier operation will result in the full document being updated.
Updated can be done with criteria, so a whole bunch of matching documents. Think "update ... where" in SQL. This allows for updating objects based on a particular snapshot, i.e. update based on id and some value in the criteria will only update when the document still has that value. This kind of update is atomic. Reliably updating multiple documents atomically (think transaction) is not possible. There's also findAndModify in 1.3 (see [7]) which allows atomically updating and returning a document.
Upserts insert when a record with the given criteria doesn't exist, otherwise updates the found record. They're executed on the collection. A normal save() will do that automatically for any given document. Think find_or_create_by in ActiveRecord.

Querying

Results are returned as cursors, walking a collection as it advances. Which explains why you potentially get records that needed to be moved, it pops up in a space that's potentially after its current position, if there's space even in a spot before the current cursor's position. Cursors are fetched in batches of 100 documents or 4 MB of data, whichever's reached first.
That's also why it's better to store similar data in a separate collection. Traversing similar data is cheaper than traversing over totally unrelated data, the bigger the size of documents compared to the documents that match your find, the more data will have to be fetched from the database and skipped if it doesn't match your criteria.
Data is returned in natural order which doesn't necessarily relate to insertion order, as data can be moved if it doesn't fit into its old spot anymore when updated. For capped collections, natural order is always insertion order.

Durability

By default, data in MongoDB is flushed to disk every 60 seconds. Writes to MongoDB (i.e. document creates, updates and deletes) are not stored on disk until the next sync. Tradeoff high write performance vs. durability. Need more durability, reduce sync delay. Closest comparison to the durability behaviour is MySQL's MyISAM.
Data is not written transactional, so if the server is killed during a write operation, the data is likely to be inconsistent or even corrupted and needs repair. Think classic file systems like ext2 or MyISAM.
In MongoDB 1.3 a database flush to disk can be enforced by sending the fsync command to the server.

Replication

Replication is the recommended way of ensuring data durability and failover in MongoDB. A new (i.e. bare and dataless) instance can be hooked onto another at any time, doing an initial cloning of all data, fetching only updates after that.
Replica pairs offers an auto-failover mechanism. Initially both settle on which is master and which is slave, the slave taking over should the master go down. Can be used e.g. in the Ruby driver using :left and :right options. There's an algorithm to handle changes when master and slave get out of sync, but it's not fully obvious to me (see [8]). Replica Pairs will be replaced by Replica Sets, allowing for more than one slave. The slave with the most recent data will be promoted master in case of the master going down. The slaves agree which one of them is the new master, so a client could ask any one server in the set which one of them is the master.
Replication is asynchronous, so updates won't propagate immediately to the slaves. There's ideas to require the right to be propagated to at least N slaves before returning the write to the client successfully (similar to the feature in MySQL 5.4). (see [9])
A master collects its writes in an opslog on which the slaves simply poll for changes. The opslog is a capped collection and therefore not a fully usable transaction log (not written to disk?) as old data is purged automatically, hence not reliable for restoring the database after a crash.
After initial clone, slaves poll once on the full opslog, subsequent polls remember the position where the previous poll ended.
Replication is not transactional, so the durability of the data on the slave is prone to the same durability conditions as the master, just in a different and still durability-increasing manner, since having a slave allows to decrease sync times on it, and therefore shortening the timespan of data not being written to disk across the setup.

Caching

With the default storage engine, caching is basically handled by the operating system's virtual memory manager, since it uses memory-mapped files. File cache == Database cache
Caching behaviour relies on the operating system, and can vary, not necessarily the same on every operating system.

Backup

If you can live with a temporary write lock on your database, MongoDB 1.3 offers fsync with lock to take a reliable snapshot of the database's files.
Otherwise, take the old school way of dumping the data using mongodump, or snapshotting/dumping from a slave database.

Storage

Data is stored in subsequently numbered data files, each new one being larger than the former, 2GB being the maximum size a data file can have.
Allocation of new datafiles doesn't seem to be exactly related to the amount of data currently being stored. E.g. storage size returned by MongoDB for a collection was 2874825392 bytes, but it had already created almost six gigabytes worth of database files. Maybe that's the result of padding space for records. I haven't found a clear documentation on this behaviour.
When MongoDB moves data into a different spot or deletes documents, it keeps track of the free space to reuse in the future. The command repairDatabase() can be use to compact it, but that's a slow and blocking operation.

Concurrency

MongoDB refrains from using any kind of locking on data, it has no notion of a transaction or isolation levels. Concurrent writes will simply overwrite each other's data, as they go straight to memory. Exceptions are modifier operations that are guaranteed to be atomic. As there is no way to update multiple records in some sort of transaction, optimistic locking is not possible, at least in a fully reliable way. Since writes are in-place and in-memory first, they're wicked fast.
Reads from the database are usually done in cursors, fetching a batch of documents lazily while iterating through it. If records in the cursors are updated while the cursor is being read from, the updated data may or may not show up. There's no kind of isolation level (as there are no locks or snapshotting). Deleted records will be skipped. If a record is updated from another process so that the size increases and the object has to be moved to another spot there's a chance it's returned twice.
There's snapshot queries, but even they may or may not return inserted and deleted records. They do ensure that even updated records will be returned only once, but are slower than normal queries.

Memory

New data is allocated in memory first, increments seem to be fully related to the amount of data saved.
MongoDB seems to be happy to hold on to whatever memory it can get, but at least during fsync it frees as much as possible. Sometimes it just went back to consuming about 512 MB real memory, other times it went down to just a couple of megs, I couldn't for the life of me make out a pattern.
When a new database file needs to be created, it looks like MongoDB is forcing all data to be flushed to disk, freeing a dramatic amount of memory. On normal fsyncs, there's no real pattern as to how MongoDB frees memory.
It's not obvious how a user can configure how much memory MongoDB can or should use, I guess it's not possible as of now. Memory-mapped files probably just use whatever's available, and be cleaned up automatically by the operating system's virtual memory system.
The need to add an additional caching layer is reduced, as object and database representation is the same, and file system and memory cache can work together to speed up access, there's no data conversion involved, at least not on MongoDB's side, data will just be sent serialized and unparsed across the wire. Obviously it depends on the use case if this is really an advantage or a secondary caching layer is still needed.

GridFS

Overcomes the 4MB limit on documents by chunking larger binary objects into smaller bits.
Can store metadata alongside file data. Metadata can be specified by the user and be arbitrary, e.g. contain access control information, tags, etc.
Chunks can be randomly access, so it's possible to fetch data easily whose position in the file is well-known. If random access is required, makes sense to keep chunks small. Default size is 256K.

Protocol Access

MongoDB's protocol is binary and in its own right proprietary, hence they offer a lot of language drivers to take that pain away from developers, but also offer a full specification on both BSON and the protocol.

Sharding

MongoDB has alpha support for sharding. Its functionality shouldn't be confused with Riak's way of partitioning, it's a whole different story. The current functionality is far from what is planned for production, so take everything listed here with a grain of salt, it merely presents the current state. The final sharding feature is supposed to be free of the restrictions listed here.
A shard ideally (but not necessarily) consists of two servers which form a replica pair, or a replica set in the future.
All shards are known to a number of config server instances that also know how and where data is partitioned to.
Data can be sharded by a specific key. That key can't be changed afterwards, neither can the key's value.
Keys chosen should be granular enough so that there's the potential of having too many records with the same key. Data is split into chunks of 50 MB so with big documents, it's probably better to store them in GridFS, as a chunk can contain a minimum of ~12 documents when all take up the available space of 4 MB.
Sharding is handled by a number of mongos instances which are connected to the shards which in turn are all known to a number of mongod config server instances. These can run on the same machines as the data-handling mongod instances, with the risk that when the servers go down they also disappear. Having backup services seems to be appropriate in this scenario.
Sharding is still in alpha, e.g. currently replicated shards aren't supported in alpha 2, so a reliable sharding setup is currently not possible. If a shard goes down, the data on it is simply unavailable until it's brought back up. Until that happens, all reads will raise an error, even when looking up data that's known to be on the still available shards.
There's no auto-balancing to move chunks to new shards, but that can be done manually.

There you go. If you have something to add or to correct, feel free to leave a comment. I'm happy to stand corrected should I have drawn wrong conclusions anywhere.

As a user of CouchDB I gotta say, I was quite sceptical about some of MongoDB's approaches of handling data. Especially durability is something that I was worried about. But while I read through the documentation and played with MongoDB I realized that it's the same story as always: It depends. It's a problem when it's a problem. CouchDB and MongoDB don't necessarily cover the same set of use cases. There's enough use cases where the durability approach of MongoDB is acceptable compared to what you gain, e.g. in development speed, or speed when accessing data, because holy crap, that stuff is fast. There's a good reason for that, as I hope you'll agree after going through these notes. I'm glad I took the time to get to know it better, because the use cases kept popping up in my head where I would prefer it over CouchDB, which isn't always a sweet treat either.

If you haven't already, do give MongoDB a spin, go through their documentation, throw data at it. It's a fun database, and the entrance barrier couldn't be lower. It's a good combination of relational database technologies, with schemaless and JavaScript sprinkled on top.

CoworkingCologne startet am 1.März 2010

2010-02-22T19:48:15+00:00

Wie schon bei Peter angekündigt. Ab dem 01. März startet das erste in der Praxis umgesetzte CoworkingCologne Projekt. Details folgen in kürze.

Just to prove I'm not dead,

2010-02-19T14:32:30+00:00

I moved the ActsAsFerret wiki and issue tracker to Github and Lighthouse.

It’s not that I don’t like Redmine any more, but unfortunately it caused me trouble bringing the whole server to a halt every few weeks. I think I could have fixed that somehow but I feel that hosting Redmine isn’t one of my core competences…

Update: Here’s the script I used to import the existing Redmine Tickets into Lighthouse.

5 Tipps für deine nächste Threadless-Bestellung (Zoll, Packstation, Versandkosten)

2010-02-17T20:20:22+00:00

Ich liebe Threadless. Das Interface, die Feature (zeige mir nur, was mir passt und verfügbar ist), den Support, die Kultur und natürlich allen voran die T-Shirts selbst. Tolle Qualität mit den besten Motiven weit und breit.

Aber wer aus Deutschland bei Theadless bestellt, braucht leider etwas Hintergrundwissen:

Die wichtigsten allgemeinen Regeln:

Nie an Packstation schicken lassen! Warum? — Kurz gesagt, kommt's dann nicht an!
Sammelbestellungen sind nicht wirklich ideal wegen des Zolls!
Wenn das Päckchen zu dick aussieht (oder sonst was), wird es einkassiert und ihr müsst es beim Zoll abholen (Umsatzsteuer + meist nervig weiter Weg).

Was ihr über den Zoll wissen müsst:

Disclaimer: Diese Angaben basieren auf meiner eigenen Erfahrung beim Zoll in Köln und dürfen gerne in den Kommentaren korrigiert werden! Danke!

Diese Grenzwerte sind wichtig. Und damit es nicht zu einfach ist, gelten für Zoll und Umsatzsteuer unterschiedliche Grenzwerte.

Reiner Warenwert > 22 € = 19 % Umsatzsteuer — basierend auf vollen Wert inkl. Versandkosten (Oder war es genau andersrum exkl. Versandkosten? Wer kann helfen?)
Reiner Warenwert < 150 € = kein Zoll
Reiner Warenwert > 150 € = 12 % Zoll da Textilien

Daraus ergeben sich für den Threadless-Besteller drei Szenarien:

Zoll merk's nicht = 0 €
Zoll merk's aber < 150 € = Umsatzsteuer nachzahlen (Das sind übrigens 205 $.)
Zoll merk's und > 150 € = Umsatzsteuer + Textil-Zoll (nach)zahlen

Diese Regeln gelten, wenn man Waren von einer Firma bezieht. Die schlauen Füchse unter euch werden jetzt fragen, wie es denn mit Geschenken aussieht, die sind irgendwie zollfrei. Doch wie auch der verlinkte Fuchs wirst auch du es damit nicht ins Ziel schaffen. Laut der netten Dame im Zoll Köln lassen Sie Päckchen nur als Geschenke durchgehen, wenn sie offensichtlich ein Privatgeschenk sind mit Geburtstagskarte drin. Im Übrigen gelten dann andere Grenzen von 44 € für die Umsatzsteuer und ein reduzierter Satz von 17,5 % oder so. Nagelt mich nicht drauf fest.

Welche ist also das ideale Verhältnis von Anzahl Shirts zu Gesamtkosten zu Versandkosten zu Zoll/Umsatzsteuer?

Keine Ahnung :-). ~~Aber hier dürfen gern die Formel-Fans unter euch einschreiten und ein paar Rechenbeispiele aufführen.~~

Unter den 22 € zu bleiben (im Szenario exkl. Versandkosten, s.o.) lohnt sich meiner Meinung nach nicht, in Relation zu den Versandkosten, weil man für 22 € ja selbst von den günstigen Shirts nur 2-3 bekommt.

Aber Andreas hat ein sehr cooles GoogleSpreadsheed erstellt. Danach ist das Beste Kostenverhälnis bei 7 Shirts (15 $ Versand).

Tipps für euren Zollbesuch:

Wenn eure Bestellung viel länger auf sich warten lässt, als sie sollte, kann das zwei Gründe haben: Sie ist zum Zoll umgeleitet worden oder verloren gegangen (beides erlebt). Bei diesen beiden Möglichkeiten ist es dann eine Erleichterung, endlich die kleine Hinweispostkarte zu bekommen, die einen bittet, sein Päckchen vom Zoll abzuholen.

Hier, was ich gelernt habe:

Die Öffnungszeiten sind… eigen (7:30-15 Uhr)
Wenn man auf die Klingel drückt, kommt wirklich irgendwann jemand zu einem, auch wenn das Gebäude sonst verlassen wirkt
Es dauert "nur" 2 Stunden von Köln Ehrenfeld nach Wahn und zurück…
Laut Zoll-Infozettel muss man nach 7 Tagen eine Lagerungsgebühr zahlen. Mir wurde sie zum Glück nicht berechnet (ein Woche zu spät).
Packstation: Ich werde nicht noch einmal etwas, das potentiell beim Zoll landen könnten an eine Packstation schicken. Auf meinem Paket war die Packstationsadresse durchgestrichen und meine richtige Adresse notiert. Ich vermute, dieser Schritt hat zusätzlich Zeit gekostet…

Zum Schluss jetzt noch ein wichtiger Tipp für Packstations-Nutzer:

Threadless + Packstation = Böse!

Offensichtlich muss man die Formel sogar weiter spannen: Grundsätzlich scheint es keine gute Idee zu sein, aus dem Ausland an eine Packstation senden zu lassen.

Meine eigene Erfahrung: Eine große Threadless-Bestellung an die Packstation ist erst ewig später angekommen. Diese musste ich vom Zoll abholen, habe sie aber bekommen. Danach habe ich noch einmal etwas bestellt mit den gleichen Adressdaten. Das ist nie angekommen. Threadless konnte mir nicht helfen, weil sie keine Trackinginformationen haben für Auslandspakete. Sie waren aber sehr zuvorkommend und haben mir später die Bestellung ersetzt (ich konnte wählen: Shirts neu schicken, Geld zurück oder Gutschein — ein toller Service!). In dieser Zeit habe ich auf der Packstationshomepage rumgesucht und @packstation per Twitter gefragt aber nirgendwo Angaben zum Empfang von Paketen aus dem Ausland gefunden. Es sollte also klappen? Nein, die Frau an der Hotline sagte direkt, dass Pakete aus dem Ausland "selbstverständlich" nicht ankommen können, da sie ja nicht über DHL verschickt werden. Ätzend! Vor allem die Informationspolitik der Packstation hierzu. Ich hoffe, Threadless nimmt meinen Vorschlag auf und warnt demnächst bei der Eingabe eine Packstationsadresse im Interface…

Wer mehr zu Packstation weiß: Bitte nutzt die Kommentare!

Und jetzt viel Spaß beim Shirtkauf!

Mozilla Labs Design Challenge: 5 of 5 Designs fail

2010-02-17T18:57:32+00:00

Mozilla Labs just asked everyone to vote for their favorite concept of the new home tab.

This is the last step of the Mozilla Labs Design Challenge in which users where asked to illustrate their ideas of the Hometab (Read more about it).

You may have a look at the 5 results after you added your eMail-Address to register yourself for voting.

This is why I am writing about it: I am sorry to say it but in my opinion all five designs fail to show a good, easy, usable concept for the hometab. I am sure everyone who contributed spend some time on their ideas (and the video they created for it) and that is a great contribution to the community (thanks!).

But from a UX and usability point of view those designs don't show a solution. Instead they are great examples of how to clutter an interface with tons of features and loose all simplicity in this process.

At least I dont see my hometab as a dashboard of widgets that show and pull tons of information on a small space, as one contribution suggested. If you want this, make your Netvibes Page or iGoogle Page your homepage. Or add yourself a weather-widget to you operation system. Dont look at the hometab as if there where no OS, Widgets and Apps around your browser. At least not until Chrome OS rules us all.

And I dont want it to be a copy of my social networks as well. I have my Tweet-App, Newsreader and all the other good apps for that. None of those apps could be as good as the apps that are already out there for years with their own community an user base. Try using Flock "the social browser" that no one uses.

And I dont want a hometab that changes every time I visit a website. Even Microsoft got it know that formulas that try to adapt to users behavior are much less effective than a UI that reinforces habits. When I am used to seeing my facebook-link at the bottom of the list I will find it there without thinking. Dont move everything around just because I use it.

That being said I voted with one 1 and four 0 on the challenge and hope, the UI-experts at mozilla will take over from here.

Its will be interesting to see if this kind of open source design will produce any mentionable results in the future. And how this design challenge will influence the community-process.

A Collection Of Redis Use Cases

2010-02-16T00:00:00+00:00

I'm gonna eat my own dog food here, and start you off with a collection of links and ideas of people using Redis. Redis' particular way of treating data requires some rethinking how to store your data to benefit from speed, atomicity and its data types. I've already written about Redis in abundance, this post's purpose is to compliment them with real-world scenarios. Maybe you can gather some ideas on how to deal with things.

There's a couple of well-known use cases already, the most popular of them being Resque, a worker queue. RestMQ, an HTTP-based worker queue using Redis, was just recently released too. Both don't make use yet of the rather new blocking pop commands like Redactor does, so there's still room for improvement, and to make them even more reliable.

Ohm is a library to store objects in Redis. While I'm not sure I'd put this layer of abstraction on top of it, it's well worth looking at the code to get inspiration. Same is true for redis-types.

A Twitter Clone using Redis

This should be the first one to check out, very detailed example on how you could implement a Twitter clone using Redis and PHP. Twitter clones using Redis are all the rage these days, there's also a clone of the original clone written in Ruby.
Sort in Redis

An instant classic. Some examples on how to use sorting in Redis.
Redis in Practice: Who's Online?

Great and distilled example of how to utilize set intersections and unions to find out which people are online at a given minute.
Building a Twitter Filter with Sinatra, Redis, and Tweetstream

Exactly what the title says. Examples of using lists to store tweets marshalled using JSON.
Where's Waldo: Track user locations with Node.js and Redis

Redis and Node.js seem to be a great match, be sure to check out the source too. Similar to the one above, different implementation. Be sure to read the prequel "Node.js For My Tiny Ruby Brain" too, some more code in there.
Sikwamic: Simple Key-Value With Comet

Comet and Redis, sitting in a tree.

Redis' simplicity, atomicity and speed make it an excellent tool when tracking things directly from the web, e.g. through WebSockets or Comet. If you can use it asynchronously, all the better.

Affiliate Click Tracking with Rack and Redis.

Simple approach to tracking clicks, I probably wouldn't use a list for all clicks, but instead have one for each path, but there's always several ways to get to your goal with Redis. Not exactly the same, but Almaz can track URLs visited by users in Rails applications.

Update: Turns out that the affiliate click tracking code above, the list is only used to push clicks into a queue, where they're popped off and handled by a worker, as pointed out by Kris in the comments.
Building a NLTK FreqDist on Redis

Calculation of frequency distribution, with data stored in Redis.
Gemcutter: Download Statistics

The RubyGems resource par excellence is going to use Redis's sorted sets to track daily download statistics. While just a proposals, the ideas are well applicable to all sorts of statistics being tracked in today's web applications.
Usage stats and Redis

More on tracking views statistics with Redis.
Vanity - Experiment Driven Development

Split testing tool based on Redis to integrate in your Rails application. Another kind of tracking statistics. If you didn't realize it up to now, Redis is an excellent tool for this kind of application. Data that you wouldn't want to load off to your main database, because let's face it, it's got enough crap to do already.
Flow Analysis & Time-based Bloom Filters

Streaming data analysis for the masses.
Crowdsourced document analysis and MP expenses

While being more prose than code, it still shows areas where Redis is a much better choice than e.g. MySQL.

Using Redis to store any suitable kind of statistics is pretty much an immediate use case for a lot of web applications. I could think of several projects I've work on that could gain something from using certain parts of their application to Redis. It's the kind of data you just don't want to clutter your database with. Clicks, view, history and all that stuff puts an unnecessary amount of data and load on it. The more data it accumulates, the harder it will be to get rid off, especially in MySQL.

Simple Note Keeping with Redis

A code snippet that's actually a fully working Sinatra app to jot down notes.
URL Shortener Service using Redis

Personal URL shortening is the new black, that's why I've written my own service.

It's not hard to tell that we're still far from having heaps of inspiration and real-life use cases to choose from, but these should give you an idea. If you want it can get a lot simpler too. When you're using Redis already, it makes sense to use it for storing Rails sessions.

Redis is a great way to share data between different processes, be it Ruby or something else. The atomic access to lists, strings and sets, together with speedy access ensures that you don't even need to worry about concurrency issues when reading and writing data. On Scalarium, we're using it mostly for sharing data between processes.

E.g., all communication between our system and clients on the instances we boot for our users is encrypted and signed. To ensure that all processes have access to the keys, they're stored conveniently in Redis. Even though that means the data is duplicated from our main database (which is CouchDB if you must know), access to Redis is a lot faster. We keep statistics about the instances in Redis too, because CouchDB is just not made for writing heaps and heaps of data quickly. Redis also tracks a request token that is used to authenticate internal requests in our asynchronous messaging system, to make sure that they can't be compromised from some external source. Each request gets assigned a unique token. The token is stored in Redis before the message is published and checked before the message is consumed. That way we turned Redis into a trusted source for shared data between web and worker processes.

The library memodis makes sharing data incredibly easy, it offers Redis-based memoization. When you assign a memodis'd attribute in your code, it'll be stored in Redis and therefore can be easily read from other processes.

Redis is incredibly versatile, and if you have a real-life Redis story or usage scenario to share, please do.

Using MongoMapper with Rails 3

2010-02-15T15:31:17+00:00

There is a really nice introduction at the MongoDB site on How to get started with MongoDB and Rails 3. I used this as a starting point but had to do a few more things before I was fully satisfied and got things running with MongoMapper. At the time of writing this the official repository is not Rails 3 compatible (because it does not implement the ActiveModel interface, yet), so I had to go with the rails3 branch of merbjedi’s repository. Use the following line to add the dependency to your Gemfile:

gem "mongo_mapper", :git =&gt; 'http://github.com/merbjedi/mongomapper.git', :branch =&gt; 'rails3'

And while you are at it, add this line to include some handy generators for Rails 3, including one for MongoMapper models:

gem "rails3-generators"

To use the generator set the orm config in your config/application.rb to :mongomapper

config.generators do |g|
      g.orm :mongomapper
      …
end

… and you are good to go!

btw: If you are like me and you prefer to have some kind of GUI for quick database lookups, check out MongoHub. It is in a very early stage of development, but at least you get to see your databases and documents.

NoSQL And You - Thoughts On Finding Right Partner In Chrime

2010-02-15T00:00:00+00:00

The NoSQL landscape is a fickle thing, new tools popping up every week, broadening a spectrum that's already close to being ungraspable, especially when you're totally new to the whole thing. There's a couple of common misconceptions and wrong-doings that people who've been playing with the tools already tend to tell newbies in the landscape.

I'm guilty as charged too, I tend to tell people about the tools I already know. Being a good thing per se, because the recommendation is based more or less on experience, it leaves out one thing that I find to be the most important philosophy about post-relational (much nicer term than NoSQL) databases: It's all about your data, about its needs and how your application needs to access them. The times of generic, one-size-fits-all tools like MySQL, PostgreSQL and the like are over, it's well worth knowing how they're different, and what tool would be the best partner in chrime to get stuff done.

No Size Fits All

While you could throw MySQL at a lot of problems, it was far from being the optimal choice in a lot of cases, but you could somehow bend it to your will. The new generation of tools tends to avoid having to be bent, instead they give you a freedom of choice. The freedom to analyse what your data is like, and what the right tool for your specific use case is.

Does that mean more work on your end? It sure does, but for the love of your data, it will be worth it. If you find the right partner, and it makes your life easier, it's a win on both ends. You'll be a happy developer (well, most of the time), and your data will be able to roam free, running naked across a meadow, hand in hand with the tool you chose.

Now I'm well aware that this sounds all bloomy, but that's what it boils down to. The choice is now up to you, that's why it's important to know what's out there, to play with the tools available, to know how and why they're different from each other.

To get you started, have a look at Vineet Gupta's excellent overview of the NoSQL landscape.

Don't Believe Everything You Hear

If someone tells you that you should try a specific tool, ask him why. If the answer is speed, or because it's written in Erlang and scales insanely well, it's time to call bullshit on him. MySQL can be fast too, that's not an issue. It's nice to be able to have a database that you can scale up to hundreds of nodes easily, but while the technology behind it is very interesting, and sometimes mind-blowing, it doesn't help if it's a pain to work with, or if there's no library support yet. Sure you could write your own, but if you're totally new to the field, you usually just want to play and learn. Hard thing to do if all you get is just an API and a very limited language support.

If someone tells you that e.g. MongoDB is fast, then there's reason for that, and it's good to be well aware of it and what consequences it has for operating your application. If someone tells you that CouchDB is awesome for building web applications, because it's built of the web, they're leaving out that a common use case like pagination is still an awful thing to implement with it. If someone tells you that Cassandra scales easily because it was built at Facebook, they're leaving out that its peculiar way of storing and accessing data is very specific to how sites like Facebook need to access their data. I could go on and on about it, but there's always two side of a story.

Before judging a tool based on just the one side, look at the other side too. It might not be as big of a problem as you thought it would, either way, you know why things are the way they are. Look for tools with sites describing particular use cases, or areas where they're just not a good fit. If the tool builders aren't aware of use cases, strengths and weaknesses, how will you be?

In the end, even though they can be problems for others, they don't necessarily are problems for you. Your particular use case might be just fine with the downsides, but on the other hand gaining high profit from the upsides. If it's not, at least you're more than free to go look somewhere else. At least now you have the (free) options to do so.

There's misconceptions out there being close to urban myths, and we're only two years or so into working with the new generation of tools. The only way you can avoid falling into a trap is to play with what's out there, to know their weaknesses and strengths. The only thing we can do to avoid having people fall into the trap is to better educate them, to give them real-world examples, use cases other than tagging and blog posts. Just saying that it scales better than xyz is not an argument, it's educating people on the wrong end.

It's Not About Speed And Scaling

If speed and scaling were our only problems, we'd be left in a big world of pain. As beautiful as these words are, I'm gonna go out on a limb and say that it's not a problem until it's a problem. Unless you're already Facebook or LinkedIn, you don't need to have that as a main factor when choosing the right tool. Sure, it's better if there's an easy way to scale up in the future, but what's the point if you needs days to get a good set up before having written a single line of code?

Most NoSQL tools were built with some sort of scaling in mind, although people tend to easily confuse scaling, distribution, sharding and partitioning, so you're safe in most cases when it comes to the point where your application needs to handle more traffic.

I'm gonna go ahead and venture the guess that if you're deciding solely based on speed and scalability, you're doing it wrong. And I rarely use that phrase. You should be deciding based on the core feature set, why it does the things it does, and what consequences it'd have on your life as a developer.

Don't Compare Apples And Oranges

No tool is like the other. Just comparing e.g. MongoDB, Redis and MySQL is the wrong way to approach your problem, especially if you just look at speed and comparing their feature set. Feature sets and speed are usually different for a reason. Instead you should be comparing every tool with your data. How much do you need to bend the data to store and access it easily. Is it even possible to store it efficiently and or your particular use case? Are potential trade-offs (e.g. data duplication to gain speedier access) worth risking? Is it the right fit in the way it handles updates, associations, writes, reads and queries in terms of your data and application? Then go right ahead and use it.

But don't just compare tools with each other whose only feature they have in common is the fact that they can store data, or things that are mostly depending on an application's specific needs. To give you an idea, this guy compares Redis and MongoDB by implementing a particular use case with both of them. That's the way you should be comparing tools.

The Heat Is On

We're going to see more tools popping up left and right, making it harder to keep up, and to make an informed decision. What I consider the best thing about most of them is that they're free. You can grab the source code, improve it or just look at how it handles your data. That's what makes them so awesome, their incentive is not to constrain your data, they're as open as possible about it, some tools even going as far as building solely on open standards to implement their whole stack (that'd be CouchDB if you're curious).

The whole point of this post is that it's up to you to find the perfect tool to hand your data to. I don't know about you, but me being able to find the right fit instead of squeezing my data into a database that tries to solve all problems at once, that's the most exciting prospect of post-relational databases for me. Our common goal should be to help people make that decision without getting too passionate about any particular tool. They all exist to fulfill some purpose, and we should be telling people about them.

There's a couple of sites to keep an eye on, e.g. MyNoSQL by Alex Popescu, he's keen on keeping up-to-date with what's going on in the NoSQL community. Another site with a growing collection of links to articles is nosql-databases.org. EngineYard published a series of blog posts on key-value stores in Ruby, in particular Cassandra, Redis, MongoDB, CouchDB, LDAP that's well worth checking out to get an idea of what's out there.

ZENDOME für das iPhone

2010-02-12T10:49:44+00:00

Es ist soweit und ZENDOME – ein Kundenprojekt an dem ich in letzter Zeit gearbeitet habe – ist nun kostenlos im App Store erhältlich. Die Applikation wurde im Auftrag der ZENDOME GmbH entwickelt und ermöglicht die Planung von individuellen Event-Landschaften, die beispielsweise auf Messen oder für Ausstellungen verwendet werden. Der Benutzer hat die Möglichkeit, die verschiedenen geodätischen Dome zur ZENDOME.domescapes zu kombinieren und damit seine Veranstaltung vorzudenken. Ist eine Planung erstellt lässt sich schnell und unkompliziert ein Angebot anfordern.

Besonderer Wert wurde auf eine intuitive Benutzung der Anwendung gelegt. Um dies zu erreichen wurde auf die aus dem allgemeinen iPhone-Kontext bekannte Gesten und Aktionen (bspw. Pinch/Zoom, Drag and Drop, Scrolling, etc. ) zurückgegriffen und einfache Interaktionsmuster benutzt. Es gibt bereits Erweiterungspläne für kommende Versionen: So ist insbesondere eine iPad-Variante angedacht, da die größere Bildschirmfläche natürlich mehr Platz für eine komfortable und detailreiche Planung bietet. Noch etwas zum technischen Hintergrund: Das Projekt wurde mit Xcode und CocoaTouch-Standardtechnologien wie UIKit, CoreGraphics und CoreAnimation innerhalb von vier Wochen realisiert und hatte insgesamt einen Arbeitsaufwand von ca. 40 Stunden. Weitere Projektdetails und Informationen plane ich in naher Zukunft in einer kurzen Fallstudie zu veröffentlichen.

Rails 3 beta is released!!

2010-02-05T04:11:31+00:00

Yeah! Rails 3 beta just got released.
So grab it while it’s hot: ;)

gem install rack bundler tzinfo i18n rack rack-test rack-mount erubis mail thor

gem install rails --prerelease

Railslove around the world

2010-02-03T11:01:54+00:00

We’re posting more and more stuff on our tumble blog “Railslove around the world”.

On the tumble we’re publishing all the good stuff we discover on the web, mostly ruby and web develoment related. So make sure to keep an eye on it. ;)

On the way to Rails 3 - a link list

2010-02-02T13:58:40+00:00

Rails 3 is coming! The first beta release of the new version is right around the corner and it should be released today or tomorrow… the release notes are already in the master branch.
For me this feels like the release of Rails 1.0 and I’m so excited about all the new changes and the great new features.
Rails 3 is a lot about changing your mindset to make use of the all new features and concepts.

I’ve collected some reading material for you about Rails 3 and the upcoming changes. Since the final Rails 3 is not yet released excpect some of the stuff that’s mentioned to be changed or already have changed in the beta. But this is a good start to catch up with all the Rails awesomeness:

Must-reads

Actually all of these links are “must-reads” but for those of you with little time. Read at least these:

official Rails 3 relase notes
Live Coding Rails 3 upgrade a free peepcode
Exploring Rails 3 - A Free Two Hour Online Conference
The Path to Rails 3 - Intorduction - A very awesome introduction to the big picture of Rails 3. read it!
Rails and Merb Merge: Rails Core (Part 4 of 6)
Rails 3 Upgrade plugin - checks your code and gives you hints to make it Rails 3 compatible (first announcment)
Greenfielding new apps with the Rails 3 beta

Router

ActiveRecord / ActiveModel

Active Record Query Interface 3.0
Rails 3.0’s ActiveModel: How To Give Ruby Classes Some ActiveRecord Magic
ActiveModel: Make Any Ruby Object Feel Like ActiveRecord
Arel source code and readme
Why Arel
validates :rails_3, :awesome => true great post about the new validations in Rails 3

ActionMailer

New ActionMailer API in Rails 3

Views / JavaScript

Controller

Rack

The CodeRack Contest
21 Rack Middlewares To Turbocharge Your Ruby Webapps
Rails on Rack(not Rails 3 specific)

Gem, Plugins, Dependencies

Generators

I18n

Rails 3 I18n changes

General Rails

I’ll update this list with upcoming Rails 3 resources.

update:

1. Added link to I18n changes (thanks @ Carlos)

The musichackday that wasn't meant to be

2010-01-30T23:00:00+00:00

As my coworker Florian would put it: Damn the sh*t f**k. For us (us being the hamburgian musichackday tourists Michael, Wolfgang and Jan with a taste for low budget travel plans), the musichackday was over before it even started. We patiently stood in line at the Ryanair dominated airport in Lübeck, but, really, we should have known better. Not a single Plane left or arrived at Lübeck after 15:00 on Friday. It was snowing like hell (a very cold hell, that is) and the snow ploughs tried their best, but, heck, it was almost impossible to drive safely with a car out there, so it probably wasn’t a too bad decision to not try to land or start there. Safety first, safety first.

But, being a screwed Ryanair customer, it was clear for us that we had no chance to get to STHLM in time, except for making the travel at maybe 3 times the expense originally planned.

This obviously being the most awesome hackday evar, I’m still quite disappointed. But I repeat myself. Wolfgang suggested early on that we should hack on regardless and so all of us met on Saturday, started with a good Breakfast with leftovers from the porkcamp and dived into the projects we had originally planned for Stockholm.

Michael deployed his small lastFM API hack called my city vs. your city at the end of Saturday and spent Sunday with optimising the calculation.

Wolfgang and I started in the Saturday afternoon with a small echonest analyzer mash with parts of my javascript soft synth work. Yes, the output is silly, yes it would need a lot of work to let it output something useful, but anyway.

I then, after we left the mindmatters office, started to integrate the javascript synth into my webloop stuff.

On sunday Wolfgang made the amie pretty and I deployed the stuff to one of my webservers. After that we recorded two screencasts to be able to be part of the presentations. And then we were able to see the videos that we recorded, streamed into the doberman office in the live stream we watched. Ah, the interwebs.

Now I am sitting here and wondering if Brian Whitman will really ship echonest tracksuits to our office as he suggested in his tweet:

I can’t believe those AMIE dudes found @echonest’s secret plans to drunkuncle.dj. #musichackday #howdoyoushipveloursuitstogermany

Har har. Here are the screencasts:

Next time I very much would prefer to attend in person, thank you very much, but I think we made the best out of it.

iPad it is

2010-01-26T23:00:00+00:00

this article has been updated on the 28th of January to correct some errors and clarify a few points

First off: I think the name is a poor choice. The web, especially the twitters seems to agree. I would have loved all the other names that flew around during the last days, like iBook, Canvas, Palette, whatever.

With that out of the way, I am pretty baffled by both the product itself but also the reactions to it.

What can I say. No multitasking feels weird with a device that can run special versions of iWork on it. (Let’s hope they do a lot of state saving and the apps start fast). Apart from that, the whole thing feels like an extremely solid, well designed piece of hardware/software.

Now here’s the real problem: While OS X on my Mac has it’s proprietary edges, the system is mostly open. iPhone and iPad are, by default, pretty much closed devices. And this leaves a lot of questions unanswered. For example, it restricts your music buying to iTunes, which is good for noone but Apple. This is, for example, not the case for iTunes on my Mac, simply because I can drop any piece of music, on CD or as mp3, into iTunes. But there’s a glimpse of hope…there are some changes (yah, NDA fun at work, so I cannot tell anything specific) in the 3.2 SDK that might allow Applications to add files to to other Applications. Maybe. We’ll see.

And now there’s iBooks: Will it restrict Book buying to the iBooks store? We don’t know yet, but at least it is based on a standard (ePub), so it might be possible to add non-DRM books (like my beloved pragprog series) directly to the iBooks app. But what about the DRM format? Will it support the defacto standard Adobe DRM and by that opening your iPad to the hundreds of titles that publishers sell for, e.g. the Sony readers? I somehow doubt that, but we will hopefully get some details on that very soon.

For me, somehow, the closed nature of the iPad is much worse than the closed nature of the iPhone, probably because it’s closer to a “real” computer and I like those to be as open as possible, thankyouverymuch.

Still, I do not agree with many commenters that the iPad is an underwhelming supersized iPhone and nothing else. Or that it cannot compete with netbooks. Or that there are no usecases.

First of all, I think people underestimate the possibilities of a 10" multitouch screen with a very high resolution. A few apps shown in the demos, especially the photos app, show a bit of the magic that will befall you as soon as you can touch this device. Yes, this will be a completely new dimension on top of what’s already possible on the iPhone. No, you probably ain’t seen nothing yet. Yes, you can quote me on that.

Also I think that people underestimate the value of a simple, thin, relatively light and extremely solid device with virtually no moving parts. This WILL be georgeous for reading (I hope that they thought of a way to lock orientation, or otherwise it will be impossible for me to read in the bed), this will be fabulous for watching movies or browsing through your music collection, but in the end, it will open up completely new ways of using a computer, simply because it stops being awkward or unfitting. It is not as clumsy as a larger Notebook (which is usually too heavy to lift it comfortably with one hand), it cannot fall over like my netbook when open, it holds natural (everyone complaining about the huge bezel should also complain about the huge margins usually employed in printed books), and, last but not least, it simply looks great. All in all, it will happily replace a “real” computer in any environment that’s not “computerish” by default. I am relatively sure that I won’t be coding web apps on an iPad anytime soon, but for gaming, reading, watching movies and simple browsing, it will be a much better fit than anything else. And maybe, possibly with the help of the keyboard dock, for some people this will be all the computer they’ll need in their life.

Am I unhappy about the closed nature of all the Shops that Apple has under their control? You bet. Am I unhappy about the missing Multitasking? For sure. Do I think that the iPad will be a success? Yessir, although I am not sure how fast it will take on at first.

One additional word on the multitasking: I don’t actually think that this decision is set in stone. Didn’t it strike you as odd that the SDK version for the iPad is not, as I would have expected, 4.0, but instead it’s just a point release? This should give you a hint that in the summer, when I expect both a new iPhone model AND a new iPhone OS major release (I would not even rule out an iPad 2), there are some major things to come. Not sure that it will feature complete multitasking, but limited background process support (especially for alternative music apps) seems to make sense for me.

Many people also complain about the missing camera and I must say that this strikes me as odd as well – The iPad would otherwise be a kick ass device for video chat. But then again, the initial iPhone didn’t have a camera either. Do you think that Apple didn’t include the camera because they didn’t want to? No, they probably did that because they had enough problems putting the iPhone together as it was, in time, without the camera. And judging from the few images I saw from the iPad, I’d say that the iPad is a comparable engineering masterpiece with little space inside the device and also with every engineering boundary pushed to the limits. Will we see a camera in iteration two of the iPad? I don’t know.

Will I buy the iPad? This largely depends on 1. how the german mobile carrier contracts will look like and 2. how apple will interpret the current dollar to euro conversion rate. But yes, I might buy one, if only to find out if the Appstore coder crowd will finally come up with some descent music apps. (Actually, after the iWork presentation, I was really hoping for a mobile GarageBand version).

An update on the web synthesizer

2010-01-25T23:00:00+00:00

After the last post on my crazy “synth in a browser” idea, I tried various other concepts to bring the idea to life. I am not very happy with the current state, but definitely see potential in some of the ideas.

After the rather brutal failure of the streaming idea (I guess, I could, with some serious hackery and a little more systematic approach, trim down latency to around 10sec which might be still way too much to be anywhere near beeing useful), I tried to explore other possibilities. If generating the sound on a server is out of question, I need to generate it in the client. Well, duh.

Now, conventional wisdom says that this is probably too much for the obvious client choice: Javascript. Plus, there’s no interface for “streaming” on the fly synthesized audio from javascript.

But, as it seems, conventional wisdom was invented before Google Chrome, the V8 javascript VM or Safari and the Nitro engine came to life. I personally think that conventional wisdom was invented for a IE 5.0.

So it should be at least worth a try. But even if it is possible to write a realtime software synthesis system in javascript (and I am only spilling a few beans when I tell you: it IS!), this leaves me with the problem of actually playing the generated sound.

My first idea was to write a Java applet. I hear you shake your head in a sad, slow motion, but here’s why: When the Hobnox guys (You know the Hobnox audio tool, right?) started the “Adobe, make some noise” campaign, because Flash 9 basically sucked for self generated audio, they actually used a Java applet to stream the sound.

Basic sound output in Java is not too complicated (You want to take a look at javax.music.sampled.*), but communicating between Javascript and a Java applet involves a very inefficient, very sucky and also very old interface: The infamous LiveConnect. I don’t know the details (and I have a vague feeling that I don’t really want to know) on how the communication actually takes place, but I suspect there is some heavy serialization/deserialization involved. This would explain why it was completely impossible to transfer the sound data as actual arrays of integers (or whatever sample format floats your boat). The only way I could actually bring the communication to life was to use Strings as byte buffers. The remaining encoding issues produce a very subtle, almost analogue sounding noise/hiss, but at least I was able to produce a proof of concept: Yes, it is possible to use Javascript in a very modern browser (Safari in this case) to calculate some basic soft synth algos in realtime. Currently this means a uninterpolated Sawtooth passing through a 4 pole state variable filter with a simple one stage envelope on volume and filter.

The drawbacks, apart from the (maybe fixable) encoding noise: While Java applets run (on OS X that is) in both Firefox and Safari, only Safari was able to handle the workload of the soft synth, Firefox simply falls flat on it’s belly, aka “slow script warning”. While in theory Google Chrome should be the ideal browser to run the synthesizer, it currently does not support Java applets on the Mac. While in theory, both Java and javascript are garbage collected languages, this obviously doesn’t mean that much in reality. I cannot rule out complete stupidity of me and my codes completely, but in my Book, browsers should not crash while simply using these two technologies.

Wanna take a look at my crappy Java and javascript hacks?

Okay, next try. Using flash was, at first not really an option, because my initial idea was to make the whole thing using mostly open technology. Plus: As written above, writing a realtime soft synth for flash is not a real challenge anymore: It has been done, more than once.

On the other hand, Flash has a near-to-ideal market penetration. So why not using a hidden flash applet the same way I wanted to use the Java applet: Simply playing the sound, while the heavy lifting is done in javascript.

At least I was able to use open source technology to try this: I recommend taking a look at haXe for your flash compiling needs.

To cut a long story short: Ongoing investigations, outlook cloudy with some sun during the night. Communication between Javascript and Flash generally seems to be a magnitude faster than between Javascript and Java, because I really was able to send a huge array of Floating Point numbers (!) from Javascript to flash. But you probably don’t want to do that. The CPU indicator in the menu bar looked more like a VU-Meter on the mixer of Paul von Dyk when I ran the example. Next I tried using integers instead of floats, but this only helped a bit. Currently I’m back to trying to do it with Strings but didn’t get it to work, again, I suspect some encoding problems.

I’ll also push it up to github.com as soon as it is usable and doesn’t chrash your browser right away.

So, if I really want to kick some cool javascript synthesizer code on the musichackday STHLM, I need to do a lot of work.

Google Map Maker TED Talk und Karten Vergleich

2010-01-24T13:44:58+00:00

Als Google Mapmaker vor einer Weile veröffentlicht wurde, habe ich selbst nicht viel davon mit bekommen. Abgesehen von dem Kommentar aus der Openstreetmap-Community, das Google sich nicht die Rechte an Daten sichern sollte, die von Nutzern freiwillig für Google erstellt werden. Ein berechtiger Einwand, den Google aber durch die Möglichkeit zum Datendownload zum Teil entkräftet.

Ein TED Talk hat mich jetzt aber noch einmal eine Blick auf Map Maker werfen lassen. In dem 1,5 Minuten kurzen Video beschreibt ein Google Mitarbeiter, das 2005 nur 15 % der Erde durch gute Karten erfasst waren. Ein Umstand, der die Hilfe in Krisensitutionen wie beim Tsunami oder aktuell in Haiti extrem erschwert. Seit dem Tsunami in 2005 haben in Google Map Maker tausende Freiwillige aus 170 Ländern Millionen von Informationsschnipseln zusammengetragen.

Visualiserung aus dem TED Talk: Entwicklung der Zeichnungen in Map Maker von 2005 bis 2009 (Vergrößern)

In dem Google Map Maker Launch Post gibt es eine weitere schöne Grafik, die das Potential visualsiert.

Karten Vergleich Google Map vs. Google Map Maker

Laut dem TED Talk war die Ursprungsidee oder -motivation also, in Kriesenregionen gutes Kartenmaterial bereitzustellen. Und tatsächlich wird man, wenn man heute die Google Map Maker-Seite besucht, dort auf Haiti und den Aufruf, Haiti sauber zu kartografieren, hingewiesen.

Das sieht dann so aus:

So sieht die Google Map Maker Karte von einem Ort in Haiti aus:

Und im Vergleich dazu die Google Maps des gleichen Ortes. Ich wüsste gerne welche Daten jetzt genauer sind.

Besonders schön finde ich, dass unten rechts im Map Maker die Personen aufgeführt werden, die an diesem Kartenausschnitt beteiligt waren. So kann man sich beispielsweise das Profil von Tony, einem in New York lebenden Studenten angucken, der unter anderem die Gemeindegrenzen für einen der Orte im Kartenausschnitt eingefügt hat.

Insgesamt ein wirklich schönes Projekt von Google mit beeindruckenden Ergebnissen.

Und jetzt, hier noch der TED Talk:

Lalitesh Katragadda: Making maps to fight disaster, build economies

Rails 3 @ DevHouseFriday

2010-01-21T01:39:57+00:00

Am Freitag ist es wieder soweit - das nächste DevHouseFriday steht an. Dieses mal sehen wir, zusammen mit Michael, hinter die Kulissen von Rails 3.0.

Wir treffen uns wie immer in unserem Büro:

Gothaer Allee 2
50969 Köln
Google Maps

Wer für die weiteren DevHouses Themenvorschläge hat, oder was präsentieren möchte, der ist jederzeit eingeladen das hier anzukündigen.

Weitere Infos gibt es natürlich auch auf Twitter und Twitter

Wir freuen uns auf zahlreichen Besuch!

Get rid off blue highlighting of focussed input fields in safari and google chrome

2010-01-16T21:16:03+00:00

If you want to get rid off the blue highlighting of focussed input fields in safari and google chrome you have to add the following style to your input fields:

input {
  outline: 0 none;
}
textarea {
  outline: 0 none;
}

The Hudson Siren - Small pieces loosely joined

2010-01-12T23:00:00+00:00

A strong, intrusive indicator for continuous integration failures is extremely useful to keep software quality high. People tend to forget the CI stage if nothing seems to go wrong. And while the usual CCTray-Stuff “kind of” works, a more physical approach may be in order for larger teams. While large “information radiator” screens might be desirable, we chose to go a lot cheaper. One of my colleagues, the recent Rails convert Christoph actually took the first, most important step and implemented the actual notification hardware, using freely available “police lights”, a simple USB driven relais board and a Linksys NSLU2. His work was pretty outstanding: It involved crosscompiling USBLib for the NSLU2 (not to forget installing Debian on it), wiring it all together and then writing some kind of network protocol for it. Now, the actual protocol is pretty simple: Open a telnet connection (aka TCP/IP-Socket) to a specific port and send a command like “BLINK 3 RED” to it. This is, of course, “Keep it simple stupid” aka KISS at it’s best.

Except that now we were faced with a problem: Our CI servers are outside of our office network and you probably don’t want to expose a relatively insecure protocol (with an implementation that’s very likely to contain bugs an attacker could exploit). So, portforwarding from our firewall was out of question. In a very KISSy move, my colleague Timo set up some SSH tunneling and then wrote a simple Hudson-Plugin which would then use the above mentioned protocol via the SSH tunnel. This worked for some time but then we split up our CI server installations (and after that only Java projects could use the Sirens) and anyway, this was a bad kludge, so when the tunnel went down some day due to a server restart, we never restarted it.

So, one day, I simply thought “what the heck” and implemented a siren web service in Sinatra. The web service would simply open the socket and send the command into our own network while it would sit on a server with a public facing Port 80. All in all it took something like 4 hours to implement all of it and deploy it on the server. The hardest problem was how to configure our OSX server to serve the app via Passenger and the built in Apache and then setting up the web configuration. The app takes POST requests on URLs like /red and /blue with additional params to specify style and length of the signal, and also has some GET actions that display help pages and utility forms to enable human endusers to engage the siren for testing purposes (or practical jokes on teams in crunch mode).

The only thing left to do was now to adapt the Hudson-Plugin from above to use the web service instead of the socket connection. And here, my rarely used generalisation instinct kicked in: Wouldn’t it be nice if we had a generic webhook plugin for hudson, so that we could trigger all kinds of actions with it? Yeah, probably, but the plugin is not-quite-finished, sorry.

So now, today, I took Timo’s source and ripped out the hardcoded socket code and replaced it with some hardcoded http code.

Which means that the mindmatters siren is back in action. And that’s a good thing.

Redis and Rails sitting on the Couch

2010-01-12T00:00:00+00:00

I've been spending some quality time with two of my new favorite tools lately (CouchDB and Redis, duh!), and while integrating them into Scalarium some needs and as a result some smaller hacks emerged. I don't want to deprive the world of their joy, so here they are.

First one is a tiny gem that will allow you to use Redis as a session store. What's so special about it, there's redis-store, right? Sure, but I couldn't for the life of me get to work reliably. Seems that's due to some oddity in Rack or something, at least that's where my interest of further investigating the issues faded, and I decided to just rip the code off MemCacheStore, and there you have it, redis-session-store. Rails-only and proud of it.

While working on it I constantly kept a monitor process open on Redis. Great feature by the way, if not awesome. I used telnet, and somehow I constantly managed to hit Ctrl-C in the terminal I had the telnet session open in. Reconnecting manually is tedious, so I give you my little redis-monitor script:

Incredibly simple, but saves those precious moments you'd waste typing everything by hand.

Last but not least, here's a hack-ish patch to make CouchPotato (great CouchDB Ruby library by the way) dump view queries into the log file. The ugly part at the end is me trying to get the log that's output at the end of each request to include the DB time for CouchDB queries.

It's not great, but works for now. We'll very likely include something decent into CouchPotato without hacking into ActionController like that. Unfortunately to get this far, there's really no other way. I tried faking ActiveRecord, but you open a whole other can of worm doing that, because a lot of code in Rails seems to rely on the existance on the ActiveRecord constant, assuming you're using the full AR stack when the constant is defined. Here's hoping that stuff is out the door in Rails 3. Haven't checked to be honest.

Dump that file into an initializer, and you're good to go (for the moment).

A most unusual hack

2010-01-08T23:00:00+00:00

Sometimes, you have to take a bath to make things come together. In my case, I was pondering porting the absolutely fabulous nanoloop to the iPhone platform. But that was only one piece of the puzzle. The other one was a podcast (Bitsundso) where one of the regulars (namely Thorsten Philipp) was explaining the brand new apple flavor of http streaming – Which is based on delivering a video or audio stream in small chunks (8-10 seconds seems to be a good value) and glueing them together using an extended version of m3u-playlists.

But first, a bit of background. Nanoloop is a music synthesizer/sequencer for the Gameboy, more specificly for the Gameboy Classic and the Gameboy advance (those are two different versions). I am a proud owner of one of the Nanoloop 2.3 cartridges, wich are handmade by Hamburgian resident and 8Bit-Superstar Oliver Wittchow. Editing music with the not-exactly-rich user interface of a handheld game console (a screen and 10 buttons: Left, Right, Up, Down, Start, Select, A, B, L and R) is challenging. Managable and, as Youtube or even my very own Soundcloud stream testifies, with usable results, but, nevertheless, far from being intuitive. Oliver did his best to streamline the experience and to make everything as consistent as possible, but still.

Now, my plan was (and still is) to either convince Oliver to port Nanoloop to the iPhone or to do a knock off of it myself. Here’s the problem: While porting the sound engine should not be a huge problem (both devices are using ARM architecture, plus the iPhone should be an order of a magnitude faster), the interface has to be completely different. Because the iPhone has no buttons. Well, strictly speaking it has four buttons and a rocker switch, but none of them are usable in your own software. What it has, though, is a touch screen. Multitouch, even.

So the real challenge of a nanoloop port would be to convert the user interface from a stricly “hold button A and use cursor keys to change the value of the current note” interface to a simple, elegant and intuitive touch interface. Because, you know, as unintuitive the button interface may seem at first, as with every game controller, after a few hours of usage, it all starts to make sense. And in the end, you are pretty fast using the damn thing. Because, there are worlds to rescue. Or songs to write. And you’d have to be extremely careful not to lose this sense of speed when converting the interface to a touch version.

Anyway, since I am a much better HTML+Javascript coder than I am an iPhone developer, I thought, well, let’s prototype the interface in HTML. I could even build a version that’s compatible to the iPhone, so one could check the usability on the target device.

And then I thought: Yeah, well, but why stop there. Why not building a web version of Nanoloop. With a serverside sound generator and with realtime streaming, using this cool new Apple technology.

So, here we go. In the following, I’ll describe parts of the puzzle and how I solved them. Please note that this is far from finished, all that’s there as I speak is a very rough prototype that has enough problems to make a hard man cry. Nevertheless, you can find the code on github .

The Interface

I started off with the interface. This is what the nanoloop interface looks most of the time:

The grey boxes are 16 steps of a simple step sequencer and the black small rectangles are the visualisation of the data you edit (the screen you look at is for editing per step pitch envelopes).

I decided to use the canvas element for the visualisation. I am still not sure if something like SVG (and raphael.gs for example) would not be better, but, hey, this is just a first shot. I also decided to use one canvas element per step. So it’s 16 little canvasses there, neatly aligned and centered on the screen. The rest is just a big fat, query based javascript hack that will need a lot of refactoring, very soon.

Don’t expect much of a good interface conversion, too. I just did the first thing that came to mind, namely popping up a bigger version of the step as soon as you click on one and closing it again after you click into this big step window. A real iphone version would most certainly allow such a click action as well as a “press-down, open big step, adjust values, release, close big step” variant. But, it’s a start.

The Synth

The next thing I tackled was the sound generation. I briefly thought about reusing my cmusic code from Soundbadge but I wasn’t sure if cmusic would be flexible enough for the stuff I had in mind. What I really wanted was using ruby for the sound generation. So I started to look for a way to render sound files from ruby. Remember: The goal was to output small chunks of audio in a format understandable by Quicktime. My thinking was “naw, encoding mp3 in ruby will probably be a dealbreaker, so simply render out wav files and encode them using a command line utility to something usable.”. And something great I found. The Ruby Application Archive spat out sndlib a library for a lot of sound related stuff written in C with bindings for Ruby and quite some other languages. You only get the source, so it’s configure>make time, but that was pretty easy. The first pifall was that the usual “sudo make install” didn’t install the ruby extension into the right place. So I had to manually copy the sndlib.bundle file (Yep, I’m on a mac, under Unix, this probably woulda been sndlib.so) to /usr/lib/ruby/site_ruby/1.8/universal-darwin10.0/ yadda, yadda. Be also prepared for what may very well be the worst documentation for a project, evar.

But the binding is actually quite nice. There’s a cool SoundData utility class which will act as a sample buffer which you can use to write to the sound files (or sound hardware, if you must).

It also contains quite a few utility functions to generate music and currently I’m using both the sawtooth generator to create the base sound for my synthesizer and the envelope generator.

Hacking the synth together was pretty easy. Writing the sound files in wav format, using the correct, fixed chunk size was a bit hairy and I also expect some bugs to lurk around there (more on that later). Bringing it all to work with Quicktime was a pretty bad experience. Not that anything in there is complicated (you most probably want to use ffmpeg for encoding the files although in theory lame should be fine), but the level of feedback you can expect from Quicktime if anything is wrong feels like it’s 1991 all over again. Error -120203 or something like that. Yeah, right, that one. The devil is in the detail, in my case, my playlist had a small syntax error.

The Streaming

Ah, while we’re at it. The playlist. To concat all the small bits of sound into a coherent music stream, you’ll need a playlist. since it is realtime streaming, it’s a playlist with a sliding window of stream urls in it. Let’s say you have generated 3 streaming parts up to now. The playlist would look something like this:

#EXTM3U
#EXT-X-TARGETDURATION:5
#EXT-X-ALLOW-CACHE:yes
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_1.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_2.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_3.ts

Now, if you have your 1000th part generated, the playlist would look like this (I left out the unimportant header bits)

#EXT-X-MEDIA-SEQUENCE:995
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_995.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_996.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_997.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_998.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_999.ts
#EXTINF:5,
http://halfmac.local/~jan/webloop/app/test_1000.ts

So, but how does the streaming work now? Easy. The client (The only client that can do this right now, is quicktime) loads the playlist, fetches the first few stream bits and starts playing them. As soon as it comes to the end of the playlist, it reloads the playlist which now should, ideally, contain links to new streaming bits. To let the client know where exactly he should look, you provide the #EXT-X-MEDIA-SEQUENCE tag.

Now, to be able to stream the generated sound, sound generation needs to be faster than realtime (because of the additional step of converting the wave to a streamable format). Luckily, this is, at least on my Core II Duo iMac, still the case. I’m most curious how well this will work on, let’s say, a vServer where I could deploy this stuff.

The Duct Tape

So I had the sound generation in place, the last thing to do was to glue the web interface and the sound generation together.

Sinatra to the rescue: Three actions, one to transmit changes from the frontend to the server, one to poll the data so that the web editor is updated if someone else at the other end of the world changes something in his view (yes, this is a distributed sound editor, please don’t get too exited…), one to fetch the playlist (Quicktime is a bit picky on the mime type the playlist has to be delivered in, that’s why I have to deliver the playlist with an action).

The last bit is almost boring: Safari, in it’s current incarnation, supports the html5 audio tag. What’s even better: It supports quicktime realtime streaming using the audio tag. Here’s the code:

<audio src="/playlist.m3u8" autoplay="autoplay" autobuffer="autobuffer" />

yip, that’s it. It starts playing (most of the time) as soon as you open the page.

Data storage for the pattern data is currently extremely high tech: A yaml file. A good idea would probably be to change this to an in-mem store (memcached, redis maybe?) and randomly write it to disk to have a backup.

The Drawbacks

Well, streaming always comes with latency. Since this http streaming doesn’t involve any kind of QoS control or other forms of stream control I don’t know shit about, http streaming seems to come with very much latency. I’m pretty sure you could tweak it by setting different chunk sizes and stuff, but here you go. Having a sound editor with over 20 seconds of latency is probably not very practical (although I must say that the nature of nanoloops editor makes it quite possible to work with it).

The other problem is that playback is not entirely bug free (as stated above). Sometimes it seems as if the chunks align perfectly, sometimes they don’t. I am not done debugging this, but maybe there’s an underlying problem I didn’t see in the first place.

The Future

First of all, the user interface code needs some mean refactoring. Then I could add different pattern views to allow editing more than one parameter. I have at least two parameters (Envelope and Sound) that are already present in the synth (and also in the pattern yaml file), so this should be fun and easy.

Next up: I was thinking a bit more about the synthesizer infrastructure and maybe I should try to aim for a bit more of a flexible infrastructure and a variable backend. The current solution works, but poorly and is extremely low tech. So, why not making the synth entirely realtime and stream based (makes it more fun to debug if you could listen directly to it) and then using the recipe of Thorsten Philipp to take a real stream and slash it into pieces. That way, I could also stream it as an icecast stream if I wanted to.

Ah, the possibilites. Good to know that there’s a music hackday coming up :)

The Video

Last but not least, I give you this:

Eine kleine schwedische Meiereikunde

2010-01-06T23:00:00+00:00

Jetzt wo der Herr Siepert nicht mehr nur Foodbloggt, könnte ich doch eigentlich auch mal anfangen, Foodzubloggen. Ich hab da leider nicht so viel kulinarische Sexyness beizutragen wie ebenjeniger, aber jeder hat mal klein angefangen.

Jeder, der mich ein bisschen besser kennt, weiss, das ich gefühlt ein halber Skandinavier bin. Ich bin fast jeden Sommer in Schweden, nach ’ner Woche da oben radebreche ich sogar ganz passabel Schwedisch. Auch Dänemark ist mir nicht fremd, im Gegensatz zu dem klassischen Dänemark-Urlauber kenne ich mich allerdings auf Seeland, Lolland und Falster wesentlich besser aus als auf dem “Festland”.

Nundenn. Wie es das Schicksal wollte, verbrachte ich den Jahreswechsel auch in Dänemark, und zwar in Lalandia, ein Ort, den der geneigte Individual-Reisende vermutlich ganz abscheulich findet (Think: Dänische Version der Center Parcs), der aber den Vorteil hatte, ein ganz hervorragendes Schwimmbad zu besitzen und Ferienhäuser mit freiem WLAN. Schon sehr Massentouristisch (und über den Jahreswechsel erschreckend voll) und für mich sicher kein Ort, an dem ich drei Wochen Sommerurlaub verbringen möchte. Für unsere spontane Flucht aus dem Großstadt-Sylvester-Wahnsinn war es jedenfalls ideal.

Aber eigentlich sollte es doch um Milchprodukte gehen, also wo war ich stehen geblieben? Ach ja. In Dänemark muss ich natürlich Ymer kaufen. Und dann kam meine Freundin auch noch mit Dingen aus Schweden (wo sie den ausgewanderten Teil Ihrer Familie besuchte). Und so entstand die Idee, dem geneigten Leser einmal kurz die Vorzüge der skandinavischen Meiereiprodukte nahezubringen. Denn ich stelle immer wieder gerade bei frischgebackenen Skandinavien-Neulingen fest, dass man sich nicht so recht rantraut an all die komischen Dinge, die sich da im Kühlregal stapeln. Was nicht weiter verwunderlich ist, denn erstens sind die Namen nicht selbsterklärend und zweitens ist die Schnittmenge mit den uns Deutschen vertrauten Milchprodukten nicht sonderlich groß.

Wie immer fängt alles mit Milch an. Die heisst in Schweden Mjölk (nicht zu verwechseln mit Mjöl, das ist Mehl, har har), in Dänemark Mælk und in Norwegen Melk. Natürlich gibt es auch in Schweden Fettreduzierte Milch, die heisst dann Lättmjölk (0,5% Fett) oder Mellanmjölk (1,5% Fett). In Norwegen kann man übrigens durchaus etwas derangiert vor dem Milchregal stehen: H-Melk ist dort Vollmilch (Helmelk) und eben nicht länger haltbar. Von allen skandinavischen Milch-Sorten ist die Norwegische übrigens, wie ich finde, die leckerste, was daran liegen könnte, das sie in der Regel bis zu 3.7% Fett hat. Eat that.

Kurzer Einschub: Die Milch kommt in Dänemark und Schweden überwiegend von Arlafoods. Arlafoods ist ein krasser Großkonzern, den man in Deutschland vor allem als Hersteller vom Frischkäse “Buko” kennt oder aber von der Butter “Kærgarden”. Arlafoods ist in 2000 aus der Schwedischen “Arla” und der Dänischen “MD Foods” enstanden. “MD Foods” stammt wiederum aus einer Fusion von MD Foods und Kløver Mæelk, den Namen und das Kleeblatt-Logo dürfte einigen langjährigen Dänemarkurlaubern durchaus ein Begriff sein. Die “Arla” wiederum hiess bis 1975 Mjölkcentralen.

Machen wir weiter mit den Gemeinsamkeiten der deutschen und skandinavischen Milchlandschaft: Sahne heisst in Schweden Grädde und in Dänemark Fløde. (Rød Grøte med Fløde anyone?). Schlagsahne kann man tatsächlich relativ direkt übersetzten: Vispgrädde und Piskefløde. Und dann: Gibt es eigentlich in normalen deutschen Supermärkten so etwas wie Kochsahne zu kaufen? Hansano behauptet zumindest, sie hätten sowas im Programm. Im wesentlichen handelt es sich dabei um eine etwas leichtere Sahne (die sich deswegen wohl auch nicht mehr zum aufschlagen eignet) mit ca. 15% Fettgehalt. Im Schwedischen haben wir es hier mit Matlagningsgrädde (oder kurz: Matgrädde) zu tun, im dänischen mit Matlavningsfløde (das musste ich jetzt tatsächlich gerade mal nachsehen).

Weiter in Richtung Fett: Butter heisst im dänischen Smør und im schwedischen Smör. Butter ist in Skandinavien traditionell gesalzen. Das muss man mögen, ich finde es großartig. In Schweden gibt es zwei Sorten: Normal- und Extragesalzen. (Normalsaltad, Extrasaltad)

Der interessante Teil kommt jetzt: Sowohl die Dänen, als auch die Schweden haben ihre ganz eigene Interpretation von Joghurt bzw. Dickmilch.

In Schweden heisst das Fil (ja, das Flachwitzpotential für Deutschsprachige Schwedenurlauber ist enorm) und ist relativ flüssig und recht Sauer (wird im Tetrapack verkauft), hat den selben Fettgehalt wie Milch (und gibt es auch in einer Lätt-Variante) und wird gern mit Früchten (z.B. frisch geflückten Blaubeeren) und Müsli gegessen. Fil gibt es auch mit Früchten. Darüberhinaus gibt es unzählige Fil- und Joghurt-Varianten, die fast alle in Tetrapacks verkauft werden (nur ein paar Joghurt-und-Müsli-Varianten gibts im Becher und Trinkjoghurt in Plastikflasche).

Eine erwähnenswerte Besonderheit ist die sog. Gräddfil, die mit 12% Fett und einer sehr cremigen Konsistenz gerade danach schreit, als Créme-Fraiche-Ersatz und als Grundlage für Dips herzuhalten (im Chipsregal finden sich in der Regel eine ganze Reihe von interessanten Pülverchen, mit denen man z.B. Gräddfil in Chili-Ananas-Dip verwandeln kann. Fragt nicht, war lecker.)

Hier mein übliches Frühstücksrezept: 1,2 Esslöffel Müsli, ein Löffel Preiselbeermarmelade und ’nen ordentlichen Schwung (Lätt-)Fil.

Kurzer Einschub zum Müsli: Das/der wird in Schweden fast ausschließlich in Mehl-ähnlichen Papierverpackungen verkauft. Meine Theorie ist, dass das der Grund ist, warum ich, im Gegensatz zu dem Plastiktüten-Seitenbacher-Krams aus Deutschland noch nie das Gefühl gehabt habe, dass das Müsli muffig schmeckt.

Die Dänen wiederum haben Ymer. Ymer schmeckt vollkommen anders als Fil, ist wesentlich dickflüssiger und unglaublich cremig, was wohl daher kommt, das zur Zubereitung von Ymer Magermilch vergoren wird und nach dem Reifungsvorgang Sahne zugesetzt wird um auf den Fettgehalt von ca. 3,5% zu kommen. Der Name Ymer kommt von , ich zitiere die Wikipedia, von dem Eisriesen Ýmir aus der nordischen Mythologie, der sich von der Milch der Urkuh Auðhumbla ernährte. Macht also alles Sinn. Ymer wird in Dänemark (und ich muss zugeben, dass ich das noch nie probiert habe) mit dem sog. Ymerdrys gegessen, was im wesentlichen gezuckertes zerkrümeltes Schwarzbrot ist. Aber auch in der Müsli-Marmeladen-Kombi schmeckt Ymer ausgezechnet. Von Ymer gibt es auch eine Light-Variante namens Ylette. Witzisch, die Dänen. Und dann gibt es noch A38, eine Joghurt-Sorte die nun wirklich den Preis für den absurdesten Produktnamen gewinnt. (Ich nehme an, es ist der technische Name der verwendeten Acidophilus-Kulturen).

Ach ja, in Norwegen gibt es auch so etwas ähnliches wie Fil, das ist jetzt lange her, aber ich habe sie noch etwas flüssiger und etwas saurer als Fil in Erinnerung und sie heisst Kulturmelk.

Noch erwähnenswert bei den flüssigen Dingen: Kakao. Die Dänen haben Mathilde, deren Maskottchen auch schon mal in einer älteren Demoscene-Produktion auftaucht. Für so eine vorgefertigte Schokomilch ist Mathilde wirklich krass gut.

Und dann: Käse. Die Dänen können da ja so einiges, ein alter Danbo ist schon was feines, oder auch der auch hierzulande recht bekannte Höhlenkäse. Noch besser: Der Blauschimmel. Wobei CASTELLO da gar nicht mein Favorit ist: Ich empfehle Høng – Wenn es was besonderes sein soll, gern auch den mit kandierten Birnenstücken. Und dann: Buko halt. Frischkäse. Nunja.

Die Schweden? Reden wir nicht drüber. Sie haben essbaren Käse, das soll man reichen. Was soll man von einem Land käsemäßig halten, wenn die beliebteste Sorte tatsächlich Haushaltskäse heisst? (Hushållsost). Seltsam ist die Begeisterung der Schweden für Hüttenkäse (Dieser Styropor-Frischkäse), was dazu führt, das es in Schweden Hüttenkäse in zahlreichen, mit Gewürzen und Früchten versetzten Varianten gibt. Ganz lecker, wenn man Hüttenkäse mag.

Interessanter da schon die Norweger: Die haben mit ihrem Jarlsberg einen ziemlich guten Käse mit sehr großen Löchern am Start (und dem klassischen Emmentaler durchaus nicht unähnlich), den man gottseidank sowohl in Schweden als auch in Deutschland oft bekommt. Aber das eigentlich interessante ist der Gudbrandsdalen, ein Ziegenkäse, bei dem bei der Herstellung die Molke und der Milchzucker karamellisiert wird. Gudbrandsdalen ist sehr schmierig und schmeckt tatsächlich wie eine Mischung aus einem Ziegenfrischkäse und, uhm, Karamellbonbons. Da es den bei uns früher im Urlaub eigentlich immer zu essen gab (wir wohnen in Schweden nur ca. 5 km von der norwegischen Grenze), dachte ich, bis ich ca. 15 war, das Ziegenkäse immer süß schmeckt. Heute bin ich dann doch eher ein Freund eines guten französischen, trocken gereiften Rohmilchkäses, aber ab und an nehme ich dann doch auch mal ein Stück von dem Karamellkäse.

Und damit wünsche ich guten Appetit auf der nächsten Skandinavienreise!

A christmas present: The website of Kuhn, Kammann & Kuhn

2009-12-30T23:43:48+00:00

What did you get for christmas?

We already made ourselves a first christmas present two weeks ago: the new website of Kuhn, Kammann & Kuhn - a Cologne-based business communication agency. Well, more precisely: the guys from Kuhn, Kammann & Kuhn made theirselves a first christmas present, but nevertheless it's also a great pleasure for us to see this jewel (ruby!) of a website working and growing in public.

Back in october, when Wendelin from KKK asked us to build their new agency website, we thought "well, usually we're building complex web apps, rarely corporate websites", but then their way to deal with both editorial and aggregated social media content in a constantly changing grid layout foo¹ grabbed our attention. And now, after eight days of exciting agile cooperation and several feature iterations, it's done - and online.

What's so special about this website?

On the one hand it's this mix of a tag-based navigation and a flexible grid of content objects. No more static sitemaps! Tags basically work as content filters, but it's not done at this point here: Every content object has its lifetime and is constantly fading until it isn't visible anymore. In addition to the tags and visibilities of an object, every collection of content objects is a well prepared composition of different content types, which gets slightly shuffled to look nicer without loosing the chronology of its items. Finally all objects are positioned and animated by a piece of JavaScript, which handles any combination of small and large boxes.

Hands on Superfeedr

Beside their editorial content, the KKK people want to aggregate a bunch of Twitter-Accounts and Weblogs constantly. Because we don't like the hassle with grabbing and parsing a growing number of feeds, we're using the Superfeedr API for the first time in this project. There's a handy (but incomplete) Rails plugin on Github you might want to use to subscribe and unsubscribe to feeds, but the HTTP POST notification parsing part isn't covered. So Feedzirra helped us to get down to the beef. After two weeks of running the app with Superfeedr we're not missing one feed entry, but the time period between publishing a tweet and Superfeedr's POST notification is often longer than the promised 15 minutes - yet. We'll see, seems like a painless feed grabbing solution anyway, if real-time doesn't mean instant delivery in your biz.

Enjoy surfing the new website of Kuhn, Kammann & Kuhn!

¹ It’s hard to find the right words. You have to ~~see~~ feel it…

Ruby En Rails 2009 Recap

2009-12-15T16:01:08+00:00

The last past days in Amsterdam for Ruby En Rails 2009 were really great.

I arrived on Thursday and had the chance to discuss a possible security vulnerability in Rails I discovered a while back with Yehuda and Mislav during lunch.

Afterwards we went to the conference dinner and met many very nice people and had a long discussion about voting systems, European vs. American culture, gun laws, and political systems.

Friday was the first conference day and started for us with being 40 minutes late as we headed up to the wrong metro station. Yehuda was supposed to give a keynote as the first session... luckily the organizers swapped the sessions. So unfortunately we missed the first session but were in time for Yehuda to give his keynote.

Yehuda talked about the Rails/Merb merge and dissected what was achieved. It was a very good, in-depth presentation about the new features of Rails 3 and what left to do. I really liked the new router DSL, actions being Rack-apps, and the death of Rails Metal (the new, slicker possibility to build simpler/smaller controllers takes care of this.).

After the coffee break I gave my Rails Security presentation about common attacks against web applications and how you can protect against them in Rails. Usually this is quite heavy stuff and people tend to sit quiet and listen as this is new to most. In Amsterdam I had some very good questions and discussions on session fixation, JavaScript high-jacking, and app reconnaissance.

Rails Security

View more documents from Jonathan Weiss.

Then we listened to Julio Javier Cicchelli talk about Rubyists.EU, an effort to make the different European Rails communities easier to find and build a common European community.

We went to lunch with James which gave us some time to catch up. Took us a while to find a decent place around the conference hall but we managed to find a restaurant where we tried to estimate the global financial burden due to Internet Explorer and what will happen to Microsoft if they were charged for the extra work needed done.

After lunch Eloy educated us about the current state of MacRuby. I'm really looking forward to spending some time hacking Mac apps in Ruby.

The next session gave an overview about monitoring, performance, and the different tools like request-log-analyzer or Nagios.

The conference was closed by Jeremy giving a keynote about Rails, Ruby, and the current state of things in the community (and a sneak-peak at ActiveRelation and what it will be able to do). It was very well received and a good closing session.

Afterwards a big group headed for dinner. We ended up in a small, very local restaurant. It included very nice food, good discussions with new friends from Finnland, and a waiter/owner who runs the restaurant homepage with Rails and asked Yehuda for help :-)

The next day was titled Geek Day and included many lightning talks (that ended up being nearly full sessions and had very good content). Parallel to the great sessions about MongoDB, DataMapper, or experience reports there was a Rails Rumble featuring five teams going on.

The most notable session of the second day was Justin Halsall, dressed for Halloween, talking about BlockHelpers and view DSLs. It was a hilarious show.

The Rumble was a great idea. In contrast to the usual Rumble the teams got a specific challenge. They should build something that improves the situation with Rails dependencies and out-of-date gems laying around in vendor/gems. The winning team would get two tickets to RailsConf 2010.

One team extended builder to list outdated gems and got their changes even merged back to builder by Yehuda on the same day. Another team extended Webistrano to accept projects dependencies and display them on the stage page. Some teams build a command line tool to extract local dependencies like your gems or even the MySQL version and push those definitions to a central place. The winning team had the the most advanced idea regarding update notification and gem-sets for applications. I could really see something like this being integrated into gemcutter. Congratulations Ludo and Michel!

Ruby En Rails day two was celebrated with a big dinner and drinks. After a very nice evening we headed back to the hotel as everybody had an early flight out. Some were still discussing Rails, some were lucky to be able to enjoy Amsterdam longer.

I really enjoyed the conference and Amsterdam. Thank you Chris and Tim Obdam for organizing the whole event!

Amazon EC2 HighMemory Instances

2009-12-15T16:01:02+00:00

Very very nice: 34.2 GB RAM and 68.4 GB RAM instances on EC2

# free -m
             total       used       free     shared    buffers     cached
Mem:         70007       2205      67801          0         28        595
-/+ buffers/cache:       1581      68425
Swap:            0          0          0

And

# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.760
cache size	: 8192 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca popcnt lahf_lm
bogomips	: 5336.34
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

[..]

We just included support for those machines in Scalarium, our EC2 cluster management plattform.

We are Railslove

2009-12-12T17:13:31+00:00

Since 2008 we have been building web applications for you. We’ve done some really great projects with our partners and we’re happy to see our own web service, channelthing, growing. So, we think it is time to introduce ourselves in more than 140 chars.

As you might know, we three, Tim, Michael and I, started Railslove two years ago. However, in our project we work with many other great web developers who deserve to get mentioned. So, to give you a better insight into our network, we’re going to reveal the “Railslove Family” over the next few weeks.

Yours,
Jan, Tim and Michael

Going solo

2009-12-10T17:42:46+00:00

A few months ago, after the birth of our daugther (yeah, Lotta!) and during my parental leave, thoughts arose about realizing an idea I’ve carried around with me since I finished my studies: Going solo, working on my own projects and earning a living as a freelance software developer. A few weeks ago I realized that plan and decided to quit my current job at neuland, though it wasn’t an easy decision: working for neuland has always been fun, especially because of the nice environment and the multitude of different personalities of the people working there – some of which are now close friends of mine. Nevertheless I felt that deep urge of doing something on my own and having sold venteria in summer makes taking the plunge a little less risky.

Alright, so where am I heading? A new focus will be iPhone/Cocoa app development and I’ll keep on working with Ruby on Rails – I’m really looking forward to all the great enhancements planned for Rails 3.0 and I just love doing web development, so that I won’t burn all the bridges

The bootstrapping went pretty well so far, I joined a really nice coworking environment and got an office at the infamous Abfertigung. As far as I can tell I’ll be surrounded by a bunch of very nice, creative and innovative people. Furthermore it’s good to know that I can, but don’t have to work from home – that makes meeting clients and finding quiet hours to get work done (I’m lookin’ atcha, Lotta) a lot easier

Hopefully I’ll find the time during the next weeks to write about some of the aspects of becoming a freelancer as I know some of you are interested in that and I’d like to take you along on that journey. I think it will be a pretty interesting time and I’ll learn tons about myself, meet nice and interesting people (including clients ) and most important of all: still have the time to enjoy my family… we shall see

Syncing Socialite across multiple Macs (using Dropbox)

2009-12-08T17:00:48+00:00

If you are working on multiple Macs (i.e. having a personal and one at work) you probably know this problem: You would like to sync some application data between these machines, so that you don’t have to manage separate todo-lists, items in your inbox that you’ve already read on one machine but are marked as unread on the other, etc. I’m using Dropbox (disclaimer: If you sign up using this link, I’ll get some more disk space, thank you!) to sync various applications (Things, Billings and Socialite for example), but you can use any other cloud storage solution if you like.

In this little tutorial I’ll explain how to sync the data with Dropbox and I’m using Socialite as an example. It basically works like this for any app that stores its data in an Application Support folder, but Socialite requires some extra steps people are frequently asking about, so here we go:

Go to your Dropbox folder and create a new folder called Synced Application Support (or whatever you like)
Quit any running instance of the application you would like to sync (in this case Socialite).
Got to your ~/Library/Application Support and move the Socialite folder into the Dropbox folder you just created (Synced Application Support)
Open your Terminal.app and symlink the moved folder:
$ ln -s ~/Dropbox/Synced\ Application\ Support/Socialite/ ~/Library/Application\ Support/Socialite
Start the app and see if it worked

After this five steps you should be up and running with most of your apps (syncing Things is as easy as that). Unfortunately Socialite requires some extra steps if you are using Twitter (and who isn’t these days), because it keeps the OAuth authentication tokens in the keychain, so that you need to sync them too. Fortunately this isn’t a big deal, because now you know how to sync your data, so go ahead and…

Quit Socialite
Open your Keychain and create a new keychain called Socialite and save it within your Synced Application Support/Socialite folder
Right click the new Socialite keychain, edit the settings and uncheck the checkboxes so that you won’t be bothered with authentication request every now and then
Use the search bar at the upper right corner to search for twitter.com and move all items used by Socialite to the new Socialite keychain
Start Socialite and see if your Twitter services are still working – if not: Delete all items in your Socialite keychain, remove the Twitter services from Socialite, restart Socialite, add them again and move the freshly created twitter.com items in your Login keychain to the Socialite keychain.

Okay, that’s it, I hope it’s helpful to some of you and I also hope that someday syncing will be baked into Socialite, though Realmac Software currenty doesn’t have any plans for implementing that.

Die Revolution wird nicht im Fernsehen...

2009-12-06T23:00:00+00:00

Ich hab mir gestern die Revolution angesehen. Und zwar die, die schon längst “am passieren iss”.
Und zwar war ich gestern beim “Holy Shit Shopping” im alten Gaswerk. Das ist der erste Weihnachtsmarkt, für den ich Eintritt bezahlen musste
und, ich denke auch der erste Weihnachtsmarkt, vor dem ich anstehen musste.

Der Anspruch der Macher ist, laut Website:

Die Weihnachts.Shopping.Lounge ist ein Kunst- und Designmarkt und präsentiert seit 6 Jahren ca. 150 handverlesene junge Designer, Kreative und Künstler, sowie eine wilde Mischung an Mode-, Schmuck- und Produktdesign, Kunst, Fotografie, Graphik, Comics und Literatur in besonderer Architektur.

Ja, Kunst gab es auch, Schmuck ’ne ganze Menge, aber ansonsten vor allem: Kunsthandwerk der modernen Variante. Also das ganze Etsy-/Dawanda-Gelöt. Und das ist schon beeindruckend, wie viel Kreativität dieses “Neue Kunsthandwerk” hervorbringt. Klar, es gibt ungefähr 200 Variationen der Handy-Hülle aus Filz, aber zumindest denkt jeder seine Idee mit dazu. Und richtig coole Sachen gabs auch, intelligentes Produktdesign, abgefahrene Ideen, gute und verrückte Mode, und und und. Wenn ich nicht so müde wäre, würde ich mir die Mühe machen und ein paar Sachen verlinken, stattdessen verweise ich auf die Links auf der Seite vom Holy Shit Shopping.

Das ist, für manche mag das anmaßend klingen, gelebte Revolution. Leise, still und heimlich etabliert sich gerade in diesem Bereich eine Kultur des Selbermachens. Des davon-leben-wollens und -könnens. Vorbei an der Offshore-Großindustrie, weil niemand mehr Lust hat, vergängliche Plastikdinge aus China zu kaufen, wenn man statt dessen mit Liebe gemachte, haltbare Dinge aus echten Materialien (t.w. auch hier: Plastik) haben kann. Und vor allem ist niemand da, der den Kleinstproduzenten, die über Etsy und Dawanda ihre Waren an den Mann bringen, erzählt, dass man nur erfolgreich sein kann, wenn man in Masse fertigt. Weil das, auch durch die fortschreitende Demokratisierung der Produktionsmittel, inzwischen gar nicht mehr so wichtig ist. Und selbst wenn: Viele Kleidungsstücke, die auf dem HSS verkauft werden, sind vermutlich nicht in Deutschland produziert. Aber auch hier: Der Zugang zu Produktionsmitteln, auch für die Massenfertigung ist auch immer einfacher geworden, durch z.B. das Internet, über das man inzwischen überall auf der Welt Produktionsanlagen mieten kann, aber auch durch den Druck der großen Konzerne an die “Zulieferindustrie”, immer flexibler zu werden – Und somit ihnen die Möglichkeit gegeben haben, auch flexibler in der Auftragsannahme zu werden. Und so scheint sich der Trend, der für Software schon ganz lange gilt, auch im Bereich der Hardware zu bestätigen: Kleinere Firmen sind im Durchschnitt flexibler, schneller, innovativer und damit mächtiger, während die Bereiche, in denen große Unternehmen bisher Vorteile hatten (Marketing-Macht, große Forschungsbudgets, Synergiemöglichkeiten im Unternehmen) immer unwichtiger werden, bzw. der Vorsprung immer kleiner wird.

Die einzige Macht, die große Unternehmen den kleinen immer noch voraushaben, ist zum einen, massiv Lobbying in der Politik zu betreiben (dazu mehr zu schreiben würde diesen Artikel sprengen) und, leider, Armadas von Anwälten zu bezahlen. Meine Vorhersage ist daher: Es wird weiterhin eine Großunternehmen-freundliche Klientelpolitik betrieben werden, und auf der anderen Seite werden, sobald die Industrie schnallt, was da passiert, anfangen, massivst die Kleinunternehmen wegen irgendwelcher Copyright- und Marktenrechtsverletzungen zu verklagen, wo immer sie können. Die Nummer mit Jack Wolfskin, wenn auch glimpflich ausgegangen, zeigt da ja schon, wie man den Kleinstproduzenten beikommt, wenn man will.

Wird das was ändern, wird das den Trend zu kleinen Unternehmen aufhalten? Ich denke und hoffe nicht. We’ll see.