Migrations, Import and Export, and shooting myself in the foot

There's this scary moment when you've started your blog and someone has read the first entry and you realize you've let yourself in for some work. So I had an idea. Perhaps the webteam would write my blog entry for me - you know like Tina the Techwriter doing her boss's blog in Dilbert.



So we chatted about Plone at our team meeting and over coffee during the week and things got rather deep. Now I've got ideas for about four posts .... but I've still got to write them myself.

Since the call for PLIPs has just come out, we thought a bit about Plone 4 and its implications for us. There's a great deal to say about this, but I might as well start at the beginning and talk about migration.

About 20% of my time, all the time, is spent on migration. We finally switched off our Plone 2.0 server at the end of last year. I'm still shifting sites from 2.5 to Plone 3. I'm now planning for Plone 4 and beyond.

Before you start asking me why I'm using this CMS if I find myself in a constant state of flux, let me explain that, in my opinion, this is just a fact of life. We've found it's bound up in the natural process of our organisation. Firstly expectations rise. This month's fashion, for instance, is rotating images. There's a great add-on for this collective.slideshowfolder, but, understandably it's only available for Plone 3. When our departments and units come asking for it, it's helpful for us to be in a situation where we can deliver - so, believe it or not, this can be a trigger to migrate the site. Secondly, the average life of a site seems to be about two years. After that, they're back asking for a make-over. There will be a new unit head, a new focus, a spurt of enthusiasm for the web, or someone in the department has asked why their site doesn't look like bbc.co.uk.

Right now, there's no reliable way to migrate a site unless you move the entire Data.fs. It's a great idea, one big bang and your site is transformed into a new shiny version. Fine, if it doesn't explode in your face and you can orchestrate everything and everyone into that one moment.
But, because we have multiple, independent sites, serving independent units and departments, migration has to happen as a staged process. It would be absolutely impossible in our scenario to contact everyone at once and say "we are migrating our sites on date x". Without a doubt someone would reply and say - "Sorry, that date is impossible we're recruiting new graduate students/undergoing an internal audit/do not want to migrate at this stage".

In my ideal content management world, then, migration becomes a process rather than an event with a separation of concerns.

I suspect that rule-based theming (Deliverance) will help here. I haven't tried this yet, but I hope that our themes won't be tied to a specific version of Plone, which means that we'll be able to implement technical changes without impact on our visual design and vice versa. I hope that it means that we can slot Plone into the design, rather than having to slot the design into Plone.

We could, for instance, have one department using a design with Plone 4 and another unit using the same design but with Plone 5. No heart-sinking moments any longer when the University changes its branding again ... and we have to weave all these changes into the Machievellian infrastructure of several different Plone themes.

The next bit to tackle then is content. It's very difficult to pin-point or isolate part of a Data.fs and where bits of a Data.fs are owned by different departments or units, trying to co-ordinate a big bang migration becomes very complicated. As a result, we've mounted each of our sites as a separate Data.fs. So we can deal with each unit and department as a separate project. This gives us an extra added bonus in that restoration of accidental deletions that have been overlooked for a few days (and of course always contain mission-critical content for that particular department) can be restored by replacing the Data.fs with no impact on other departments or users. However, it is also a hugely extravagant use of server resources - as we've discovered to our cost.

A proper import/export solution within Plone would really help us . Extracting small parts of the content of a site would allow us to stagger migration over a period of time. We could, for instance, set up a new site behind the scenes, allow us and the department to take their time getting used to it. Then once they're ready to run with it, drop in any content they'd created on their live site during that interim period.

I've been investigating Jarn's transmogrifier for just this reason. On its own, it's too abstract a solution for me to understand or implement. However, Quintagroup have made a usable Plone product based on it. It would be great to see a similar strategy as part of the Plone core. There'd be the added bonus that those (unlike me) who undertake a formal assessment of Plone before adopting it could tick the "Exit Strategy" box - that prenupt of the software procurement world.

Our extravagant Data.fs habit has now got us into deep water. I'm currently constructing a bid for additional funding to support a proper, resilient, hosting infrastructure for our sites. Thinking about migration and the pace of change, I've realised that this infrastructure will have to accommodate at least three versions of Plone at any one time. Asking Senior Management for funds is always a bit like Russion roulette - "Is there some cheaper, more cost effective way of doing this? What's this migration business? What's this constant change? Why aren't we using that commercial system we got free with our Groupware solution?" (you know the one I mean, it starts with "S" and ends with "t").

Even writing this blog and discussing it with the team, has given me some ideas about how to pre-empt these arguments, but you can see how worrying they can be. Change is inevitable, but big, complicated changes in complex environments are difficult to manage. It would be good to demonstrate to Senior Management that Plone is ideally positioned to handle this well.