Migrations, Import and Export, and shooting myself in the foot

There's this scary moment when you've started your blog and someone has read the first entry and you realize you've let yourself in for some work. So I had an idea. Perhaps the webteam would write my blog entry for me - you know like Tina the Techwriter doing her boss's blog in Dilbert.



So we chatted about Plone at our team meeting and over coffee during the week and things got rather deep. Now I've got ideas for about four posts .... but I've still got to write them myself.

Since the call for PLIPs has just come out, we thought a bit about Plone 4 and its implications for us. There's a great deal to say about this, but I might as well start at the beginning and talk about migration.

About 20% of my time, all the time, is spent on migration. We finally switched off our Plone 2.0 server at the end of last year. I'm still shifting sites from 2.5 to Plone 3. I'm now planning for Plone 4 and beyond.

Before you start asking me why I'm using this CMS if I find myself in a constant state of flux, let me explain that, in my opinion, this is just a fact of life. We've found it's bound up in the natural process of our organisation. Firstly expectations rise. This month's fashion, for instance, is rotating images. There's a great add-on for this collective.slideshowfolder, but, understandably it's only available for Plone 3. When our departments and units come asking for it, it's helpful for us to be in a situation where we can deliver - so, believe it or not, this can be a trigger to migrate the site. Secondly, the average life of a site seems to be about two years. After that, they're back asking for a make-over. There will be a new unit head, a new focus, a spurt of enthusiasm for the web, or someone in the department has asked why their site doesn't look like bbc.co.uk.

Right now, there's no reliable way to migrate a site unless you move the entire Data.fs. It's a great idea, one big bang and your site is transformed into a new shiny version. Fine, if it doesn't explode in your face and you can orchestrate everything and everyone into that one moment.
But, because we have multiple, independent sites, serving independent units and departments, migration has to happen as a staged process. It would be absolutely impossible in our scenario to contact everyone at once and say "we are migrating our sites on date x". Without a doubt someone would reply and say - "Sorry, that date is impossible we're recruiting new graduate students/undergoing an internal audit/do not want to migrate at this stage".

In my ideal content management world, then, migration becomes a process rather than an event with a separation of concerns.

I suspect that rule-based theming (Deliverance) will help here. I haven't tried this yet, but I hope that our themes won't be tied to a specific version of Plone, which means that we'll be able to implement technical changes without impact on our visual design and vice versa. I hope that it means that we can slot Plone into the design, rather than having to slot the design into Plone.

We could, for instance, have one department using a design with Plone 4 and another unit using the same design but with Plone 5. No heart-sinking moments any longer when the University changes its branding again ... and we have to weave all these changes into the Machievellian infrastructure of several different Plone themes.

The next bit to tackle then is content. It's very difficult to pin-point or isolate part of a Data.fs and where bits of a Data.fs are owned by different departments or units, trying to co-ordinate a big bang migration becomes very complicated. As a result, we've mounted each of our sites as a separate Data.fs. So we can deal with each unit and department as a separate project. This gives us an extra added bonus in that restoration of accidental deletions that have been overlooked for a few days (and of course always contain mission-critical content for that particular department) can be restored by replacing the Data.fs with no impact on other departments or users. However, it is also a hugely extravagant use of server resources - as we've discovered to our cost.

A proper import/export solution within Plone would really help us . Extracting small parts of the content of a site would allow us to stagger migration over a period of time. We could, for instance, set up a new site behind the scenes, allow us and the department to take their time getting used to it. Then once they're ready to run with it, drop in any content they'd created on their live site during that interim period.

I've been investigating Jarn's transmogrifier for just this reason. On its own, it's too abstract a solution for me to understand or implement. However, Quintagroup have made a usable Plone product based on it. It would be great to see a similar strategy as part of the Plone core. There'd be the added bonus that those (unlike me) who undertake a formal assessment of Plone before adopting it could tick the "Exit Strategy" box - that prenupt of the software procurement world.

Our extravagant Data.fs habit has now got us into deep water. I'm currently constructing a bid for additional funding to support a proper, resilient, hosting infrastructure for our sites. Thinking about migration and the pace of change, I've realised that this infrastructure will have to accommodate at least three versions of Plone at any one time. Asking Senior Management for funds is always a bit like Russion roulette - "Is there some cheaper, more cost effective way of doing this? What's this migration business? What's this constant change? Why aren't we using that commercial system we got free with our Groupware solution?" (you know the one I mean, it starts with "S" and ends with "t").

Even writing this blog and discussing it with the team, has given me some ideas about how to pre-empt these arguments, but you can see how worrying they can be. Change is inevitable, but big, complicated changes in complex environments are difficult to manage. It would be good to demonstrate to Senior Management that Plone is ideally positioned to handle this well.

Fatal Attraction

Well, we met on the internet. It was one of those casual, chance encounters. We weren't formally introduced and I can't exactly tell you the moment when I realised that this was it - what I'd been looking for. One brief, sublime download later and a few heady days of discovery over the Christmas holidays and Plone and I were definitely an item.

That was a few years ago now, we were both young (ish) and innocent and we've come a long way since then. In my University, I was one of the early adopters of content management (although that's not saying much as our University is never really "early" about anything). So I tell my colleagues, when they are casting about for the "perfect" content management system, that they need to stop playing the field and make a commitment. Content management is like marriage - it's for the long term, it's a challenge, it's not perfect and divorce is likely to be a bit messy.

So did I foist this grand passion on my unsuspecting family in the University? To a certain extent, yes, but they've got along OK and grown to appreciate Plone's strengths and accept its weaknesses. We've accepted the compromises. Plone has suited our use-case.

We offer a "service" to a number of departments and units in one Division of our University, helping them create public facing websites. There's two of us now full-time, me as project manager and integrator, my colleague as designer. Right now we also have a part-time trainer and support officer. If we need serious development work we have to buy it in on a fairly limited budget.

We've got about 50 websites in our CMS at the moment and about 150 web editors. Key to the operation of the University is the concept of subsidiarity, to devolve decision-making wherever appropriate. To put it bluntly, we can't "tell" anyone to do anything. Each department or unit is entitled to express its own individuality, and, if we think it would be a good idea for things to proceed in a certain direction, we must persuade and encourage. So the obvious outcome is that each of these 50 websites is subtly different. This is a lavish way to behave and a tremendous strain on resources and talent - it is no place for the systems thinker or the control freak. You have to go with the flow.

We are evolving a core set of products to underpin the production of these sites, but we also need to provide plenty of straightforward customization options. When we design or develop anything, we have to have one eye open to the long term implications - how will this affect the other sites on our servers, where can we build in flexibility and choice, what kind of future maintenance are we looking at? The sites evolve too. Enthusiasms wax and wane. We don't just walk in, undertake a project, finish it and walk away. We get people started on thinking about their site - give them a site to play with and let them progress things at their own pace.

Plone has been good at this. Lots of options for through-the-web customization. Simple ways to override things through acquisition. A bit of a learning curve for the integrator, but powerful enough with a bit of Python and a lot of templating. Then, a straightforward user-interface which most people "get" within an hour or so of being introduced.

Just recently though, I've noticed a shift. We're both of us getting bigger and more sophisticated, but I'm worried that we're growing apart. Even the simplest things seem so much more complicated and our Division's growing needs, to create good looking, flexible sites, seem so much harder to meet. I'm juggling viewlets, portlets, Python classes, trying to differentiate between things I can change through the web, things I can't. And ... none of my friends seem to take to Plone anymore, they're introduced, but that's it - first impressions, too complex, too overwhelming. And ... I'm really worried about the future.

It's time to work things out. I moan, grumble, worry, consider my options. But I've realised that I've got to stop expecting Plone to read my mind and be a bit more open about what I, the integrator, and my team, the designer and the trainer, and our clients, the Division really want and need. It's easy to complain, much harder to articulate what it is that's going wrong.

Plone, we need to talk. Will you listen?