Generating Plone theming mockups with Chameleon

  • 0

Some days ago there was a question at the Plone IRC-channel, whether the Plone theming tool supports template inheritance [sic]. The answer is no, but let's play a bit with the problem.

The prefered theming solution for Plone,, is based on Diazo theming engine, which allows to make a Plone theme from any static HTML mockup. To simplify a bit, just get a static HTML design, write a set of Diazo transformation rules, and you'll have a new Plone theme.

The ideal behind this theming solution is to make the theming story for Plone the easiest in the CMS industry: Just buy a static HTML design and you could use it as a theme as such. (Of course, the complexity of the required Diazo transformation rules depends on the complexity of the theme and themed content.)

But back to the original problem: Diazo encourages the themer to use a plenty of different HTML mockups to keep the transformation rules simple. One should not try to generate theme elements for different page types in Diazo transformation rules, but use dedicated HTML mockups for different page types. But what if the original HTML design came only with a very few selected mockups, and creating the rest from those is up to you. You could either copy and paste, or...

Here comes a proof of concept script for generating HTML mockups from TAL using Chameleon template compiler (and Nix to remove need for virtualenv, because of Python dependencies).

But at first, why TAL? Because METAL macros of TAL can be used to make the existing static HTML mockups into re-usable macros/mixins with customizable slots with minimal effort.

For example, an existing HTML mockup:

Here be dragons.

Could be made into a re-usable TAL template (main_template.html) with:

<metal:master define-macro="master">
<div metal:define-slot="content">
Here be dragons.

And re-used in a new mockup with:

<html metal:use-macro="main_template.macros.master">
<div metal:fill-slot="content">
Thunderbirds are go!

Resulting a new compiled mockup:

Thunderbirds are go!

The script maps all direct sub-directories and files with .html suffix in the same directory with the compiled template into its TAL namespace, so that macros from those can be reached with METAL syntax metal:use-macro="filebasename.macros.macroname" or metal:use-macro="templatedirname['filebasename'].macros.macroname".

Finally, here comes the example code:

#! /usr/bin/env nix-shell
#! nix-shell -i python -p pythonPackages.chameleon pythonPackages.docopt pythonPackages.watchdog
"""Chameleon Composer

Copyright (c) 2015 Asko Soukka <>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.


  ./ <filename>
  ./ src/front-page.html
  ./ <source> <destination> [--watch]
  ./ src build
  ./ src build --watch

from __future__ import print_function
from chameleon import PageTemplateFile
from chameleon import PageTemplateLoader
from docopt import docopt
from watchdog.observers import Observer
from watchdog.observers.polling import PollingObserver
from watchdog.utils import platform
import os
import sys
import time

def render(template):
    assert os.path.isfile(template)

    # Add siblings as templates into compilation context for macro-use
    context = {}
    dirname = os.path.dirname(template)
    for name in os.listdir(dirname):
        path = os.path.join(dirname, name)
        basename, suffix = os.path.splitext(name)
        if os.path.isdir(path):
            context[basename] = PageTemplateLoader(path, '.html')
        elif suffix == '.html':
            context[basename] = PageTemplateFile(path)

    return PageTemplateFile(template)(**context).strip()

class Composer(object):
    def __init__(self, source, destination):
        self.source = source
        self.destination = destination
        self.mapping = {}

    def update(self):
        source = self.source
        destination = self.destination
        mapping = {}

        # File to file
        if os.path.isfile(source) and os.path.splitext(destination)[-1]:
            mapping[source] = destination

        # File to directory
        elif os.path.isfile(source) and not os.path.splitext(destination)[-1]:
            mapping[source] = os.path.join(
                os.path.splitext(os.path.basename(source))[0] + '.html'

        # Directory to directory
        elif os.path.isdir(source):
            for filename in os.listdir(source):
                path = os.path.join(source, filename)
                if os.path.splitext(path)[-1] != '.html':
                mapping[path] = os.path.join(
                    os.path.splitext(os.path.basename(path))[0] + '.html'

        self.mapping = mapping

    def __call__(self):
        for source, destination in self.mapping.items():
            if os.path.dirname(destination):
                if not os.path.isdir(os.path.dirname(destination)):
            with open(destination, 'w') as output:
                print('{0:s} => {1:s}'.format(source, destination))

    # noinspection PyUnusedLocal
    def dispatch(self, event):
        # TODO: Build only changed files

    def watch(self):
        if platform.is_darwin():
            observer = PollingObserver()  # Seen FSEventsObserver to segfault
            observer = Observer()
        observer.schedule(self, self.source, recursive=True)
            while True:
        except KeyboardInterrupt:

if __name__ == '__main__':
    arguments = docopt(__doc__, version='Chameleon Composer 1.0')

    if arguments.get('<filename>'):

    composer = Composer(arguments.get('<source>'),

    if arguments.get('--watch'):
        print('Watching {0:s}'.format(arguments.get('<source>')))

Building Docker containers from scratch using Nix

  • 0

Nix makes it reasonable to build Docker containers from scratch. The resulting containers are still big (yet I heard there's ongoing work to make Nix builds more lean), but at least you don't need to think about choosing and keeping the base images up to date.

Next follows an example, how to make a Docker image for Plone with Nix.

Creating Nix expression with collective.recipe.nix

At first, we need Nix expression for Plone. Here I use one built with my buildout based generator, collective.recipe.nix. It generates a few exression, including plone.nix and plone-env.nix. The first one is only really usable with nix-shell, but the other one can be used building a standalone Plone for Docker image.

To create ./plone-env.nix, I need a buildout environment in ./default.nix:

with import <nixpkgs> {}; {
  myEnv = stdenv.mkDerivation {
    name = "myEnv";
    buildInputs = [
    shellHook = ''
      export SSL_CERT_FILE=~/.nix-profile/etc/ca-bundle.crt

And a minimal Plone buildout using my recipe in ./buildout.cfg:

extends =
parts = plone
versions = versions

recipe = plone.recipe.zope2instance
eggs = Plone
user = admin:admin

recipe = collective.recipe.nix
eggs =

zc.buildout =
setuptools =

And finally produce both plone.nix and the required plone-env.nix with:

$ nix-shell --run buildout

Creating Docker container with Nix Docker buildpack

Next up is building the container with our Nix expression with the help of a builder container, which I call Nix Docker buildpack.

At first, we need to clone that:

$ git clone
$ cd nix-build-pack-docker

And build the builder:

$ cd builder
$ docker build -t nix-build-pack --rm=true --force-rm=true --no-cache=true .
$ cd ..

Now the builder can be used to build a tarball, which only contains the built Nix derivation Plone. Let's copy the created plone-env.nix into the current working directory and run:

$ docker run --rm -v `pwd`:/opt nix-build-pack /opt/plone-env.nix

After a while, that directory should contain file called plone-env.nix.tar.gz, which only contains two directories in its root: /nix for the built derivation and /app for easy access symlinks, like /app/bin/python.

Now we need ./Dockerfile for building the final Plone image:

FROM scratch
ADD plone.env.nix.tar.gz /
USER 1000
ENTRYPOINT ["/app/bin/python"]

And finally, a Plone image can be built with

$ docker build -t plone --rm=true --force-rm=true --no-cache=true .

Running Nix-built Plone container

To run Plone in a container with the image built above, we still need the configuration for Plone. We can the normal buildout generated configuration, but we need to

  1. remove from parts/instance.
  2. fix paths to match in parts/instance/zope.conf to match the mounted paths in Docker container (/opt/...)
  3. create some temporary directory to be mounted into container

Also, we need a small wrapper to call the Plone instance script, ./, because we cannot use the buildout generated one:

import sys
import plone.recipe.zope2instance.ctl

    ['-C', '/opt/parts/instance/etc/zope.conf']
    + sys.argv[1:]

When these are in place, within the buildout directory, we should now be able to run Plone in Docker container with:

$ docker run --rm -v `pwd`:/opt -v `pwd`/tmp:/tmp -P plone /opt/ fg

The current working directory is mapped to /opt and some temporary directory is mapped to /tmp (because our image didn't contain even a /tmp).

Note: When I tried this out, for some reason (possibly because VirtualBox mount with boot2docker), I had to remove ./var/filestorage/Data.fs.tmp between runs or I got errors on ZODB writes.

Creating Nix-expressions with buildout

  • 0

The greatest blocker for using Nix or complex Python projects like Plone, I think, is the work needed to make all required Python-packages (usally very specific versions) available in nix expression. Also, in the most extreme, that would require every version for every package in PyPI in nixpkgs.

Announcing collective.recipe.nix

collective.recipe.nix is my try for generating nix expressions for arbitrary Python projects. It's an experimental buildout recipe, which re-uses zc.recipe.egg for figuring out all the required packages and their dependencies.

Example of usage

At first, bootstrap your environment by defining python with buildout in ./default.nix:

with import <nixpkgs> {}; {
  myEnv = stdenv.mkDerivation {
    name = "myEnv";
    buildInputs = [
    shellHook = ''
      export SSL_CERT_FILE=~/.nix-profile/etc/ca-bundle.crt

And example ./buildout.cfg:

parts =

recipe = collective.recipe.nix
eggs = i18ndude

recipe = collective.recipe.nix
eggs = zest.releaser[recommended]

recipe = collective.recipe.nix
eggs = robotframework
propagated-build-inputs =

recipe = collective.recipe.nix
eggs = sphinx
propagated-build-inputs =

Run the buildout:

$ nix-shell --run buildout

The recipe generates three kind of expressions:

  • default [name].nix usable with nix-shell
  • buildEnv based [name]-env.nix usable with nix-build
  • buildPythonPackage based [name]-package.nix usable with nix-env -i -f

So, now you should be able to run zest.releaser with:

$ nix-shell releaser.nix --run fullrelease

You could also build Nix-environment with symlinks in folder ./releaser or into a Docker image with:

$ nix-build releaser-env.nix -o releaser

Finally, you could install zest.releaser into your current Nix-profile with:

$ nix-env -i -f releaser-zest_releaser.nix

Stateless Nix environments revisited

  • 0

It's almost a year, since I tried to bend Nix package manager to fit my own workflows for the first time. I disliked the recommended way of describing nix environments in global configuration and activating and deactivating them in statefull way. Back then, I worked my way around by defining a wrapper to make local nix-expressions callable executables.

Consider that deprecated.

Nix 1.9 introduced shebang support to use Nix-built interpreter in callable scripts. This alone is a major new feature and solves most of my use cases, where I wanted to define required Nix-dependencies locally, as close to their usage as possible.

Still, I do have a one more use case: For example, want to run make with an environment, which has locally defined Nix-built dependencies. Because the make in this particular example results just a static PDF file, it does not make sense to make that project into a Nix derivation itself. (Neither does it make much sense to make Makefile an executable.)

Of course, I start with defining my dependencies into a Nix derivation, and to make that more convenient with nix-shell, I save that into a file called ./default.nix:

with import <nixpkgs> {}; {
  myEnv = stdenv.mkDerivation {
    name = "myEnv";
    buildInputs = [
      (texLiveAggregationFun { paths = [
      (rWrapper.override { packages = with rPackages; [

Now, I can run make in that environment in a stateless manner with:

$ nix-shell --pure --run "make clean all"

Unfortunately, while that works, it's a bit long command to type every time.

Initially, I would have preferred to be able to define local callable script named ./make, which was possible with my old approach. Yet, this time I realized, that I can reach almost the same result by defining the following bash function to help:

function nix() { echo nix-shell --pure --run \"$@\" | sh; }

or with a garbage collection root to avoid re-evaluation the expression on every call:

function nix() {
    if [ ! -e shell.drv ]; then
        nix-instantiate --indirect --add-root $PWD/shell.drv
    echo nix-shell $PWD/shell.drv --pure --run \"$@\" | sh;

With this helper in place, I can run any command from the locally defined default.nix with simply:

$ nix make clean all

Customize Plone 5 default theme on the fly

  • 0

When I recently wrote about, how to reintroduce ploneCustom for Plone5 TTW (through the web) by yourself, I got some feedback that it was the wrong thing to do. And the correct way would always be to create your custom theme.

If you are ready to let the precious ploneCustom go, here's how to currently customize the default Barceloneta theme on the fly by creating a new custom theme.

Inherit a new theme from Barceloneta

So, let's customize a brand new Plone 5 site by creating a new theme, which inherits everything from Barceloneta theme, yet allows us to add additional rules and styles:

  1. Open Site Setup and Theming control panel.

  2. Create New theme, not yet activated, with title mytheme (or your own title, once you get the concept)

  3. In the opened theme editor, replace the contents of rules.xml with the following code:

    <?xml version="1.0" encoding="UTF-8"?>
      <!-- Import Barceloneta rules -->
      <xi:include href="++theme++barceloneta/rules.xml" />
      <rules css:if-content="#visual-portal-wrapper">
        <!-- Placeholder for your own additional rules -->
  4. Still in the theme editor, add New file with name styles.less and edit and Save it with the following content:

    /* Import Barceloneta styles */
    @import "++theme++barceloneta/less/barceloneta.plone.less";
    /* Customize navbar color */
    @plone-sitenav-bg: pink;
    @plone-sitenav-link-hover-bg: darken(pink, 20%);
    /* Customize navbar text color */
    .plone-nav > li > a {
      color: @plone-text-color;
    /* Customize search button */
    #searchGadget_form .searchButton {
      /* Re-use mixin from Barceloneta */
      .button-variant(@plone-text-color, pink, @plone-gray-lighter);
    /* Inspect Barceloneta theme (and its less-folder) for more... */

But before activating the new theme, there's one more manual step to do...

Register and build a new LESS bundle

We just created a new LESS file, which would import the main Barceloneta LESS file at first, and then add our own additional styles with using some features from LESS syntax. To actually make that LESS file into a usable CSS (through the browser), we need register a new bundle for it, and build it:

  1. Open Site Setup and Resource Registries control panel.

  2. Add resource with name mytheme and a single CSS/LESS file with path ++theme++mytheme/styles.less to locate the file we just added into our theme:
  3. Save.

  4. Add bundle with name mytheme, requiring mytheme resoure, which we just created and Does your bundle contain any RequireJS or LESS files? checked:
  5. Save.

  6. Build mytheme bundle.

Now you should be ready to return back to Theming control panel, activate the theme, and see the gorgeous pink navigation bar:

Note: To really be a good citizen and follow the rules, there's a few additional steps:

  1. Add production-css setting into your theme's manifest.cfg to point to the compiled CSS bundle:

    title = mytheme
    description =
    production-css = /++plone++static/mytheme-compiled.css
  2. In Resource Registries, disable mytheme bundle by unchecking its Enabled checkbox and clicking Save.

  3. Deactivate and activate the theme once.

Technically this changes the CSS bundle to be registered as a so called Diazo bundle instead of a regular bundle. The difference is that Diazo bundle is always rendered last and can therefore override any CSS rule introduced the other enabled bundles. Also, as a Diazo bundle it get disabled and enabled properly when the active gets changed.

ploneCustom for Plone 5

  • 0

No more custom skins folder with infamous ploneCustom in Plone 5, they said.

Well, they can take away the skins folder, but they cannot take away our ploneCustom. I know, that the recommended way of customizing Plone 5 is via a custom theme through the Theming control panel from Site Setup. Still, sometimes you only need to add a few custom rules on top of an existing theme and creating a completely new theme would feel like an overkill.

Meet the new resource registry

One of the many big changes in Plone 5 is the completely new way how CSS and JavaScript resources are managed. Plone 5 introduces a completely new Resource Registries control panel and two new concepts to manage CSS ja JavaScipt there: resources and resource bundles.

Resource is a single CSS / LESS file, a single JavaScript file, or one of both, to provide some named feature for Plone 5. For example, a new embedded JavaScript based applet could be defined as a resource containing both its JavaScript code and required CSS /LESS stylesheet. In addition to those single files, JavaScript-files can depend on named requirejs modules provided by the other resources. Also LESS files can include any amount of available other LESS files. (LESS is superset of CSS with some optional superpowers like hierarchical directives, variables or optimized includes.)

Resource Bundle is a composition of named resources, which is eventually built into a single JavaScript and/or CSS file to be linked with each rendered page. When the page is rendered, bundles are linked (using either script-tags or stylesheet link-tags) in an order depending on their mutual dependencies. Bundles can be disabled and they can have conditions, so bundles are somewhat comparable to the legacy resource registry registrations in Plone 4 and earlier.

Now that you should be familiar with the concepts, you can bring our precious ploneCustom back to life.

Defining the next generation ploneCustom

These steps will define a new ploneCustom bundle, which provides both a custom CSS (with LESS) and a custom JavaScript file to allow arbitrary site customizations without introducing a new theme.

Creating and editing

At first, you need to add the actual LESS and JavaScript files. Instead of the deprecated skins custom folder you can add them into your Plone 5 site by using the old friend, ZMI (Zope Management Interface).

If you are running evelopment site, please, open the following url: https://localhost:8080/Plone/portal_resources/manage_main

This portal_resources is the new database (ZODB) based storage for any kind of custom resources (introduced with the new Theming control panel in Plone 4.3). Its functionality is based on plone.resource, but right now you only need to know how to use it with Plone 5 resource registries.

  1. So, in portal_resources, add a new BTreeFolder2 with name plone:
  2. Then navigate into that folder (select plone and press Edit button) and add an another BTreeFolder2 with name custom and navigate into that folder until you are at portal_resources/plone/custom:
  3. Now Add a new File named ploneCustom.js and another named ploneCustom.less:
  4. And, finally, you can navigate into those files (select and press Edit button) to edit and save them with your CSS and JavaScript:

    The example JavaScript above would only annoy to to tell that it works:

    jQuery(function($) {
        alert("Hello World!");

    The example CSS above would replace the portal logo with a custom text:

    #portal-logo:before {
      display: inline-block;
      content: "My Plone Site";
      font-size: 300%;
    #portal-logo img {
      display: none;

    In addition to that, you could add a little bit extra to learn more. These following lines would re-use button classes from Bootstrap 3 resources shipped with with Plone 5 (beta). This is an example of how to use LESS to cherry pick just a few special CSS rules from Bootstrap 3 framework and apply them next to the currently active theme:

    @import (reference) "../++plone++static/components/bootstrap/less/bootstrap.less";
    #searchGadget_form .searchButton {

Registering and enabling

To register the resource and add it into a bundle (or create a new one), go to Resource Registries control panel (e.g. at http://localhost:8080/@@resourceregistry-controlpanel). Click Add resource to show the add resource form and fill it like in the screenshot below:

Note that the strings ++plone++custom/ploneCustom.js and ++plone++custom/ploneCustom.less are actually relative (public) URLs for the resources you just added into portal_resources.

After saving the resoure by clicking Save, click Add bundle to create a new bundle for your ploneCustom-resource. Fill-in the opened form as follows:

Note that the bundle depends on Plone bundle. That makes it getting loaded only after Plone bundle, which includes jQuery, which our custom JavaScript code depends on. (Later you may wonder, why jQuery was not required with requirejs. That would also work and is recommended for other libraries, but currently you can rely on jQuery being globally available after Plone bundle has been loaded.)

When you have saved the new ploneCustom resource bundle, it will appear into the Bundles list on the left. The final step is to click the Build button below the ploneCustom bundle label in that list. That will open a popup model to overview the build progress.

Once the build is done, you can click Close and reload the page to see your new ploneCustom bundle being applied for your site:

Note how the Plone logo has been replaced with a custom text and the Search button has been style after Bootstrap 3 button styles. (Also, you should now have seen an annoying alert popup from your ploneCustom JavasScript.)

To modify your ploneCustom bundle, just go to edit the file and and return to Resource Registries control panel to click the Build button again.

Now you have your ploneCustom back in Plone 5. Congratulations!

P.S. Don't forget that you can also tweak (at least the default) Plone theme a lot from the Resource Registries control panel without ploneCustom bundle simply by changing theme's LESS variables and building Plone bundle.

EXTRA: TTW ReactJS App in Plone

The new Resource Registries may feel complex to begin with, but once you get used to them, they are blessing. Just define dependencies properly, and never again you need to order Plone CSS and JavaScript resources manually, and never again (well, once add-ons get update into this new configuration) should add-ons break your site by re-registering resources into broken order.

As an example, let's implement a ReactJS Hello World for Plone TTW using the new resource registry:

At first, you need to register ReactJS library as a resource. You could upload the library into portal_resources, but for a quick experiment you can also refer to a cloud hosted version ( So, go to Resource Registries control panel and Add resource with the following details:

Note how the library is defined to be wrapped for requirejs with name react013. (Plone 5 actually ships with ReactJS library, but because the version in the first beta is just 0.10, we need to add newer version with a version specific name.)

Next, go to portal_resources/plone/custom/manage_main as before and add a new file called reactApp.js with the following ReactJS Hello World as its contents:

], function(React) {

'use strict';

var ExampleApplication = React.createClass({
  render: function() {
    var elapsed = Math.round(this.props.elapsed  / 100);
    var seconds = elapsed / 10 + (elapsed % 10 ? '' : '.0' );
    var message = 'React has been successfully running for ' + seconds + ' seconds.';
    return React.createElement("p", null, message);

var start = new Date().getTime();

setInterval(function() {
    React.createElement(ExampleApplication, {elapsed: new Date().getTime() - start}),
}, 50);

return ExampleApplication;


jQuery(function($) {

Note how ReactJS is required as react013, and how the example application is required as reactApp at the bottom (using jQuery onLoad convention).

Of course, also reactApp must be defined as a new resource at Resource Registries control panel. It should depend on previously added resource react013 being wrapped for requirejs and export itself for requirejs as reactApp:

Finally, you can Add bundle for this example reactApp:

And after Save, Build the bundle from the button below the new bundle name in Bundles list:

Note that, because the cloud hosted ReactJS library was used, the new bundle contains only the code from reactApp.js and requirejs will require ReactJS from the cloud on-demand. If you would have added the library into portal_resources, it would have been included in the resulting bundle.

After page reload, your ReactJS Hello World should be alive:

Transmogrifier, the Python migration pipeline, also for Python 3

  • 0

TL;DR; I forked collective.transmogrifier into just transmogrifier (not yet released) to make its core usable without Plone dependencies, use Chameleon for TAL-expressions, installable with just pip install and compatible with Python 3.

Transmogrifier is one of the many great developer tools by the Plone community. It's a generic pipeline tool for data manipulation, configurable with plain text INI-files, while new re-usable pipeline section blueprints can be implemented and packaged in Python. It could be used to process any number of things, but historically it's been mainly developed and used as a pluggable way to import legacy content into Plone.

A simple transmogrifier pipeline for dumping news from Slashdot to a CSV file could look like:

pipeline =

blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url =

blueprint = transmogrifier.to_csv
fieldnames =
filename = slashdot.csv

Actually, in time of writing this, I've yet to do any Plone migrations using transmogrifier. But when we recently had a reasonable size non-Plone migration task, I knew not to re-invent the wheel, but to transmogrify it. And we succeeded. Transmogrifier pipeline helped us to design the migration better, and splitting data processing into multiple pipeline sections helped us to delegate the work between multiple developers.

Unfortunately, currently collective.transmogrifier has unnecessary dependencies on CMFCore, is not installable without long known good set of versions and is missing any built-int command-line interface. At first, I tried to do all the necessary refactoring inside collective.transmogrifier, but eventually a fork was required to make the transmogrifier core usable outside Plone-environments, be compatible with Python 3 and to not break any existing workflows depending on the old transmogrifier.

So, meet the new transmogrifier:

  • can be installed with pip install (although, not yet released at PyPI)
  • new mr.migrator inspired command-line interface (see transmogrif --help for all the options)
  • new base classes for custom blueprints
    • transmogrifier.blueprints.Blueprint
    • transmogrifier.blueprints.ConditionalBlueprint
  • new ZCML-directives for registering blueprints and re-usable pipelines
    • <transmogrifier:blueprint component="" name="" />
    • <transmogrifier:pipeline id="" name="" description="" configuration="" />
  • uses Chameleon for TAL-expressions (e.g. in ConditionalBlueprint)
  • has only a few generic built-in blueprints
  • supports z3c.autoinclude for package transmogrifier
  • fully backwards compatible with blueprints for collective.transmogrifier
  • runs with Python >= 2.6, including Python 3+

There's still much work to do before a real release (e.g. documenting and testing the new CLI-script and new built-in blueprints), but let's still see how it works already...

P.S. Please, use a clean Python virtualenv for these examples.

Example pipeline

Let's start with an easy installation

$ pip install git+
$ transmogrify --help
Usage: transmogrify <pipelines_and_overrides>...
   transmogrify --list
   transmogrify --show=<pipeline>

and with example filesystem pipeline.cfg

pipeline =

blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url =

blueprint = transmogrifier.to_csv
fieldnames =
filename = slashdot.csv

and its dependencies

$ pip install feedparser

and the results

$ transmogrify pipeline.cfg
INFO:transmogrifier:CSVConstructor:to_csv wrote 25 items to /.../slashdot.csv

using, for example, Python 2.7 or Python 3.4.

Minimal migration project

Let's create an example migration project with custom blueprints using Python 3. In addition to transmogrifier, we need venusianconfiguration for easy blueprint registration and, of course, actual depedencies for our blueprints:

$ pip install git+
$ pip install git+
$ pip install fake-factory

Now we can implement custom blueprints in, for example,

from venusianconfiguration import configure

from transmogrifier.blueprints import Blueprint
from faker import Faker

class FakerContacts(Blueprint):
    def __iter__(self):
        for item in self.previous:
            yield item

        amount = int(self.options.get('amount', '0'))
        fake = Faker()

        for i in range(amount):
            yield {
                'address': fake.address()

and see them registered next to the built-in ones (or from the other packages hooking into transmogrifier autoinclude entry-point):

$ transmogrify --list --include=blueprints

Available blueprints

Now, we can make an example pipeline.cfg

pipeline =

blueprint = faker_contacts
amount = 2

blueprint = transmogrifier.to_csv

and enjoy the results

$ transmogrify pipeline.cfg to_csv:filename=- --include=blueprints
"534 Hintz Inlet Apt. 804
Schneiderchester, MI 55300",Dr. Garland Wyman
"44608 Volkman Islands
Maryleefurt, AK 42163",Mrs. Franc Price DVM
INFO:transmogrifier:CSVConstructor:to_csv saved 2 items to -

An alternative would be to just use the shipped mr.bob-template...

Migration project using the template

The new transmogrifier ships with an easy getting started template for your custom migration project. To use the template, you need a Python environment with mr.bob and the new transmogrifier:

$ pip install mr.bob readline  # readline is an implicit mr.bob dependency
$ pip install git+

Then you can create a new project directory with:

$ mrbob bobtemplates.transmogrifier:project

Once the new project directory is created, inside the directory, you can install rest of the depdendencies and activate the project with:

$ pip install -r requirements.txt
$ python develop

Now transmogrify knows your project's custom blueprints and pipelines:

$ transmogrify --list

Available blueprints

Available pipelines
    Example: Generates uppercase mock addresses

And the example pipeline can be executed with:

$ transmogrify myprojectname_example
ISSAC KOSS I,"PSC 8465, BOX 1625
APO AE 97751"
TESS FAHEY,"PSC 7387, BOX 3736
APO AP 13098-6260"
INFO:transmogrifier:CSVConstructor:to_csv wrote 2 items to -

Please, see created README.rst for how to edit the example blueprints and pipelines and create more.

Mandatory example with Plone

Using the new transmogrifier with Plone should be as simply as adding it into your buildout.cfg next to the old transmogrifier packages:

extends =
parts = instance plonesite
versions = versions

extensions = mr.developer
soures = sources
auto-checkout = *

transmogrifier = git

recipe = plone.recipe.zope2instance
eggs =
user = admin:admin
zcml =

recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

setuptools =
zc.buildout =

Let's also write a fictional migration pipeline, which would create Plone content from Slashdot RSS-feed:

pipeline =

blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url =

blueprint = transmogrifier.set
modules = uuid
id = python:str(modules['uuid'].uuid4())

blueprint = transmogrifier.set
portal_type = string:Document
text = path:item/summary
_path = string:slashdot/${item['id']}

blueprint = collective.transmogrifier.sections.folders

blueprint = collective.transmogrifier.sections.constructor

blueprint =

blueprint = transmogrifier.to_expression
modules = transaction
expression = python:modules['transaction'].commit()
mode = items

Now, the new CLI-script can be used together with bin/instance -Ositeid run provided by plone.recipe.zope2instance so that transmogrifier will get your site as its context simply by calling zope.component.hooks.getSite:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

With Plone you should, of course, still use Python 2.7.

Funnelweb example with Plone

Funnelweb is a collection of transmogrifier blueprints an pipelines for scraping any web site into Plone. I heard that its example pipelines are a little outdated, but they make a nice demo anywyay.

Let's extend our previous Plone-example with the following funnelweb.cfg buildout to include all the necessary transmogrifier blueprints and the example funnelweb.ttw pipeline:

extends = buildout.cfg

eggs +=

We also need a small additional pipeline commit.cfg to commit all the changes made by funnelweb.ttw:

pipeline = commit

blueprint = transmogrifier.interval
modules = transaction
expression = python:modules['transaction'].commit()

Now, after the buildout has been run, the following command would use pipelines funnelweb.ttw and commit.cfg to somewhat scrape my blog into Plone:

$ bin/instance -OPlone run bin/transmogrify funnelweb.ttw commit.cfg crawler:url= "crawler:ignore=feeds\ncsi.js" --context=zope.component.hooks.getSite

For tuning the import further, the used pipelines could be easily exported into filesystem, customized, and then executed similarly to commit.cfg:

$ bin/instance -OPlone run bin/transmogrify --show=funnelweb.ttw > myfunnelweb.cfg