Loading...

Write testable documentation with Robot Framework

If you have background as a Python programmer, you must be familiar with doctests -- testable code examples embedded into documentation. When Timo Stollenwerk presented Robot Framework in his TDD-presentation in Plone Conference 2012, the first question from the audience was, can Robot Framework test be in embedded into documentation, similarly to Python doctests.

Soon you can.

I've been mentoring Vivek Kumar Verma in this years Google Summer of Code to implement a better ReStructureText parser into Robot Framework. The first part of his GSOC project was to actually enhance the parser, and the second part is implementing a Sphinx-plugin to execute embedded Robot Framework tests while compiling the documentation. You can follow Vivek's dive into Python world in his blog and find him as viku__ on Freenode-channels #plone and #robotframework.

Issues related to Vivek's GSOC 2013 work for the enhanced ReST-parser can be filed at https://github.com/datakurre/robotframework/issues until the GSOC ends.

How does it work?

As you may have guessed from the title. The first part of Vivek's GSOC is pretty much completed and is waiting for a review from the Robot Framework team. It's a minimal, but very powerful enhancement for the old Robot Framework's ReST parser. While the old parser supported only so called table syntax, the new parser allows you to use the famous plain text format when embedding your Robot Framework tests into ReST-documents.

Try to find the test cases from the following ReStructuredText-example:

Your first Robot Framework doctest
==================================

With the Robot Framework plain text syntax, a minimal test suite
would consists of ``*** Test Cases ***`` header and at least
one test case, like:

.. code:: robotframework

   *** Test Cases ***

   Foo is always Foo
       Should be equal  Foo  Foo

One ``*** Test Cases ***``-header may be followed by as many
tests as required, like:

.. code:: robotframework

   *** Test Cases ***

   Foo is still Foo
       Should be equal  Foo  Foo

   Foo is never Bar
       Should not be equal  Foo  Bar

With Vivek, we decided to re-use the existing code-directive of Docutils for embedding plain text Robot Framework tests suites into ReStructuredText-documents. Each document can contain as many code-directives with robotframework-language as required and Robot Framework will concatenate their contents into a single Robot Framework test suite.

There is a small price, of course. Docutils will tag code-directives with robotframework-language only when also Pygments-package is installed. Luckily you will want to have it, because since with Pygments (>= 1.6) you will also get syntax highlighting your embedded Robot Framework tests.

Try it out

Even Vivek's GSOC-work has not been merged into Robot Framework yet, you can try it out with the following steps:

  1. Create and activate a new Python virtual environment.
  1. Install the dependencies:

    $ pip install docutils Pygments
    
  2. Install GSOC-patched version of Robot Framework 2.8.1:

    $ pip install https://github.com/datakurre/robotframework/tarball/GSOC2013DEMO/robotframework-2.8.1-GSOC2013.0.tar.gz
    
  3. Create an example test suite example.rst:

    Your first Robot Framework doctest
    ==================================
    
    With the Robot Framework plain text syntax, a minimal test suite
    would consists of ``*** Test Cases ***`` header and at least
    one test case, like:
    
    .. code:: robotframework
    
       *** Test Cases ***
    
       Foo is always Foo
           Should be equal  Foo  Foo
    
    One ``*** Test Cases ***``-header may be followed by as many
    tests as required, like:
    
    .. code:: robotframework
    
       *** Test Cases ***
    
       Foo is still Foo
           Should be equal  Foo  Foo
    
       Foo is never Bar
           Should not be equal  Foo  Bar
    
  4. Run the suite:

    $ pybot example.rst
    
    ========================================================================
    Example
    ========================================================================
    Foo is always Foo                                               | PASS |
    ------------------------------------------------------------------------
    Foo is still Foo                                                | PASS |
    ------------------------------------------------------------------------
    Foo is never Bar                                                | PASS |
    ------------------------------------------------------------------------
    Example                                                         | PASS |
    3 critical tests, 3 passed, 0 failed
    3 tests total, 3 passed, 0 failed
    ========================================================================
    Output:  /.../output.xml
    Log:     /.../log.html
    Report:  /.../report.html
    
  5. And create a nicely highlighted documentation:

    $ pygmentize -S default -f html > colors.css
    $ rst2html.py --syntax-highlight=short --stylesheet-path=colors.css example.rst > example.html
    

A mandatory Plone example

Yes, you can also embed a Plone example into a short ReST-document.

  1. As always, we need a bootstrap.py:

    $ curl -O http://downloads.buildout.org/2/bootstrap.py
    
  2. And buildout.cfg:

    [buildout]
    extends = http://dist.plone.org/release/4.3-latest/versions.cfg
    find-links =
        https://github.com/datakurre/robotframework/tarball/GSOC2013DEMO/robotframework-2.8.1-GSOC2013.0.tar.gz
    versions = versions
    parts = robot
    
    [versions]
    robotframework = 2.8.1-GSOC2013.0
    
    [robot]
    recipe = zc.recipe.egg
    eggs =
        docutils
        Pygments
        plone.app.robotframework
    scripts = pybot
    
  3. Run the buildout:

    $ python bootstrap.py
    $ bin/buildout
    
  4. Add the following plone.rst:

    Adding a new page in Plone
    ==========================
    
    .. raw:: html
    
       <!-- This class allows us to hide the test setup part -->
       <style type="text/css">.hidden { display: none; }</style>
    
    .. code:: robotframework
       :class: hidden
    
       *** Settings ***
    
       Resource  plone/app/robotframework/server.robot
    
       Suite Setup  Setup
       Suite Teardown  Teardown
       Test Setup  Test Setup
       Test Teardown  Test Teardown
    
        *** Keywords ***
    
        Setup
            Setup Plone site  plone.app.robotframework.testing.AUTOLOGIN_ROBOT_TESTING
    
        Test Setup
            Import library  Remote  ${PLONE_URL}/RobotRemote
            Enable autologin as  Site Administrator
            Set autologin username  test-user-1
    
        Test Teardown
            Set Zope layer  plone.app.robotframework.testing.AUTOLOGIN_ROBOT_TESTING
            ZODB TearDown
            ZODB SetUp
    
        Teardown
            Teardown Plone Site
    
    This is how you can test adding new document in Plone, once your test user
    has logged in with enough permissions.
    
    .. code:: robotframework
    
        *** Test Cases ***
    
        Add new page
            Go to  ${PLONE_URL}
    
            Click link  css=#plone-contentmenu-factories dt a
            Element should be visible
            ...    css=#plone-contentmenu-factories dd.actionMenuContent
    
            Click link  css=a#document
            Wait Until Page Contains Element  css=#archetypes-fieldname-title input
    
            Input Text  title  This is the title
            Input Text  description  This is the summary.
    
            Click button  Save
            Element should contain  css=#parent-fieldname-title  This is the title
    
            Capture page screenshot  new-page.png
    
    And if the test is successful, you will see the resulting image below:
    
    .. image:: new-page.png
    
  5. Run the suite:

    $ bin/pybot plone.rst
    ========================================================================
    Plone
    ========================================================================
    Add new page                                                    | PASS |
    ------------------------------------------------------------------------
    Plone                                                           | PASS |
    1 critical test, 1 passed, 0 failed
    1 test total, 1 passed, 0 failed
    ========================================================================
    Output:  /.../output.xml
    Log:     /.../log.html
    Report:  /.../report.html
    
  6. And (if you have Pygments and Docutils installed into your current path) create a nicely highlighted documentation:

    $ pygmentize -S default -f html > colors.css
    $ rst2html.py --syntax-highlight=short --stylesheet-path=colors.css plone.rst > plone.html
    

I'm not sure if it will be possible (or even reasonable) to write actual tests in such a descriptive language that they can be included into final documentation. Yet, embedding screenshot-resulting tests into documentation would make sense, because those would keep documentation screenshots up to date and would also work as acceptance tests for the documented features.

That's why the rest of Vivek's GSOC will be about running the tests within Sphinx: the same compilation run could both execute tests to create screenshots and include those screenshots into the final documentation.