Topics

Data transformation

You can have more control over how JSON is generated from your data by supplying a transform function. It runs on each state after conversion and before the record is written to the newt table.

The transform contract

A transform function accepts two positional arguments:

The transform function must return a new state string or None. If None is returned, the original state is used. A transform function may return an empty string to indicate that a record should be skipped and not written to the newt table.

Flattening a persistent mapping

For example, persistent.mapping.PersistentMapping objects store their data in a data attribute, so their JSON representation is more complex than we might want. Here is a transform that replaces the JSON representation of a PersistentMapping with its data:

import json

def flatten_persistent_mapping(class_name, state):
    if class_name == 'persistent.mapping.PersistentMapping':
       state = json.loads(state)
       state = state['data']
       return json.dumps(state)

Supply it from Python

We can supply a transform function to the Python constructor using the transform keyword argument:

import newt.db

conn = newt.db.connection('', transform=flatten_persistent_mapping)

Supply it from text configuration

To specify a transform in text configuration, use a transform option to supply the dotted name of your transform function in the newt configuration element:

%import newt.db

<newtdb>
  <zodb>
    <relstorage>
      keep-history false
      <newt>
        transform myproject.flatten_persistent_mapping
        <postgresql>
          dsn dbname=''
        </postgresql>
      </newt>
    </relstorage>
  </zodb>
</newtdb>