Topics

JSON serialization

This topic explains some details about how data are serialized to JSON, beyond the behaviour of the standard Python json module.

JSON is a lossy view

The first thing to note is that JSON serialization is lossy. This is why Newt DB saves data in both pickle and JSON format. The serialization nevertheless preserves class information, and it supports cyclic data structures using a combination of persistent references and references within a record.

Plain Python instances

Instances that are not persistent are converted to JSON objects with a :: property giving their dotted class names. In the common case of objects with their instance dictionaries used as their pickled state, the object attributes become properties.

For example, given a class MyClass in module mymodule:

class MyClass:

    def __init__(self, a, b):
        self.a = a
        self.b = b

The JSON serialization would look like:

{"::": "mymodule.MyClass", "a": 1, "b": 2}

State that is not a dictionary

For instances with pickled state that is not a dictionary, a JSON object is created with a state property containing the serialized state and a :: property with the dotted class name.

Arguments to __new__

Objects that take arguments to their __new__ method will have the arguments serialized in the ::() property.

Reference ids within an object

If a record has cycles and an object in the record is referenced more than once, then the object will have an ::id property whose value is an internal reference id.

For objects like lists and sets, which are not normally serialized as objects, when an object is referenced more than once it is wrapped in a “shared” object with an ::id property and a value property.

Cycles within a record

Cyclic data structures are allowed within persistent object records, although they are extremely rare. When there is a cycle, objects that are referenced more than once:

• have ::id properties that assign them references within the record;
• if they are lists or similar, have their state wrapped in a “shared” object;
• are replaced with reference objects in all but one of the references. Reference objects have a single property giving the internal id of the object being referenced.

Here is an example:

from newt.db.tests.testjsonpickle import I
i = I(a=1)
d = dict(b=1)
l = [i, i, d, d]
l.append(l)

The serialization of the list l would be equivalent to:

{
  "::": "shared",
  "::id": 0,
  "value": [
    {
      "::": "newt.db.tests.testjsonpickle.I",
      "::id": 2,
      "a": 1
    },
    {"::->": 2},
    {
      "::id": 5,
      "b": 1
    },
    {"::->": 5},
    {"::->": 0}
  ]
}

References within a record like these are difficult to work with, which is a good reason to avoid cycles within a record.

Persistent objects

Persistent objects are stored in four columns of the newt table:

The class name and state are separated, and the state does not have a :: property containing the dotted class name. The ghost_pickle field contains the class name and __new__ arguments if necessary. It is used to create new objects when searching.

Persistent references

When one persistent object references another persistent object, the reference is serialized with a reference object having a property whose value is the object id of the referenced object. For example, the serialization of a subtask object containing a reference to a parent task would be equivalent to:

{
  "title": "Do something",
  "parent": {"::=>": 42}
}

Cycles among persistent objects are common and do not present any problems for serialization, because persistent objects are serialized separately.

This differs from versions of Newt DB before 0.4.0. Earlier versions represented persistent references as objects with a :: property with the value persistent and an id property whose value is an integer object id or a list containing an integer object id and a dotted class name. Those attributes will be retained until Newt DB version 1, at which point they will no longer be included.

Dates and times

datetime.date objects and datetime.datetime instances without time zones are converted to strings using their isoformat methods.

datetime.datetime instances with time zones are serialized as objects with a :: property of datetime, a value property with their ISO formatted value, and a tz property containing a JSON serialization of their time zones.