Start here

Newt DB architecture

Newt DB is a thin layer over two mature projects. Understanding that layer makes the rest of the manual easier to read, and it explains why the JSON copy of your data is read only.

Built on ZODB and RelStorage

Newt DB builds on ZODB and RelStorage. Both are mature open source projects with years of production experience.

ZODB is an object oriented database for Python. It provides transparent object persistence. ZODB has a pluggable storage layer, and Newt DB uses RelStorage to store data in PostgreSQL.

RelStorage is the bridge between ZODB and PostgreSQL. It keeps the native ZODB records in database tables and handles the transaction and packing rules that PostgreSQL expects.

What Newt DB adds

Newt DB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL jsonb column. The JSON data supplement the native data to support indexing, search and access from applications that are not written in Python.

Because the JSON format is lossy compared with the native format, the native format is still used for loading objects from the database. For this reason the JSON data are read only. Newt DB never tries to rebuild an object from JSON, so no information is lost on the way back in.

Newt DB adds a search API for searching the PostgreSQL JSON data and returning persistent objects. It also provides a convenience API for raw data searches, which returns rows rather than objects.

Convenience without concealment

Finally, Newt DB adds convenience APIs to support its intended audience more directly. These are meant to augment ZODB and RelStorage, not hide them. Some of them are just aliases. It is possible to integrate Newt DB with existing ZODB applications without rewriting the object model.

The search helpers are documented in the reference. The conversion rules are documented in JSON serialization, and the operational tools are covered by the asynchronous updater and following changes topics.