Topics

Asynchronous JSON conversion

Normally Newt DB converts data to JSON as it is saved, and any indexes on that JSON are updated at the same time. For write heavy workloads you can move that work into a separate daemon.

Why run it asynchronously

With the default JSON index, informal tests show Newt DB write performance to be around 10 percent slower than RelStorage. Adding a text index brought performance down to the point where writes took twice as long, but they were still fairly fast: several hundred per second on a laptop.

If you have a lot of indexes to update, or write performance is critical, you may want to use the ability of Newt DB to update the JSON data asynchronously. Doing so allows the primary transactions to execute more quickly.

Updating indexes asynchronously will usually be more efficient, because the asynchronous updater batches updates. When indexes are updated for many objects in the same transaction, less data has to be written per transaction.

If you want to try Newt DB with an existing RelStorage and PostgreSQL database, you can use the updater to populate Newt DB without changing your application, and introduce the use of the Newt DB search API gradually.

Caveats

Because updates are asynchronous, search results may not always reflect the current data.
Packing requires some special care, as discussed below.
You need to run a separate daemon, newt-updater, in addition to your database server.

Using the asynchronous updater

To use the asynchronous updater, omit the newt tag from your database configuration:

%import newt.db

<newtdb foo>
  <zodb>
    <relstorage>
      keep-history false
      <postgresql>
        dsn postgresql://localhost/mydb
      </postgresql>
    </relstorage>
  </zodb>
</newtdb>

Then run the newt-updater script:

newt-updater postgresql://localhost/mydb

Run this with a daemonizer such as supervisord or ZDaemon. The updater uses PostgreSQL notifications as its signal, with a polling timeout as a backstop. The PostgreSQL NOTIFY documentation explains the underlying mechanism.

Options

Command options
Option	What it does
`-l, --logging-configuration`	Logging configuration. This can be a log level such as `INFO`, the default, or the path to a ZConfig logging configuration file.
`-g, --gc-only`	Collect garbage and exit. This removes Newt DB records that do not have corresponding database records.
`-G, --no-gc`	Do not perform garbage collection on startup. Garbage collection is normally performed on startup unless this option is used.
`--nagios`	Check the status of the updater by checking its lag. The option takes two numbers separated by a comma: the lag for OK, and the maximum lag before the updater is in error.
`-t, --poll-timeout`	Specify a poll timeout in seconds. Normally the updater is notified of changes; if it is not notified within this window it polls anyway. The default is 300 seconds.
`-m, --transaction-size-limit`	The target transaction batch size. It loosely limits the number of records processed in a batch. Larger batches incur less overhead, but long lasting transactions can interfere with other processing. The default is 100 thousand records.
`-T, --remove-delete-trigger`	Remove the Newt DB delete trigger if it exists. The trigger is incompatible with the updater and can cause deadlock errors if packing runs while the updater is active.
`-d, --driver`	Provide an explicit PostgreSQL driver name, either `psycopg2` or `psycopg2cffi`. By default the appropriate driver is selected automatically.
`--compute-missing`	Compute missing newt records. Rather than processing new records, process records written up to the current time and stop. Only missing records are updated. This requires PostgreSQL 9.5 or later.

The --compute-missing option is used to compute newt records after adding Newt DB to an existing PostgreSQL RelStorage application. The ZODB users topic puts that in context.

Garbage collection

The asynchronous updater tracks new database inserts and updates. When a database is packed, records are removed without generating updates. Those deletes will not be reflected in Newt DB. You can tell the updater to clean up Newt DB records for which there are no longer database records by either restarting it, or running it with the -g option:

newt-updater -g postgresql://localhost/mydb

This tells the updater to just collect garbage. Run it right after running zodbpack. The removal query deletes rows from newt where no matching object_state row exists.

Monitoring

When running an external updater such as newt-updater, you want some way to monitor that it is working correctly. The --nagios option provides a Nagios compatible plugin:

newt-updater postgresql://localhost/mydb --nagios 3,99

The argument to --nagios is a pair of numbers giving limits for OK and warning alerts. They are based on how far behind the updater is. With the example above, the monitor considers the updater to be OK if it is 3 seconds behind or less, in error if it is more than 99 seconds behind, and of concern otherwise.

Any monitoring system compatible with the Nagios plugin API can be used. The monitor output includes the lag, which is how far behind the updater is in seconds, as a performance metric.