API reference

Create a Newt newt.db.Connection. Keyword options can be used to provide either ZODB options or RelStorage options.

Create a Newt DB database object. Keyword options can be used to provide either ZODB options or RelStorage options.

A Newt DB object is a thin wrapper around a ZODB.DB object. When its open method is called, it returns newt.db.Connection objects.

Create a RelStorage storage using the Newt PostgreSQL adapter. Keyword options can be used to provide either ZODB options or RelStorage options.

Create a PostgreSQL, not Newt, database connection. This function should be used rather than, for example, calling psycopg2.connect, because it can use other PostgreSQL drivers depending on the Python environment and the available modules. The psycopg documentation covers the driver itself.

A wrapper for ZODB.Connection.Connection objects. newt.db.Connection objects provide extra helper methods for searching and for transaction management.

Query for objects satisfying criteria. This is a convenience wrapper for the search method. The first argument is SQL text for query criteria to be included in an SQL where clause.

The method appends its second argument to select * from newt where, so it may also contain code that can be included after a where clause, such as an ORDER BY clause.

Query parameters may be provided as either positional arguments or keyword arguments. They are inserted into the query where there are placeholders of the form %s for positional arguments, or %(NAME)s for keyword arguments. A sequence of Newt objects is returned.

Search for Newt objects using an SQL query. Query parameters may be provided as either positional arguments or keyword arguments. They are inserted into the query where there are placeholders of the form %s for positional arguments or %(NAME)s for keyword arguments.

The query results must contain the columns zoid and ghost_pickle. It is simplest and costs nothing to select all columns, using *, from the newt table. A sequence of Newt objects is returned.

Query for a batch of objects satisfying criteria. Like the where method, this is a convenience wrapper for the search_batch method.

Query parameters are provided using the args argument, which may be a tuple or a dictionary. They are inserted into the query where there are placeholders of the form %s for an arguments tuple or %(NAME)s for an arguments dictionary.

The batch_start and batch_size arguments specify the result batch. An ORDER BY clause should be used to order results. The total result count and the sequence of batch result objects are returned.

The query parameters, args, may be omitted. In this case batch_size will be None and the other arguments will be rearranged appropriately. batch_size is required. You might use this feature if you pre inserted data using a database cursor's mogrify method.

Query for a batch of Newt objects. Query parameters are provided using the args argument, which may be a tuple or a dictionary. They are inserted into the query where there are placeholders of the form %s for an arguments tuple or %(NAME)s for an arguments dictionary.

The batch_start and batch_size arguments specify the result batch. An ORDER BY clause should be used to order results. The total result count and the sequence of batch result objects are returned.

The query parameters, args, may be omitted. In this case batch_size will be None and the other arguments will be rearranged appropriately. batch_size is required. You might use this feature if you pre inserted data using a database cursor's mogrify method.

Query the Newt PostgreSQL database for raw data. Query parameters may be provided as either positional arguments or keyword arguments. They are inserted into the query where there are placeholders of the form %s for positional arguments, or %(NAME)s for keyword arguments. A sequence of data tuples is returned.

Compute and return SQL to set up a Newt text index. The resulting SQL contains a statement to create a PL/pgSQL function and an index creation statement that uses it.

The first argument is the name of the function to be generated. The second argument is a single expression or property name, or a sequence of expressions or property names. If expressions are given, they are evaluated against the Newt JSON state column. Values consisting of alphanumeric characters, including underscores, are treated as names; other values are treated as expressions.

Additional arguments C, B and A can supply expressions or names for text to be extracted with different weights for ranking. See the PostgreSQL text search ranking documentation.

The config argument may specify which text search configuration to use. If it is not specified, the server configured default is used.

Set up a Newt full text index. The create_text_index_sql method is used to compute SQL, which is then executed to set up the index. This can take a long time on an existing database with many records.

The SQL is executed against the database associated with the given connection, but a separate connection is used, so its execution is independent of the current transaction.

Get a database cursor for reading. The returned cursor can be used to make PostgreSQL queries and to perform safe SQL generation using the cursor's mogrify method. The caller must close the returned cursor after use.

Create a data update iterator. The iterator returns an iterator of batches, where each batch is an iterator of records. Each record is a triple consisting of an integer transaction id, an integer object id and data.

A sample use:

import newt.db
import newt.db.follow
connection = newt.db.pg_connection('')
for batch in newt.db.follow.updates(connection):
    for tid, zoid, data in batch:
        print(tid, zoid, len(data))

If no end_tid is provided, the iterator will iterate until interrupted.

conn

A PostgreSQL database connection.

start_tid

Start tid, expressed as an integer. The iterator starts at the first transaction after this tid.

end_tid

End tid, expressed as an integer. The iterator stops at this point, or at the end of data, whichever is less. If the end tid is None, the iterator will run indefinitely, returning new data as they are committed.

batch_limit

A soft batch size limit. When a batch reaches this limit, it will end at the next transaction boundary. The purpose of this limit is to limit read transaction size.

internal_batch_size

The size of the internal PostgreSQL iterator. Data are not loaded from PostgreSQL all at once. Server side cursors are used, and data are loaded from the server in internal_batch_size batches.

poll_timeout

When no end_tid is specified, this specifies how often to poll for changes. A trigger is created and used to notify the iterator of changes, so changes are detected quickly. The poll timeout is just a backstop.

Get the current progress for a follow client. Return the last saved integer transaction id for the client, or -1 if one has not been saved before.

A follow client often updates some other data based on the data returned from updates. It may stop and restart later. To do this, it calls set_progress_tid to save its progress and later calls get_progress_tid to find where it left off. It can then pass the returned tid as start_tid to updates.

The connection argument must be a PostgreSQL connection string or connection. The id parameter identifies which progress is wanted. It should uniquely identify the client, and generally the dotted name of the client module is used. This allows multiple clients to have their progress tracked.

Set the current progress for a follow client. See get_progress_tid.

The connection argument must be a PostgreSQL connection string or connection. The id argument is a string identifying a client. It should generally be a dotted name, usually the client module name, and it must uniquely identify the client. The tid argument is the most recently processed transaction id as an integer.

Listen for Newt database updates. Returns an iterator that returns integer transaction ids or None values.

The purpose of this method is to determine whether there are updates. If transactions are committed very quickly, then not all of them will be returned by the iterator. None values indicate that poll_timeout seconds have passed since the last update.

dsn

A PostgreSQL connection string.

timeout_on_start

Force None to be returned immediately after listening for notifications. This is useful in some special cases to avoid having to wait for changes that happened before the iterator began listening.

poll_timeout

A timeout after which None is returned if there are no changes. This is a backstop to the PostgreSQL notification system.

__call__(id, data)

Convert data from a ZODB data record to data used by Newt DB. The data returned is a class name, ghost pickle and state triple. The state is a JSON formatted string. The ghost pickle is a binary string that can be used to create a ZODB ghost object.

If there is an error converting data, if the data is empty, or if the skip_class function returns a true value, then (None, None, None) is returned.

id

A data identifier, such as an object id, used when logging errors.

data

Pickle data to be converted.

__init__(skip_class=None, transform=None)

Create a callable for converting database data to Newt JSON.

skip_class

A callable that will be called with the class name extracted from the data. If the callable returns a true value, then data will not be converted to JSON and (None, None, None) is returned. The default, available as the skip_class attribute of the Jsonifier class, skips objects from the BTrees package and blobs.

transform

A function that transforms a record's state JSON. If provided, it should accept a class name and a state string in JSON format. It should return a new state string or None. If None is returned, the original state is used. If the function returns an empty string, then the Jsonifier returns (None, None, None). In other words, a transform that returns an empty string is equivalent to a skip_class function that returns True. Returning anything other than None or a string is an error and behaviour is undefined.

An unpickler that returns JSON.

Usage:

apickle = pickle.dumps([1,2])
unpickler = JsonUnpickler(apickle)
json_string = unpickler.load()
unpickler.pos == len(apickle)
True