Storage¶

Backends¶

PostgreSQL¶

class kinto.core.storage.postgresql.Storage(client, max_fetch_size, *args, readonly=False, **kwargs)¶

Storage backend using PostgreSQL.

Recommended in production (requires PostgreSQL 9.4 or higher).

Enable in configuration:

kinto.storage_backend = kinto.core.storage.postgresql

Database location URI can be customized:

kinto.storage_url = postgresql://user:pass@db.server.lan:5432/dbname

Alternatively, username and password could also rely on system user ident or even specified in ~/.pgpass (see PostgreSQL documentation).

Note

Some tables and indices are created when kinto migrate is run. This requires some privileges on the database, or some error will be raised.

Alternatively, the schema can be initialized outside the python application, using the SQL file located in kinto/core/storage/postgresql/schema.sql. This allows to distinguish schema manipulation privileges from schema usage.

A connection pool is enabled by default:

kinto.storage_pool_size = 10
kinto.storage_maxoverflow = 10
kinto.storage_max_backlog = -1
kinto.storage_pool_recycle = -1
kinto.storage_pool_timeout = 30
kinto.cache_poolclass =
    kinto.core.storage.postgresql.pool.QueuePoolWithMaxBacklog

The max_backlog limits the number of threads that can be in the queue waiting for a connection. Once this limit has been reached, any further attempts to acquire a connection will be rejected immediately, instead of locking up all threads by keeping them waiting in the queue.

See dedicated section in SQLAlchemy documentation for default values and behaviour.

Note

Using a dedicated connection pool is still recommended to allow load balancing, replication or limit the number of connections used in a multi-process deployment.

Redis¶

See Kinto Redis driver plugin repository for more information.

Memory¶

class kinto.core.storage.memory.Storage(*args, readonly=False, **kwargs)¶

Storage backend implementation in memory.

Useful for development or testing purposes, but stored data is lost after each server restart.

Enable in configuration:

kinto.storage_backend = kinto.core.storage.memory

API¶

Implementing a custom storage backend consists in implementating the following interface:

class kinto.core.storage.Missing¶

Dummy value to represent a value that is completely absent from an object.

Handling these correctly is important for pagination.

class kinto.core.storage.Filter(field, value, operator)¶

Filtering properties.

field¶: Alias for field number 0

operator¶: Alias for field number 2

value¶: Alias for field number 1

class kinto.core.storage.Sort(field, direction)¶

Sorting properties.

direction¶: Alias for field number 1

field¶: Alias for field number 0

class kinto.core.storage.StorageBase¶

Storage abstraction used by resource views.

It is meant to be instantiated at application startup. Any operation may raise a HTTPServiceUnavailable error if an error occurs with the underlying service.

Configuration can be changed to choose which storage backend will persist the objects.

Raises:: HTTPServiceUnavailable

id_generator = <kinto.core.storage.generators.UUID4 object>¶: Id generator used when no one is provided for create.

initialize_schema(dry_run=False)¶

Create every necessary objects (like tables or indices) in the backend.

This is executed when the kinto migrate command is run.

Parameters:: dry_run (bool) – simulate instead of executing the operations.

flush()¶: Remove every object from this storage.

resource_timestamp(resource_name, parent_id)¶

Get the highest timestamp of every objects in this resource_name for this parent_id.

Note

This should take deleted objects into account.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.

Returns:

the latest timestamp of the resource.

Return type:

int

all_resources_timestamps(resource_name)¶

Get the highest timestamp of every objects in this resource_name for each parent_id.

Note

This should take deleted objects into account.

Parameters:: resource_name (str) – the resource name.
Returns:: the latest timestamp of the resource by parent_id.
Return type:: dict[str, int]

create(resource_name, parent_id, obj, id_generator=None, id_field='id', modified_field='last_modified')¶

Create the specified obj in this resource_name for this parent_id. Assign the id to the object, using the attribute kinto.core.resource.model.Model.id_field.

Note

This will update the resource timestamp.

Raises:

kinto.core.storage.exceptions.UnicityError

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
obj (dict) – the object to create.

Returns:

the newly created object.

Return type:

dict

get(resource_name, parent_id, object_id, id_field='id', modified_field='last_modified')¶

Retrieve the object with specified object_id, or raise error if not found.

Raises:

kinto.core.storage.exceptions.ObjectNotFoundError

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
object_id (str) – unique identifier of the object

Returns:

the stored object.

Return type:

dict

update(resource_name, parent_id, object_id, obj, id_field='id', modified_field='last_modified')¶

Overwrite the obj with the specified object_id.

If the specified id is not found, the object is created with the specified id.

Note

This will update the resource timestamp.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
object_id (str) – unique identifier of the object
obj (dict) – the object to update or create.

Returns:

the updated object.

Return type:

dict

delete(resource_name, parent_id, object_id, id_field='id', with_deleted=True, modified_field='last_modified', deleted_field='deleted', last_modified=None)¶

Delete the object with specified object_id, and raise error if not found.

Deleted objects must be removed from the database, but their ids and timestamps of deletion must be tracked for synchronization purposes. (See kinto.core.storage.StorageBase.get_all())

Note

This will update the resource timestamp.

Raises:

kinto.core.storage.exceptions.ObjectNotFoundError

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
object_id (str) – unique identifier of the object
with_deleted (bool) – track deleted object with a tombstone

Returns:

the deleted object, with minimal set of attributes.

Return type:

dict

delete_all(resource_name, parent_id, filters=None, sorting=None, pagination_rules=None, limit=None, id_field='id', with_deleted=True, modified_field='last_modified', deleted_field='deleted')¶

Delete all objects in this resource_name for this parent_id.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
filters (list of kinto.core.storage.Filter) – Optionnally filter the objects to delete.
sorting (list of kinto.core.storage.Sort) – Optionnally sort the objects by attribute. Each sort instruction in this list refers to a field and a direction (negative means descending). All sort instructions are cumulative.
pagination_rules (list of list of kinto.core.storage.Filter) – Optionnally paginate the deletion of objects. This list of rules aims to reduce the set of objects to the current page. A rule is a list of filters (see filters parameter), and all rules are combined using OR.
limit (int) – Optionnally limit the number of objects to be deleted.
with_deleted (bool) – track deleted objects with a tombstone

Returns:

the list of deleted objects, with minimal set of attributes.

Return type:

list

purge_deleted(resource_name, parent_id, before=None, max_retained=None, id_field='id', modified_field='last_modified')¶

Delete all deleted object tombstones in this resource_name for this parent_id.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
before (int) – Optional timestamp to limit deletion (exclusive).
max_count (int) – Optional maximum of tombstones to keep per collection.

Returns:

The number of deleted objects.

Return type:

int

get_all(*args, **kwargs)¶

Legacy method to support code that relied on the old API where the storage’s get_all() would return a tuple of (<list of objects paginated>, <count of all>). Since then, we’re being more explicit and expecting the client to deliberately decide if they need a paginated list or a count.

This method exists solely to make the transition easier.

list_all(resource_name, parent_id, filters=None, sorting=None, pagination_rules=None, limit=None, include_deleted=False, id_field='id', modified_field='last_modified', deleted_field='deleted')¶

Retrieve all objects in this resource_name for this parent_id.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent, possibly containing a wildcard ‘*’. (This can happen when implementing “administrator” operations on a Resource, for example, like kinto.plugins.accounts.)
filters (list of kinto.core.storage.Filter) – Optionally filter the objects by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see kinto.core.utils.COMPARISON). All filters are combined using AND.
sorting (list of kinto.core.storage.Sort) – Optionnally sort the objects by attribute. Each sort instruction in this list refers to a field and a direction (negative means descending). All sort instructions are cumulative.
pagination_rules (list of list of kinto.core.storage.Filter) – Optionnally paginate the list of objects. This list of rules aims to reduce the set of objects to the current page. A rule is a list of filters (see filters parameter), and all rules are combined using OR.
limit (int) – Optionnally limit the number of objects to be retrieved.
include_deleted (bool) – Optionnally include the deleted objects that match the filters.

Returns:

the limited list of objects of matching objects in the resource (deleted ones excluded).

Return type:

list

count_all(resource_name, parent_id, filters=None, id_field='id', modified_field='last_modified', deleted_field='deleted')¶

Return a count of all objects in this resource_name for this parent_id.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the parent resource, possibly containing a wildcard ‘*’. (This can happen when implementing “administrator” operations on a UserResource, for example.)
filters (list of kinto.core.storage.Filter) – Optionally filter the objects by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see kinto.core.utils.COMPARISON). All filters are combined using AND.

Returns:

the total number of matching objects in the resource (deleted ones excluded).

Return type:

int

trim_objects(resource_name: str, parent_id: str, filters: list, max_objects: int, id_field: str = 'id', modified_field: str = 'last_modified') → int¶

Trim the last N objects in the specified resource matching the filters.

Parameters:

resource_name (str) – the resource name.
parent_id (str) – the resource parent.
filters (list) – list of kinto.core.storage.Filter to filter objects.
max_objects (int) – maximum number of objects to keep.
id_field (str) – the id field name.
modified_field (str) – the modified field name.

Returns:

the number of deleted objects.

Return type:

int

Exceptions¶

Exceptions raised by storage backend.

exception kinto.core.storage.exceptions.BackendError(original=None, message=None, *args, **kwargs)¶

A generic exception raised by storage on error.

Parameters:: original (Exception) – the wrapped exception raised by underlying library.

exception kinto.core.storage.exceptions.ReadonlyError(original=None, message=None, *args, **kwargs)¶: An error raised when a write operation is attempted on a read-only instance.

exception kinto.core.storage.exceptions.RecordNotFoundError¶: Deprecated exception name.

exception kinto.core.storage.exceptions.ObjectNotFoundError¶: An exception raised when a specific object could not be found.

exception kinto.core.storage.exceptions.IntegrityError(original=None, message=None, *args, **kwargs)¶

exception kinto.core.storage.exceptions.UnicityError(field, *args, **kwargs)¶

An exception raised on unicity constraint violation.

Raised by storage backend when the creation or the modification of a object violates the unicity constraints defined by the resource.

Store custom data¶

Storage can be used to store arbitrary data.

data = {'subscribed': datetime.now()}
user_id = request.authenticated_userid

storage = request.registry.storage
storage.create(resource_name='__custom', parent_id='', obj=data)

See the Model class to manipulate collections of records.

Storage¶

Backends¶

PostgreSQL¶

Redis¶

Memory¶

API¶

Exceptions¶

Store custom data¶

Kinto

Navigation

Related Topics