.. _tutorial-write-plugin: How to write a Kinto plugin? ============================ *Kinto* plugins allow to add extra-features to *Kinto*. Most notably: * Respond to internal events (e.g. notify third-party) * Add endpoints for custom URLs (e.g. new hook URL) * Add custom endpoint renderers (e.g. XML instead of JSON) *Kinto* plugins are :ref:`Python modules loaded on startup `. In this tutorial, we will build a plugin for `ElasticSearch `_, a full-text search engine. The plugin will: * Initialize an indexer on startup; * Index the records when they're created, updated, or deleted. * Add a new ``/{collection}/search`` endpoint; Plugins are built using the Pyramid ecosystem. Run ElasticSearch ----------------- We will run a local install of *ElasticSearch* on ``localhost:9200``. Using Docker it is pretty straightforward: :: sudo docker run -p 9200:9200 elasticsearch It is also be installed manually using the `official instructions `_. Include me ---------- First, create a Python package and install it locally. For example: :: $ pip install cookiecutter $ cookiecutter gh:kragniz/cookiecutter-pypackage-minimal [...] $ cd kinto_elasticsearch $ python setup.py develop In order to be included, a package must define an ``includeme(config)`` function. For example, in :file:`kinto_elasticsearch/__init__.py`: .. code-block:: python def includeme(config): print("I am the ElasticSearch plugin!") Add it the :file:`config.ini` file: .. code-block:: ini kinto.includes = kinto_elasticsearch Our message should now appear on ``kinto start``. Simple indexer -------------- Let's define a simple indexer class in :file:`kinto_elasticsearch/indexer.py`. It can search and index records, using the official Python package: :: $ pip install elasticsearch It is a wrapper basically, and the code is kept simple for the simplicity of this tutorial: .. code-block:: python import elasticsearch class Indexer: def __init__(self, hosts): self.client = elasticsearch.Elasticsearch(hosts) def search(self, bucket_id, collection_id, query, **kwargs): indexname = '%s-%s' % (bucket_id, collection_id) return self.client.search(index=indexname, doc_type=indexname, body=query, **kwargs) def index_record(self, bucket_id, collection_id, record, id_field='id'): indexname = '%s-%s' % (bucket_id, collection_id) if not self.client.indices.exists(index=indexname): self.client.indices.create(index=indexname) record_id = record[id_field] index = self.client.index(index=indexname, doc_type=indexname, id=record_id, body=record, refresh=True) return index def unindex_record(self, bucket_id, collection_id, record, id_field='id'): indexname = '%s-%s' % (bucket_id, collection_id) record_id = record[id_field] result = self.client.delete(index=indexname, doc_type=indexname, id=record_id, refresh=True) return result And a simple method to load from configuration: .. code-block:: python from pyramid.settings import aslist def load_from_config(config): settings = config.get_settings() hosts = aslist(settings.get('elasticsearch.hosts', 'localhost:9200')) indexer = Indexer(hosts=hosts) return indexer Initialize on startup --------------------- We now need to initialize the indexer when Kinto starts. It happens in the ``includeme()`` function. .. code-block:: python :emphasize-lines: 5 from . import indexer def includeme(config): # Register a global indexer object config.registry.indexer = indexer.load_from_config(config) Add a search view ----------------- Add an endpoint definition in :file:`kinto_elasticsearch/views.py`: .. code-block:: python import logging from kinto.core import Service logger = logging.getLogger(__name__) search = Service(name="search", path='/buckets/{bucket_id}/collections/{collection_id}/search', description="Search") @search.post() def get_search(request): bucket_id = request.matchdict['bucket_id'] collection_id = request.matchdict['collection_id'] query = request.body # Access indexer from views using registry. indexer = request.registry.indexer try: results = indexer.search(bucket_id, collection_id, query) except Exception as e: logger.exception(e) results = {} return results Enable the view: .. code-block:: python :emphasize-lines: 7,8 from . import indexer def includeme(config): # Register a global indexer object config.registry.indexer = indexer.load_from_config(config) # Activate end-points. config.scan('kinto_elasticsearch.views') This new URL should now be accessible, but return no result: :: $ http POST "http://localhost:8888/v1/buckets/example/collections/notes/search .. code-block:: http HTTP/1.1 200 OK Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff Content-Length: 2 Content-Type: application/json; charset=UTF-8 Date: Wed, 20 Jan 2016 12:01:50 GMT Server: waitress {} Index records on change ----------------------- When records change, we index them. When they are deleted, we unindex them. Let's define a function ``on_resource_changed()`` that will be called when an action is performed on records. .. code-block:: python :emphasize-lines: 2,19-21 def on_resource_changed(event): indexer = event.request.registry.indexer resource_name = event.payload['resource_name'] if resource_name != "record": return bucket_id = event.payload['bucket_id'] collection_id = event.payload['collection_id'] action = event.payload['action'] for change in event.impacted_records: if action == 'delete': indexer.unindex_record(bucket_id, collection_id, record=change['old']) else: indexer.index_record(bucket_id, collection_id, record=change['new']) And then we bind this function with the *Kinto-Core* events: .. code-block:: python :emphasize-lines: 1,12,13 from kinto.core.events import ResourceChanged from . import indexer def includeme(config): # Register a global indexer object config.registry.indexer = indexer.load_from_config(config) # Activate end-points. config.scan('kinto_elasticsearch.views') # Plug the callback with resource events. config.add_subscriber(on_resource_changed, ResourceChanged) Declare API capabilities ------------------------ Arbitrary capabilities can be declared and exposed in the :ref:`root URL `. Clients can rely on this to detect optional features on the server, like our indexer! .. code-block:: python :emphasize-lines: 14-17 from kinto.core.events import ResourceChanged from . import indexer def includeme(config): # Register a global indexer object config.registry.indexer = indexer.load_from_config(config) # Activate end-points. config.scan('kinto_elasticsearch.views') # Plug the callback with resource events. config.add_subscriber(on_resource_changed, ResourceChanged) config.add_api_capability("indexed_search", description="Search records using ElasticSearch", url="https://my-super-indexer-for-kinto.org") .. note:: Any argument passed to ``config.add_api_capability()`` will be exposed in the root URL. Default configuration and environment variables ----------------------------------------------- A helper allows to read configuration values from :ref:`environment variables ` and provide default values. .. code-block:: python :emphasize-lines: 1,6-8,11,12 from kinto.core import load_default_settings from kinto.core.events import ResourceChanged from . import indexer DEFAULT_SETTINGS = { 'elasticsearch.refresh_enabled': False } def includeme(config): # Load settings from environment and apply defaults. load_default_settings(config, DEFAULT_SETTINGS) # Register a global indexer object config.registry.indexer = indexer.load_from_config(config) # Activate end-points. config.scan('kinto_elasticsearch.views') # Plug the callback with resource events. config.add_subscriber(on_resource_changed, ResourceChanged) config.add_api_capability("indexed_search", description="Search records using ElasticSearch", url="https://my-super-indexer-for-kinto.org") In this example, if the environment variable ``KINTO_ELASTICSEARCH_REFRESH_ENABLED`` is set to ``true``, it will override the setting ``kinto.elasticsearch.refresh_enabled`` from the ``.ini`` file. Test it altogether ------------------ We're almost done! Now, let's check if it works properly. Create a bucket and collection: :: $ http --auth alice:s3cr3t --verbose PUT http://localhost:8888/v1/buckets/example $ http --auth alice:s3cr3t --verbose PUT http://localhost:8888/v1/buckets/example/collections/notes Add a new record: :: $ echo '{"data": {"note": "kinto"}}' | http --auth alice:s3cr3t --verbose POST http://localhost:8888/v1/buckets/example/collections/notes/records It should now be possible to search for it: :: $ http --auth alice:s3cr3t --verbose POST http://localhost:8888/v1/buckets/default/collections/assets/search .. code-block:: http :emphasize-lines: 20-24 HTTP/1.1 200 OK Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff Content-Length: 333 Content-Type: application/json; charset=UTF-8 Date: Wed, 20 Jan 2016 12:02:05 GMT Server: waitress { "_shards": { "failed": 0, "successful": 5, "total": 5 }, "hits": { "hits": [ { "_id": "453ff779-e967-4b08-99b9-5c16af865a67", "_index": "example-assets", "_score": 1.0, "_source": { "id": "453ff779-e967-4b08-99b9-5c16af865a67", "last_modified": 1453291301729, "note": "kinto" }, "_type": "example-assets" } ], "max_score": 1.0, "total": 1 }, "timed_out": false, "took": 20 } Going further ------------- This plugins implements the basic functionnality. In order to make it a first-class plugin, it would require: * Check that user has ``read`` permission on the collection before searching * Create the index when the collection is created * Create a mapping if the collection has a JSON schema * Delete the index when the bucket or collection are deleted If you feel like doing it, we would be very glad to help you! More documentation ------------------ Some more details about Kinto internals are given in the :ref:`Kinto core docs `!