Events

Introduction

An event can be anything happening in your system that something external wants to be notified about.

The most basic type of event is the Event class, from which other more complicated types of events can be built. The basic event does not have any protocol specification, so any payload is accepted.

An Event is decoupled from subscribers and dispatchers and simply describes an event that can be subscribed to and dispatched to subscribers.

In this guide we will first be describing the basic event building blocks, so that you understand how they work, then move on to the API you are most likely to be using: the ModelEvent class and @webhook_model decorator used to associate events with database models.

Defining events

Say you would like to define an event that dispatches whenever a new user is created, you can do so by creating a new Event object, giving it a name and assigning it to a variable:

from thorn import Event

on_user_created = Event('user.created')

Currently this event is merely defined, and won’t be dispatched under any circumstance unless you manually do so by calling on_user_created.send().

Since the event name is user.created it’s easy to imagine this being sent from something like a web view responsible for creating users, or whenever a user model is changed.

Naming events

Subscribers can filter events by simple pattern matching, so event names should normally be composed out of a category name and an event name, separated by a single dot:

"category.name"

A subscription to "user.*" will match events "user.created", "user.changed", and "user.removed"; while a subscription to "*.created" will match "user.created", "article.created", and so on. A subscription to "*" will match all events.

ModelEvent names may include model instance’s field values. For example, you could define "user.{.username}", and events will be fired as user.alice, user.bob and so on.

Sending events

from userapp import events
from userapp.models import User


def create_user(request):
    username = request.POST['username']
    password = request.POST['password']
    user = User.objects.create(username=username, password=password)
    events.on_user_created.send({
        'uuid': user.uuid,
        'username': user.username,
        'url': 'http://mysite/users/{0}/'.format(user.uuid),
    })

Timeouts and retries

Dispatching an event will ultimately mean performing one or more HTTP requests if there are subscribers attached to that event.

Many HTTP requests will be quick, but some of them will be problematic, especially if you let arbitrary users register external URL callbacks.

A web server taking too long to respond can be handled by setting a socket timeout such that an error is raised. This timeout error can be combined with retries to retry at a later time when the web server is hopefully under less strain.

Slow HTTP requests is usually fine when using the Celery dispatcher, merely blocking that process/thread from doing other work, but when dispatching directly from a web server process it can be deadly, especially if the timeout settings are not tuned properly.

The default timeout for web requests related to an event is configured by the THORN_EVENT_TIMEOUT setting, and is set to 3 seconds by default.

Individual events can override the default timeout by providing either a timeout argument when creating the event:

>>> on_user_created = Event('user.created', timeout=10.0)

or as an argument to the send() method:

>>> on_user_created.send(timeout=1.5)

In addition to the web server being slow to respond, there are other intermittent problems that can occur, such as a 500 (Internal Server Error) response, or even a 404 (Resource Not Found).

The right way to deal with these errors is to retry performing the HTTP request at a later time and this is configured by the event retry policy settings:

>>> on_user_created = Event(
...     'user.created',
...     retry=True,
...     retry_max=10,
...     retry_delay=60.0,
... )

The values used here also happen to be the default setting, and can be configured for all events using the THORN_RETRY, THORN_RETRY_MAX and THORN_RETRY_DELAY settings.

Serialization

Events are always serialized using the json serialization format [*], which means the data you provide in the webhook payload must be representable in json or an error will be raised.

The built-in data types supported by json are:

  • int
  • float
  • string
  • dictionary
  • list

In addition Thorn adds the capability to serialize the following Python types:

Model events

In most cases your events will actually be related to a database model being created, changed, or deleted, which is why Thorn comes with a convenience event type just for this purpose, and even a decorator to easily add webhook-capabilities to your database models.

This is the thorn.ModelEvent event type, and the @webhook_model() decorator.

We will be giving an example in a moment, but first we will discuss the message format for model events.

Message format

The model events have a standard message format specification, which is really more of a header with arbitrary data attached.

An example model event message serialized by json would look like this:

{"event": "(str)event_name",
 "ref": "(URL)model_location",
 "sender": "(User pk)optional_sender",
 "data": {"event specific data": "value"}}

The most important part here is ref, which is optional but lets you link back to the resource affected by the event.

We will discuss reversing models to provide the ref later in this chapter.

Decorating models

The easiest way to add webhook-capabilities to your models is by using the @webhook_model() decorator.

Here’s an example decorating a Django ORM model:

from django.db import models

from thorn import ModelEvent, webhook_model


@webhook_model
class Article(models.Model):
    uuid = models.UUIDField()
    title = models.CharField(max_length=128)
    state = models.CharField(max_length=128, default='PENDING')
    body = models.TextField()

    class webhooks:
        on_create = ModelEvent('article.created')
        on_change = ModelEvent('article.changed')
        on_delete = ModelEvent('article.removed')
        on_publish = ModelEvent(
            'article.published', state__now_eq='PUBLISHED',
        ).dispatches_on_change()

        def payload(self, article):
            return {
                'title': article.title,
            }

    @models.permalink
    def get_absolute_url(self):
        return ('blog:article-detail', None, {'uuid': self.uuid})

Why is this example using Django?

Rest assured that Thorn is not a Django-specific library and should be flexible enough to integrate with any framework, but we have to use something for these generic examples, and Django is the only framework currently supported.

Please get in touch if you want to add support for additional frameworks, it’s not as tricky as it sounds and we can help!

The webhooks we want to define is deferred to a private class inside the model.

The attributes of this class are probably a bit confusing at first, but how expressive this interface is will be apparent once you learn more about them.

So let’s discuss the decorator arguments one by one:

  1. on_create = ModelEvent('article.created')

    Here we specify an event to be sent every time a new object of this model type is created.

    The webhook model decorator can accept an arbitrary number of custom events, but there are three types of events the decorator already knows how to dispatch: on_create, on_change and on_delete. For any additional events you are required to specify the dispatch mechanism (see later explanation of the on_publish argument).

    The name "article.created" here is the event name that subscribers can use to subscribe to this event.

  2. on_change = ModelEvent('article.changed')

    Just like on_create and on_delete the decorator does not need to know when an on_change event is to be dispatched: it will be sent whenever an object of this model type is changed.

  3. on_delete = ModelEvent('article.deleted')

    I’m sure you can guess what this one does already! This event will be sent whenever an object of this model type is deleted.

  4. on_publish = ModelEvent('article.published', state__now_eq='PUBLISHED')

    Here we define a custom event type with an active filter.

    The filter (state__now_eq='PUBLISHED') in combination with the specified dispatch type (.dispatched_on_change) means the event will only be sent when 1) an Article is changed and 2) the updated state changed from something else to "PUBLISHED".

    The decorator will happily accept any argument starting with on_ as an event associated with this model, and any argument to ModelEvent ending with __eq, __ne, __gt, __gte, __lt, __lte, __is, __is_not, __contains, __startswith or __endswith will be regarded as a filter argument.

    You can even use Q objects to create elaborate boolean structures, which is described in detail in the Filtering section.

  5. def webhook_payload

    This method defines what to include in the data section of the webhooks sent for this model.

  6. @models.permalink

    This tells Thorn how to get the canonical URL of an object of this model type, which is used as the ref field in the webhook message payload.

    In this case, when using Django, will translate directly into:

    >>> from django.core.urlresolvers import reverse
>>> reverse('blog:article_detail', kwargs={'uuid': article.uuid})
http://example.com/blog/article/3d90c42c-d61e-4579-ab8f-733d955529ad/

ModelEvent objects

This section describes the ModelEvent objects used with the @webhook_model() decorator in greater detail.

Signal dispatch

Django signals and bulk updates

A limitation with database signals in Django is that signals are not dispatched for bulk operations (objects.delete()/ objects.update()), so you need to dispatch events manually when you use this functionality.

A model event will usually be dispatched in reaction to a signal [*]_, on Django this means connecting to the post_save and post_delete signals.

By signals we mean an implementation of the Observer Pattern, such as django.dispatch.Signal, celery.utils.dispatch.Signal, or blinker (used by Flask).

There are three built-in signal dispatch handlers:

  1. Send when a new model object is created:

    >>> ModelEvent('...').dispatches_on_create()

  2. Send when an existing model object is changed:

    >>> ModelEvent('...').dispatches_on_change()

  3. Send when an existing model object is deleted:

    >>> ModelEvent('...').dispatches_on_delete()

  4. Send when a many-to-many relation is added

    >>> ModelEvent('...').dispatches_on_m2m_add('tags')

    Argument is the related field name, and in this example tags is defined on the model as tags = ManyToManyField(Tag). The event will dispatch whenever Model.tags.add(related_object) happens.

  5. Send when a many-to-many relation is removed

    >>> ModelEvent('...').dispatches_on_m2m_remove('tags')

    Argument is the related field name, and in this example tags is defined on the model as tags = ManyToManyField(Tag). The event will dispatch whenever Model.tags.remove(related_object) happens.

  6. Send when a many-to-many field is cleared

    >>> ModelEvent('...').dispatches_on_m2m_clear('tags')

    Argument is the related field name, and in this example tags is defined on the model as tags = ManyToManyField(Tag). The event will dispatch whenever Model.tags.clear() happens.

The webhook model decorator treats the on_create, on_change, and on_delete arguments specially so that you don’t have to specify the dispatch mechanism for these, but that is not true for any custom events you specify by using the on_ argument prefix to webhook_model.

Side effects in signals

Performing side-effects such as network operations inside a signal handler can make your code harder to reason about.

You can always send events manually, so you can opt-out of using signal-invalidation, but it’s also a very convenient feature and it tends to work well.

Using signal-invalidation means that whenever a model instance is saved (using model.save()), or deleted, the signal handler will automatically also invalidate the cache for you by communicating with the cache server.

This has the potential of disrupting your database transaction in several ways, but we do include some options for you to control this:

  • signal_honors_transaction=True
    default:False (see note below)

    New in version 1.5.

    Example enabling this option:

    ModelEvent(signal_honors_transaction=True, ...)

    When this option is enabled, the actual communication with the cache server to invalidate your keys will be moved to a transaction.on_commit handler.

    This means that if there are multiple webhooks being sent in the same database transaction they will be sent together in one go at the point when the transaction is committed.

    It also means that if the database transaction is rolled back, all the webhooks assocatied with that transaction will be discarded.

    This option requires Django 1.9+ and is disabled by default. It will be enabled by default in Thorn 2.0.

  • propagate_errors
    default:True

    New in version 1.5.

    Example disabling this option:

    ModelEvent(propagate_errors=False, ...)

    By default errors raised while sending a webhook will be logged and ignored (make sure you have Python logging setup in your application).

    You can disable this option to have errors propagate up to the caller, but note that this means a model.save() call will roll back the database transaction if there’s a problem sending the webhook.

Modifying event payloads

The data field part of the resulting model event message will be empty by default, but you can define a special method on your model class to populate this with data relevant for the event.

This callback must be named webhook_payload, takes no arguments, and can return anything as long as it’s json-serializable:

class Article(models.Model):
    uuid = models.UUIDField()
    title = models.CharField(max_length=128)
    state = models.CharField(max_length=128, default='PENDING')
    body = models.TextField()

    def webhook_payload(self):
        return {
            'title': self.title,
            'state': self.state,
            'body': self.body[:1024],
        }

You should carefully consider what you include in the payload to make sure your messages are as small and lean as possible, so in this case we truncate the body of the article to save space.

Tip

Do we have to include the article body at all?

Remember that the webhook message will include the ref field containing a URL pointing back to the affected resource, so the recipient can request the full contents of the article if they want to.

Including the body will be a question of how many of your subscribers will require the full article text. If the majority of them will, including the body will save them from having to perform an extra HTTP request, but if not, you have drastically increased the size of your messages.

Modifying event headers

You can include additional headers for the resulting model event message by defining a special method on your model class.

This callback must be named webhook_headers, takes no arguments, and must return a dictionary:

from django.conf import settings
from django.db import models

class Article(models.Model):
    uuid = models.UUIDField()
    title = models.CharField(max_length=128)
    state = models.CharField(max_length=128, default='PENDING')
    body = models.TextField()
    user = models.ForeignKey(settings.AUTH_USER_MODEL)

    class webhooks:

        def headers(self, article):
            return {
                'Authorization':
                    'Bearer {}'.format(article.user.access_token),
            }

Event senders

If your model is associated with a user and you want subscribers to filter based on the owner/author/etc. of the model instance, you can include the sender_field argument:

from django.contrib.auth import get_user_model
from django.db import models


@webhook_model(
    sender_field='author.account.user',
)
class Article(models.Model):
    author = models.ForeignKey(Author)


class Author(models.Model):
    account = models.ForeignKey(Account)


class Account(models.Model):
    user = models.ForeignKey(get_user_model())

URL references

To be able to provide a URL reference back to your model object the event needs to know how to call django.core.urlresolvers.reverse() (or equivalent in your web framework) and what arguments to use.

A best practice when writing Django apps is to always add a get_absolute_url method to your models:

class Article(models.Model):

    @models.permalink
    def get_absolute_url(self):
        return ('article:detail', None, {'uuid': self.uuid})

If you define this method, then Thorn will happily use it, but some times you may also want to define alternate reversing strategies for specific events (such as article.deleted: when the article is deleted referring to the URL of the article does not make sense, but you could point to the category an article belongs to for example).

This is where the model_reverser helper comes in, which simply describes how to turn an instance of your model into the arguments used for reverse.

The signature of model_reverser is:

model_reverser(view_name, *reverse_args, **reverse_kwargs)

The positional arguments will be the names of attributes to take from the model instance, and the same for keyword arguments.

So if we imagine that the REST API view of our article app is included like this:

url(r'^article/', include(
    'apps.article.urls', namespace='article'))

and the URL routing table of the Article app looks like this:

urlpatterns = [
    url(r'^$',
        views.ArticleList.as_view(), name='list'),
    url(r'^(?P<uuid>[0-9a-fA-F-]+)/$',
        views.ArticleDetail.as_view(), name='detail'),
]

We can see that to get the URL of a specific article we need 1) the name of the view (article:detail), and 2) a named uuid keyword argument:

>>> from django.core.urlresolvers import reverse
>>> article = Article.objects.get(uuid='f3f2b22b-8630-412a-a320-5b2644ed723a')
>>> reverse('article:detail', kwargs={'uuid': article.uuid})
http://example.com/article/f3f2b22b-8630-412a-a320-5b2644ed723a/

So to define a reverser for this model we can use:

model_reverser('article:detail', uuid='uuid')

The uuid='uuid' here means take the uuid argument from the identically named field on the instance (article.uuid).

Any attribute name is accepted as a value, and even nested attributes are supported:

model_reverser('broker:position',
               account='user.profile.account')
#               ^^ will be taken from instance.user.profile.account

Filtering

Model events can filter models by matching attributes on the model instance.

The most simple filter would be to match a single field only:

ModelEvent('article.changed', state__eq='PUBLISHED')

and this will basically transform into the predicate:

if instance.state == 'PUBLISHED':
    send_event(instance)

This may not be what you want since it will always match even if the value was already set to "PUBLISHED" before. To only match on the transition from some other value to "PUBLISHED" you can use now_eq instead:

ModelEvent('article.changed', state__now_eq='PUBLISHED')

which will transform into the predicate:

if (old_value(instance, 'state') != 'PUBLISHED' and
        instance.state == 'PUBLISHED'):
    send_event(instance)

Transitions and performance

Using the now_* operators means Thorn will have to fetch the old object from the database before the new version is saved, so an extra database hit is required every time you save an instance of that model.

You can combine as many filters as you want:

ModelEvent('article.changed',
           state__eq='PUBLISHED',
           title__startswith('The'))

In this case the filters form an AND relationship and will only continue if all of the filters match:

if instance.state == 'PUBLISHED' and instance.title.startswith('The'):
    send_event(instance)

If you want an OR relationship or to combine boolean gates, you will have to use Q objects:

from thorn import ModelEvent, Q


ModelEvent(
    'article.changed',
    Q(state__eq='PUBLISHED') | Q(state__eq='PREVIEW'),
)

You can also negate filters using the ~ operator:

ModelEvent(
    'article.changed',
    (
        Q(state__eq='PUBLISHED') |
        Q(state__eq='PREVIEW') &
        ~Q(title__startswith('The'))
    )
)

Which as our final example will translate into the following pseudo-code:

if (not instance.title.startswith('The') and
        (instance.state == 'PUBLISHED' or instance.state == 'PREVIEW')):
    send_event(instance)

Tip

Thorn will happily accept Django’s Q objects, so you don’t have to import Q from Thorn when you already have one from Django.

Note that you are always required to specify __eq when specifying filters:

ModelEvent('article.created', state='PUBLISHED')      # <--- DOES NOT WORK

ModelEvent('article.created', state__eq='PUBLISHED')  # <-- OK! :o)

Supported operators

Operator Description
eq=B value equal to B (__eq=True tests for truth)
now_eq=B value equal to B and was previously not equal to B
ne=B value not equal to B (__eq=False tests for falsiness)
now_ne=B value now not equal to B, but was previously equal to B
gt=B value is greater than B: A > B
now_gt=B value is greater than B, but was previously less than B
gte=B value is greater than or equal to B: A >= B
now_gte=B value greater or equal to B, previously less or equal
lt=B value is less than B: A < B
now_lt=B value is less than B, previously greater than B
lte=B value is less than or equal to B: A <= B
now_lte=B value less or equal to B, previously greater or equal.
is=B test for object identity: A is B
now_is=B value is now identical, but was not previously
is_not=B negated object identity: A is not B
now_is_not=B value is no longer identical, but was previously
in={B, …} value is a member of set: A in {B, …}
now_in={B, …} value is now member of set, but was not before
not_in={B, …} value is not a member of set: A not in {B, …}
now_not_in={B, …} value is not a member of set, but was before
contains=B value contains element B: B in A
now_contains=B value now contains element B, but did not previously
startswith=B string starts with substring B
now_startswith=B string now startswith B, but did not previously
endswith=B string ends with substring B
now_endswith=B string now ends with B, but did not previously

Tips

  • Test for truth/falsiness

    There are two special cases for the eq operator: __eq=True and _eq=False is functionally equivalent to if A and if not A so any true-ish or false-ish value will be a match.

    Similarly with ne the cases __ne=True and __ne=False are special and translates to if not A and if A respectively.

  • Use A__is=None for testing that A is None

  • contains is not limited to strings!

    This operator supports any object supporting the __contains__ protocol so in addition to strings it can also be used for sets, lists, tuples, dictionaries and other containers. E.g.: B in {1, 2, 3, 4}.

  • The transition operators (__now_*) may affect Django database performance.

    Django signals does provide a way to get the previous value of a database row when saving an object, so Thorn is required to manually re-fetch the object from the database shortly before the object is saved.

Sending model events manually

The webhook model decorator will add a new webhooks attribute to your model that can be used to access the individual model events:

>>> on_create = Article.webhooks.events['on_create']

With this you can send the event manually just like any other Event:

>>> on_create.send(instance=article, data=article.webhook_payload())

There’s also .send_from_instance which just takes a model instance as argument and will send the event as if a signal was triggered:

>>> on_create.send_from_instance(instance)

The payload will then look like:

{
    "event": "article.created",
    "ref": "http://example.com/article/5b841406-60d6-4ca0-b45e-72a9847391fb/",
    "sender": null,
    "data": {"title": "The Mighty Bear"},
}

Footnotes

[*]Thorn can easily be extended to support additional serialization formats. If this is something you would like to work on then please create an issue on the Github issue tracker or otherwise get in touch with the project.

Security

The REST Hooks project has an excellent guide on security and webhooks here: http://resthooks.org/docs/security/