Monitoring Celery Queue Length with RabbitMQ

Monitoring Celery Queue Length with RabbitMQ

Earlier this week Matthew Schinckel wrote a post about how he monitors Celery queue sizes with a Redis backend.

RabbitMQ (https://www.rabbitmq.com/ is also a popular backend for Celery, and it took us a long time to get good visibility into our task queues and backlogs. So we'd like to share our solution for monitoring RabbitMQ celery queues for others that might be in a similar situation.

Management Command Aliases

Management Command Aliases

Django Management Commands

A quick minor efficiency tip for today that'll make your life at the Django command line incrementally better.

You're using django-extensions in your project, right? If not, go and rectify that problem immediately.

The best feature™ that django-extensions provides is the shell_plus command. You know the worst thing about `shell_plus`? The name. That's a long name to type out into your shell, especially since it's snake_case, and I type the command approximately 15,000 times per day. Wouldn't it be nice to alias `shell_plus` to something like `sh`? Well, now you can!

Django on Kubernetes

Django on Kubernetes

This is part 2 of our Kubernetes hackday series. Part 1 goes over how we spent the day and what the goals and motivations are.


For part 2, we're going to delve into the architecture required for running a Django application on Kubernetes, as well as some of the tooling we used to assist us.
This post will assume some knowledge with deploying and operating a production web application. I'm not going to spend much time going over the terminology that Kubernetes uses either. What I hope to do in this post is present enough information to kickstart your own migration. Kubernetes is big - and knowing what to research now and what to put off until later is really tricky.
I'm also going to describe this deployment in terms of Amazon EKS simply because I was on the EKS team, and most of our applications are already on AWS.

All Hands on Deck - Kubernetes Hackday (Part 1)

On the 5th of November 2018, the IT team at Kogan.com started another hackday. This time, we set our goal on learning Kubernetes. We wanted to answer the question; how exactly can we leverage container orchestration to make our deployment process faster and more efficient?

In order to achieve our goal, we set out to deploy one of our major apps that controls customer subscription preferences with Kubernetes on two different cloud providers, Google Cloud Platform (GCP GKE) and Amazon Web Services (AWS EKS). By doing this, we hoped we could understand the pros and cons of each platform, while learning the intricacies of Kubernetes deployment at the same time.

This will be a two part blog series. The first part will be a short overview of our motivation, goals, as well as how the day actually unfolded. The second part will be more technical, focus on the approaches of the two teams, and discuss the pros and cons of each platform.

June Hackday - Lifx Smart Tiles (Part 3)

June Hackday - Lifx Smart Tiles (Part 3)

Last June we had a Hackday at Kogan.com! This Hackday’s focus on displaying information and providing useful alerts using hardware and software. Teams were asked to express something they find interesting through one of the available mediums.

In this third and final part of the series Jake will talk a bit about Team LifxnChill.

June Hackday - Team KASX (Part 1)

June Hackday - Team KASX (Part 1)

Last June we had a Hackday at Kogan.com! Team KASX was given the task of creating an app using React Native for a mobile device that could be used to display information such as Kogan’s stock price.

Faster Django Tests by Disabling Signals

Django signals are a form of Inversion Of Control that developers can use to trigger changes based (usually) on updates to models.

The canonical example is automatically creating a UserProfile when a User instance is created:

from django.conf import settings
from django.db.models.signals import post_save
from django.dispatch import receiver

@receiver(post_save, sender=settings.AUTH_USER_MODEL)
def post_save_receiver(sender, instance, created, **kwargs):
    UserProfile.objects.create(user=instance, ...)

Signals are useful for connecting your own behaviour to events that you might not have control over, such as those generated by the framework or libary code. In general, you should prefer other methods like overriding model save methods if you have the ability, as it makes code easier to reason about.

Signal receivers can sometimes be much more complex than creating a single object, and there can be many receivers for any given event, which multiply the time it takes to perform simple actions.

Tests and signals

Often you won't actually need your signals to execute when you're running your test suite, especially if you're creating and deleting many thousands of model instances. Disconnecting signals is tricky though, especially if the disconnect/reconnect logic can be stacked.

An easier, but more invasive method for suspending signal receivers is to check a global setting, and return early.

from django.conf import settings

@receiver(post_save, sender=MyModel)
def mymodel_post_save(sender, **kwargs):
    if settings.DEBUG:
        return
    work()

This has two drawbacks. Firstly, it's messy, and you need to remember to add the check to each receiver. Secondly, it depends on a specific setting that you might need to have turned off when running your tests, which makes it harder to test the actual receiver.

Selectively disabling signals

We can take this most recent idea of checking a setting, and wrap it up nicely in our own decorator. When running your tests, you can override the SUSPEND_SIGNALS setting per test method or class.

Here's a gist that does just that

import functools

from django.conf import settings
from django.dispatch import receiver

def suspendingreceiver(signal, **decorator_kwargs):
    def our_wrapper(func):
        @receiver(signal, **decorator_kwargs)
        @functools.wraps(func)
        def fake_receiver(sender, **kwargs):
            if settings.SUSPEND_SIGNALS:
                return
            return func(sender, **kwargs)
        return fake_receiver
    return our_wrapper

And using this decorator in your test suite is straightforward:

@override_settings(SUSPEND_SIGNALS=True)
class MyTestCase(TestCase):
    def test_method(self):
        Model.objects.create()  # post_save_receiver won't execute

@suspendingreceiver(post_save, sender=MyModel)
def mymodel_post_save(sender, **kwargs):
    work()

And just like that, we can skip signal receivers in our test suite as we like!

Kogame (Koh-Gah-Mi) - A real time game in Django

Kogame (Koh-Gah-Mi) - A real time game in Django

For our March hackday this year we decided to build a multiplayer game using

[Django Channels](https://channels.readthedocs.io/en/latest/). We've been keeping

an eye on channels over the time, and thought with the release of channels 2.0 that

it was the right time to dive in and get some practical experience. The organisers

didn't want to do yet-another-chat-system implementation, so they decided to make

things a bit more interesting, and look at writing a real-time game.

React JS Melbourne - May Meetup

The React meetup was on again at Kogan HQ last Tuesday!

Four presenters covered three exciting topics; including the use of React in non-profit organisations, understanding the new Context API and scaling a React team and their tools from 2 to 20 devs.

Due to the large waitlist for this quarter's event, we trialled a live feed via Google hangouts which went quite well!

The event attracted 200 RSVP’s who consumed many burritos throughout the evening.

The event attracted 200 RSVP’s who consumed many burritos throughout the evening.

Want to come along next time? Tune in to  our Meetup Page .

Want to come along next time? Tune in to our Meetup Page.

 

Links

Harnessing the Power of Volunteers for Community Benefit - Mike King & Matt Wiseman (Back2Bikes)

Github repo - https://github.com/Back2bikes/attendance-app

 

Understanding the New Context API - Mair Swartz


Github repo - https://github.com/Mair/react-meetup-context-api