Flower - Celery monitoring tool¶

Flower is a web based tool for monitoring and administrating Celery clusters

Features¶

Real-time monitoring using Celery Events
- Task progress and history
- Ability to show task details (arguments, start time, runtime, and more)
- Graphs and statistics
Remote Control
- View worker status and statistics
- Shutdown and restart worker instances
- Control worker pool size and autoscale settings
- View and modify the queues a worker instance consumes from
- View currently running tasks
- View scheduled tasks (ETA/countdown)
- View reserved and revoked tasks
- Apply time and rate limits
- Configuration viewer
- Revoke or terminate tasks
Broker monitoring
- View statistics for all Celery queues
- Queue length graphs
HTTP API
Basic Auth, GitHub OAuth2 and Google OpenID authentication
Prometheus integration

Contents¶

Features¶

Real-time monitoring using Celery Events
- Task progress and history
- Ability to show task details (arguments, start time, runtime, and more)
- Graphs and statistics
Remote Control
- View worker status and statistics
- Shutdown and restart worker instances
- Control worker pool size and autoscale settings
- View and modify the queues a worker instance consumes from
- View currently running tasks
- View scheduled tasks (ETA/countdown)
- View reserved and revoked tasks
- Apply time and rate limits
- Configuration viewer
- Revoke or terminate tasks
Broker monitoring
- View statistics for all Celery queues
- Queue length graphs
HTTP API
Basic Auth, GitHub OAuth2 and Google OpenID authentication
Prometheus integration

Screenshots¶

Worker dashboard:

Task dashboard:

Worker tasks:

Grafana dashboard for celery:

Worker info:

Task info:

Configuration viewer:

Installation¶

Installing flower with pip is simple

$ pip install flower

Development version can be installed with

$ pip install https://github.com/mher/flower/zipball/master#egg=flower

Usage¶

Important Please note that from version 1.0.1 Flower uses Celery 5 and has to be invoked in the same style as celery commands do.

The key takeaway here is that the Celery app’s arguments have to be specified after the celery command and Flower’s arguments have to be specified after the flower sub-command.

This is the template to follow:

celery [celery args] flower [flower args]

Core Celery args that you may want to set:

-A, --app
-b, --broker
--result-backend

More info on available Celery command args.

For Flower command args see here.

Usage Examples¶

Launch the Flower server at specified port other than default 5555 (open the UI at http://localhost:5566):

$ celery flower --port=5566

Specify Celery application path with address and port for Flower:

$ celery -A proj flower  --address=127.0.0.6 --port=5566

Launch using docker:

$ docker run -p 5555:5555 mher/flower

Launch with unix socket file:

$ celery flower --unix-socket=/tmp/flower.sock

Broker URL and other configuration options can be passed through the standard Celery options (notice that they are after Celery command and before Flower sub-command):

$ celery --broker=amqp://guest:guest@localhost:5672// flower

Configuration¶

Flower can be configured from the command line:

$ celery flower --auto_refresh=False

Using flowerconfig.py configuration file:

# RabbitMQ management api
broker_api = 'http://guest:guest@localhost:15672/api/'

# Enable debug logging
logging = 'DEBUG'

Or, using the environment variables. All flower options should be prefixed with FLOWER_:

$ export FLOWER_BASIC_AUTH=foo:bar

Options passed through the command line have precedence over the options defined in the configuration file. The configuration file name and path can be changed with conf option.

$ celery flower --conf=celeryconfig.py

Options¶

Standard Celery configuration settings can be overridden in the configuration file. See Celery Configuration reference for a complete listing of all the available settings, and their default values.

Celery command line options also can be passed to Flower. For example the –broker sets the default broker URL:

$ celery -A proj --broker=amqp://guest:guest@localhost:5672// flower

For a full list of options see:

$ celery --help

address ¶

Run the http server on a given address. Address may be either an IP address or hostname. If it’s a hostname, the server will listen on all IP addresses associated with the name. Address may be an empty string or None to listen on all available interfaces.

auth ¶

Enables authentication. auth is a regexp of emails to grant access. For more info see authentication.

auto_refresh ¶

Refresh dashboards automatically (by default, auto_refresh=True)

basic_auth ¶

Enables HTTP Basic authentication. basic_auth is a comma separated list of username:password. See HTTP Basic Authentication for more info.

broker_api ¶

Flower uses RabbitMQ Management Plugin to get info about queues. broker_api is a URL of RabbitMQ HTTP API including user credentials.

$ celery -A proj flower --broker_api=http://username:password@rabbitmq-server-name:15672/api/

Note

By default the management plugin is not enabled. To enable it run:

$ rabbitmq-plugins enable rabbitmq_management

Note

The port number for RabbitMQ versions prior to 3.0 is 55672.

ca_certs ¶

A path to ca_certs file. The ca_certs file contains a set of concatenated “certification authority” certificates, which are used to validate certificates passed from the other end of the connection. For more info see Python SSL

certfile ¶

A path to SSL certificate file

conf ¶

A path to the configuration file (by default, flowerconfig.py)

db ¶

A database file to use if persistent mode is enabled (by default, db=flower)

debug ¶

Enable the debug mode (by default, debug=False)

enable_events ¶

Periodically enable Celery events by using enable_events command (by default, enable_event=True)

format_task ¶

Modifies the default task formatting. format_task function should be defined in the flowerconfig.py configuration file. It accepts a task object and returns the modified version.

format_task is useful for filtering out sensitive information.

The example below shows how to filter arguments and limit display lengths:

from flower.utils.template import humanize

def format_task(task):
    task.args = humanize(task.args, length=10)
    task.kwargs.pop('credit_card_number')
    task.result = humanize(task.result, length=20)
    return task

inspect_timeout ¶

Sets worker inspect timeout (by default, inspect_timeout=1000 in milliseconds)

keyfile ¶

A path to SSL key file

max_workers ¶

Maximum number of workers to keep in memory (by default, max_workers=5000)

max_tasks ¶

Maximum number of tasks to keep in memory (by default, max_tasks=10000)

natural_time ¶

Show time relative to the refresh time (by default, natural_time=True)

persistent ¶

Enable persistent mode. If the persistent mode is enabled Flower saves the current state and reloads on restart (by default, persistent=False)

port ¶

Run the http server on a given port (by default, port=5555)

state_save_interval ¶

Sets the interval for saving state. state_save_interval=0 means that periodic saving is disabled (by default, state_save_interval=0 in milliseconds)

xheaders ¶

Enable support of X-Real-Ip and X-Scheme headers (by default, xheaders=False)

tasks_columns ¶

Specifies list of comma-delimited columns on /tasks/ page. all value enables all columns. Columns on the page can be reordered using drag and drop.

(by default, tasks_columns=”name,uuid,state,args,kwargs,result,received,started,runtime,worker”)

Available columns are:

name

uuid

state

args

kwargs

result

received

started

runtime

worker

retries

revoked

exception

expires

eta

url_prefix ¶

Enables deploying Flower on non-root URL

For example to access Flower on http://example.com/flower run it with:

$ celery flower --url_prefix=flower

NOTE: The old nginx rewrite is no longer needed

unix_socket ¶

Run flower using UNIX socket file

auth_provider ¶

Sets authentication provider

Google flower.views.auth.GoogleAuth2LoginHandler

GitHub flower.views.auth.GithubLoginHandler

GitLab flower.views.auth.GitLabLoginHandler

See Authentication for usage examples

purge_offline_workers ¶

Time (in seconds) after which offline workers are automatically removed from dashboard.

If omitted, offline workers remain on the dashboard.

Tasks filtering¶

By now, tasks can be filtered by worker, type, state, received and started datetime. Also, filtering by args/kwargs/result/state value available.

Task filter syntax¶

Flower uses github-style syntax for args/kwargs/result filtering.

foo find all tasks containing foo in args, kwargs or result

args:foo find all tasks containing foo in arguments

kwargs:foo=bar find all tasks containing foo=bar keyword

result:foo find all tasks containing foo in result

state:FAILURE find all failed tasks

If the search term contains spaces it should be enclosed in ” (e.g. args:”hello world”).

For examples, see tests/utils/test_search.py.

API Reference¶

GET /api/workers¶

List workers

Example request:

GET /api/workers HTTP/1.1
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 1526
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Jul 2015 01:32:38 GMT
Etag: "fcdd75d85a82b4052275e28871d199aac1ece21c"
Server: TornadoServer/4.0.2

{
    "celery@worker1": {
        "active_queues": [
            {
                "alias": null,
                "auto_delete": false,
                "binding_arguments": null,
                "bindings": [],
                "durable": true,
                "exchange": {
                    "arguments": null,
                    "auto_delete": false,
                    "delivery_mode": 2,
                    "durable": true,
                    "name": "celery",
                    "passive": false,
                    "type": "direct"
                },
                "exclusive": false,
                "name": "celery",
                "no_ack": false,
                "queue_arguments": null,
                "routing_key": "celery"
            }
        ],
        "conf": {
            "CELERYBEAT_SCHEDULE": {},
            "CELERY_INCLUDE": [
                "celery.app.builtins",
                "__main__"
            ],
            "CELERY_SEND_TASK_SENT_EVENT": true,
            "CELERY_TIMEZONE": "UTC"
        },
        "registered": [
            "tasks.add",
            "tasks.echo",
            "tasks.error",
            "tasks.retry",
            "tasks.sleep"
        ],
        "stats": {
            "broker": {
                "alternates": [],
                "connect_timeout": 4,
                "heartbeat": null,
                "hostname": "127.0.0.1",
                "insist": false,
                "login_method": "AMQPLAIN",
                "port": 5672,
                "ssl": false,
                "transport": "amqp",
                "transport_options": {},
                "uri_prefix": null,
                "userid": "guest",
                "virtual_host": "/"
            },
            "clock": "918",
            "pid": 90494,
            "pool": {
                "max-concurrency": 4,
                "max-tasks-per-child": "N/A",
                "processes": [
                    90499,
                    90500,
                    90501,
                    90502
                ],
                "put-guarded-by-semaphore": false,
                "timeouts": [
                    0,
                    0
                ],
                "writes": {
                    "all": "100.00%",
                    "avg": "100.00%",
                    "inqueues": {
                        "active": 0,
                        "total": 4
                    },
                    "raw": "1",
                    "total": 1
                }
            },
            "prefetch_count": 16,
            "rusage": {
                "idrss": 0,
                "inblock": 211,
                "isrss": 0,
                "ixrss": 0,
                "majflt": 6,
                "maxrss": 26996736,
                "minflt": 11450,
                "msgrcv": 4968,
                "msgsnd": 1227,
                "nivcsw": 1367,
                "nsignals": 0,
                "nswap": 0,
                "nvcsw": 1855,
                "oublock": 93,
                "stime": 0.414564,
                "utime": 0.975726
            },
            "total": {
                "tasks.add": 1
            }
        },
        "timestamp": 1438049312.073402
    }
}

Query Parameters:
	refresh – run inspect to get updated list of workers workername – get info for workername status – only get worker status info
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request

POST /api/worker/shutdown/(.+)¶

Shut down a worker

Example request:

POST /api/worker/shutdown/celery@worker2 HTTP/1.1
Content-Length: 0
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json; charset=UTF-8

{
    "message": "Shutting down!"
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown worker

POST /api/worker/pool/restart/(.+)¶

Restart worker’s pool

Example request:

POST /api/worker/pool/restart/celery@worker2 HTTP/1.1
Content-Length: 0
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 56
Content-Type: application/json; charset=UTF-8

{
    "message": "Restarting 'celery@worker2' worker's pool"
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – pool restart is not enabled (see CELERYD_POOL_RESTARTS) 404 Not Found – unknown worker

POST /api/worker/pool/grow/(.+)¶

Grow worker’s pool

Example request:

POST /api/worker/pool/grow/celery@worker2?n=3 HTTP/1.1
Content-Length: 0
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 58
Content-Type: application/json; charset=UTF-8

{
    "message": "Growing 'celery@worker2' worker's pool by 3"
}

Query Parameters:
	n – number of pool processes to grow, default is 1
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – failed to grow 404 Not Found – unknown worker

POST /api/worker/pool/shrink/(.+)¶

Shrink worker’s pool

Example request:

POST /api/worker/pool/shrink/celery@worker2 HTTP/1.1
Content-Length: 0
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 60
Content-Type: application/json; charset=UTF-8

{
    "message": "Shrinking 'celery@worker2' worker's pool by 1"
}

Query Parameters:
	n – number of pool processes to shrink, default is 1
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – failed to shrink 404 Not Found – unknown worker

POST /api/worker/pool/autoscale/(.+)¶

Autoscale worker pool

Example request:

POST /api/worker/pool/autoscale/celery@worker2?min=3&max=10 HTTP/1.1
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 66
Content-Type: application/json; charset=UTF-8

{
    "message": "Autoscaling 'celery@worker2' worker (min=3, max=10)"
}

Query Parameters:
	min – minimum number of pool processes max – maximum number of pool processes
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – autoscaling is not enabled (see CELERYD_AUTOSCALER) 404 Not Found – unknown worker

POST /api/worker/queue/add-consumer/(.+)¶

Start consuming from a queue

Example request:

POST /api/worker/queue/add-consumer/celery@worker2?queue=sample-queue
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 40
Content-Type: application/json; charset=UTF-8

{
    "message": "add consumer sample-queue"
}

Query Parameters:
	queue – the name of a new queue
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – failed to add consumer 404 Not Found – unknown worker

POST /api/worker/queue/cancel-consumer/(.+)¶

Stop consuming from a queue

Example request:

POST /api/worker/queue/cancel-consumer/celery@worker2?queue=sample-queue
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 52
Content-Type: application/json; charset=UTF-8

{
    "message": "no longer consuming from sample-queue"
}

Query Parameters:
	queue – the name of queue
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 403 Forbidden – failed to cancel consumer 404 Not Found – unknown worker

GET /api/tasks¶

List tasks

Example request:

GET /api/tasks HTTP/1.1
Host: localhost:5555
User-Agent: HTTPie/0.8.0

Example response:

HTTP/1.1 200 OK
Content-Length: 1109
Content-Type: application/json; charset=UTF-8
Etag: "b2478118015c8b825f7b88ce6b660e5449746c37"
Server: TornadoServer/3.1.1

{
    "e42ceb2d-8730-47b5-8b4d-8e0d2a1ef7c9": {
        "args": "[3, 4]",
        "client": null,
        "clock": 1079,
        "eta": null,
        "exception": null,
        "exchange": null,
        "expires": null,
        "failed": null,
        "kwargs": "{}",
        "name": "tasks.add",
        "received": 1398505411.107885,
        "result": "'7'",
        "retried": null,
        "retries": 0,
        "revoked": null,
        "routing_key": null,
        "runtime": 0.01610181899741292,
        "sent": null,
        "started": 1398505411.108985,
        "state": "SUCCESS",
        "succeeded": 1398505411.124802,
        "timestamp": 1398505411.124802,
        "traceback": null,
        "uuid": "e42ceb2d-8730-47b5-8b4d-8e0d2a1ef7c9",
        "worker": "celery@worker1"
    },
    "f67ea225-ae9e-42a8-90b0-5de0b24507e0": {
        "args": "[1, 2]",
        "client": null,
        "clock": 1042,
        "eta": null,
        "exception": null,
        "exchange": null,
        "expires": null,
        "failed": null,
        "kwargs": "{}",
        "name": "tasks.add",
        "received": 1398505395.327208,
        "result": "'3'",
        "retried": null,
        "retries": 0,
        "revoked": null,
        "routing_key": null,
        "runtime": 0.012884548006695695,
        "sent": null,
        "started": 1398505395.3289,
        "state": "SUCCESS",
        "succeeded": 1398505395.341089,
        "timestamp": 1398505395.341089,
        "traceback": null,
        "uuid": "f67ea225-ae9e-42a8-90b0-5de0b24507e0",
        "worker": "celery@worker1"
    }
}

Query Parameters:
	limit – maximum number of tasks offset – skip first n tasks sort_by – sort tasks by attribute (name, state, received, started) workername – filter task by workername taskname – filter tasks by taskname state – filter tasks by state received_start – filter tasks by received date (must be greater than) format %Y-%m-%d %H:%M received_end – filter tasks by received date (must be less than) format %Y-%m-%d %H:%M
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request

GET /api/task/types¶

List (seen) task types

Example request:

GET /api/task/types HTTP/1.1
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 44
Content-Type: application/json; charset=UTF-8

{
    "task-types": [
        "tasks.add",
        "tasks.sleep"
    ]
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request

GET /api/queues/length¶

Return length of all active queues

Example request:

GET /api/queues/length
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 94
Content-Type: application/json; charset=UTF-8

{
    "active_queues": [
        {"name": "celery", "messages": 0},
        {"name": "video-queue", "messages": 5}
    ]
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 503 Service Unavailable – result backend is not configured

GET /api/task/info/(.*)¶

Get a task info

Example request:

GET /api/task/info/91396550-c228-4111-9da4-9d88cfd5ddc6 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, compress
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 575
Content-Type: application/json; charset=UTF-8

{
    "args": "[2, 2]",
    "client": null,
    "clock": 25,
    "eta": null,
    "exception": null,
    "exchange": null,
    "expires": null,
    "failed": null,
    "kwargs": "{}",
    "name": "tasks.add",
    "received": 1400806241.970742,
    "result": "'4'",
    "retried": null,
    "retries": null,
    "revoked": null,
    "routing_key": null,
    "runtime": 2.0037889280356467,
    "sent": null,
    "started": 1400806241.972624,
    "state": "SUCCESS",
    "succeeded": 1400806243.975336,
    "task-id": "91396550-c228-4111-9da4-9d88cfd5ddc6",
    "timestamp": 1400806243.975336,
    "traceback": null,
    "worker": "celery@worker1"
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task

POST /api/task/apply/(.+)¶

Execute a task by name and wait results

Example request:

POST /api/task/apply/tasks.add HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, compress
Content-Length: 16
Content-Type: application/json; charset=utf-8
Host: localhost:5555

{
    "args": [1, 2]
}

Example response:

HTTP/1.1 200 OK
Content-Length: 71
Content-Type: application/json; charset=UTF-8

{
    "state": "SUCCESS",
    "task-id": "c60be250-fe52-48df-befb-ac66174076e6",
    "result": 3
}

Query Parameters:
	args – a list of arguments kwargs – a dictionary of arguments
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task

POST /api/task/async-apply/(.+)¶

Execute a task

Example request:

POST /api/task/async-apply/tasks.add HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, compress
Content-Length: 16
Content-Type: application/json; charset=utf-8
Host: localhost:5555

{
    "args": [1, 2]
}

Example response:

HTTP/1.1 200 OK
Content-Length: 71
Content-Type: application/json; charset=UTF-8
Date: Sun, 13 Apr 2014 15:55:00 GMT

{
    "state": "PENDING",
    "task-id": "abc300c7-2922-4069-97b6-a635cc2ac47c"
}

Query Parameters:
	args – a list of arguments kwargs – a dictionary of arguments options – a dictionary of apply_async keyword arguments
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task

POST /api/task/send-task/(.+)¶

Execute a task by name (doesn’t require task sources)

Example request:

POST /api/task/send-task/tasks.add HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, compress
Content-Length: 16
Content-Type: application/json; charset=utf-8
Host: localhost:5555

{
    "args": [1, 2]
}

Example response:

HTTP/1.1 200 OK
Content-Length: 71
Content-Type: application/json; charset=UTF-8

{
    "state": "SUCCESS",
    "task-id": "c60be250-fe52-48df-befb-ac66174076e6"
}

Query Parameters:
	args – a list of arguments kwargs – a dictionary of arguments
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task

GET /api/task/result/(.+)¶

Get a task result

Example request:

GET /api/task/result/c60be250-fe52-48df-befb-ac66174076e6 HTTP/1.1
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 84
Content-Type: application/json; charset=UTF-8

{
    "result": 3,
    "state": "SUCCESS",
    "task-id": "c60be250-fe52-48df-befb-ac66174076e6"
}

Query Parameters:
	timeout – how long to wait, in seconds, before the operation times out
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 503 Service Unavailable – result backend is not configured

POST /api/task/abort/(.+)¶

Abort a running task

Example request:

POST /api/task/abort/c60be250-fe52-48df-befb-ac66174076e6 HTTP/1.1
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 61
Content-Type: application/json; charset=UTF-8

{
    "message": "Aborted '1480b55c-b8b2-462c-985e-24af3e9158f9'"
}

Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 503 Service Unavailable – result backend is not configured

POST /api/task/timeout/(.+)¶

Change soft and hard time limits for a task

Example request:

POST /api/task/timeout/tasks.sleep HTTP/1.1
Content-Length: 44
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

soft=30&hard=100&workername=celery%40worker1

Example response:

HTTP/1.1 200 OK
Content-Length: 46
Content-Type: application/json; charset=UTF-8

{
    "message": "time limits set successfully"
}

Query Parameters:
	workername – worker name
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task/worker

POST /api/task/rate-limit/(.+)¶

Change rate limit for a task

Example request:

POST /api/task/rate-limit/tasks.sleep HTTP/1.1
Content-Length: 41
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

ratelimit=200&workername=celery%40worker1

Example response:

HTTP/1.1 200 OK
Content-Length: 61
Content-Type: application/json; charset=UTF-8

{
    "message": "new rate limit set successfully"
}

Query Parameters:
	workername – worker name
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request 404 Not Found – unknown task/worker

POST /api/task/revoke/(.+)¶

Revoke a task

Example request:

POST /api/task/revoke/1480b55c-b8b2-462c-985e-24af3e9158f9?terminate=true
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:5555

Example response:

HTTP/1.1 200 OK
Content-Length: 61
Content-Type: application/json; charset=UTF-8

{
    "message": "Revoked '1480b55c-b8b2-462c-985e-24af3e9158f9'"
}

Query Parameters:
	terminate – terminate the task if it is running signal – name of signal to send to process if terminate (default: ‘SIGTERM’)
Request Headers:
	Authorization – optional OAuth token to authenticate
Status Codes:	200 OK – no error 401 Unauthorized – unauthorized request

Authentication¶

Protecting your Flower instance from unwarranted access is important if it runs in an untrusted environment. Below, we outline the various forms of authentication supported by Flower.

HTTP Basic Authentication¶

Securing Flower with Basic Authentication is easy.

The –basic_auth option accepts user:password pairs separated by a comma. If configured, any client trying to access this Flower instance will be prompted to provide the credentials specified in this argument:

$ celery flower --basic_auth=user1:password1,user2:password2

Google OAuth 2.0¶

Flower supports Google OAuth 2.0. This way you can authenticate any user with a Google account. Google OAuth 2.0 authentication is enabled using the –auth, –oauth2_key, –oauth2_secret and –oauth2_redirect_uri options.

–auth is a regular expression, for granting access only to the specified email pattern. –oauth2_key and –oauth2_secret are your credentials from your Google Developer Console. –oauth2_redirect_uri is there to specify what is the redirect_uri associated to your key and secret

For instance, if you want to grant access to me@gmail.com and you@gmail.com:

$ celery flower --auth="me@gmail.com|you@gmail.com" --oauth2_key=... --oauth2_secret=... --oauth2_redirect_uri=http://flower.example.com/login

Alternatively, you can set environment variables instead of command line arguments:

$ export FLOWER_OAUTH2_KEY=...
$ export FLOWER_OAUTH2_SECRET=...
$ export FLOWER_OAUTH2_REDIRECT_URI=http://flower.example.com/login
$ celery flower --auth=.*@example\.com

Okta OAuth¶

Flower also supports Okta OAuth. Flower should be registered in <https://developer.okta.com/docs/guides/add-an-external-idp/openidconnect/register-app-in-okta/> before getting started. See Okta OAuth API docs for more info.

Okta OAuth should be activated using –auth_provider option. The client id, secret and redirect uri should be provided using –oauth2_key, –oauth2_secret, –oauth2_redirect_uri options or using FLOWER_OAUTH2_KEY, FLOWER_OAUTH2_SECRET, FLOWER_OAUTH2_REDIRECT_URI environment variables.

The URL from which OAuth2 API URLs will be built should be set using FLOWER_OAUTH2_OKTA_BASE_URL

environment variable:

$ export FLOWER_OAUTH2_KEY=7956724aafbf5e1a93ac
$ export FLOWER_OAUTH2_SECRET=f9155f764b7e466c445931a6e3cc7a42c4ce47be
$ export FLOWER_OAUTH2_REDIRECT_URI=http://localhost:5555/login
$ export FLOWER_OAUTH2_OKTA_BASE_URL=https://my-company.okta.com/oauth2
$ celery flower --auth_provider=flower.views.auth.OktaLoginHandler --auth=.*@example\.com

GitHub OAuth¶

Flower also supports GitHub OAuth. Flower should be registered in <https://github.com/settings/applications/new> before getting started. See GitHub OAuth API docs for more info.

GitHub OAuth should be activated using –auth_provider option. The client id, secret and redirect uri should be provided using –oauth2_key, –oauth2_secret and –oauth2_redirect_uri options or using FLOWER_OAUTH2_KEY, FLOWER_OAUTH2_SECRET and FLOWER_OAUTH2_REDIRECT_URI environment variables.

$ export FLOWER_OAUTH2_KEY=7956724aafbf5e1a93ac
$ export FLOWER_OAUTH2_SECRET=f9155f764b7e466c445931a6e3cc7a42c4ce47be
$ export FLOWER_OAUTH2_REDIRECT_URI=http://localhost:5555/login
$ celery flower --auth_provider=flower.views.auth.GithubLoginHandler --auth=.*@example\.com

GitLab OAuth¶

Flower also supports GitLab OAuth. Flower should be registered in <https://gitlab.com/profile/applications> before getting started. See GitLab OAuth2 API docs for more info.

GitLab OAuth should be activated using –auth_provider option. The client id, secret and redirect uri should be provided using –oauth2_key, –oauth2_secret and –oauth2_redirect_uri options or using FLOWER_OAUTH2_KEY, FLOWER_OAUTH2_SECRET and FLOWER_OAUTH2_REDIRECT_URI environment variables.

A list of allowed GitLab groups can be specified using the: FLOWER_GITLAB_AUTH_ALLOWED_GROUPS environment variable (e.g. group1,group2/subgroup).

The default minimum required group access level can be changes by FLOWER_GITLAB_MIN_ACCESS_LEVEL environment variable. See Group and project members API for details.

$ export FLOWER_OAUTH2_KEY=7956724aafbf5e1a93ac $ export FLOWER_OAUTH2_SECRET=f9155f764b7e466c445931a6e3cc7a42c4ce47be $ export FLOWER_OAUTH2_REDIRECT_URI=http://localhost:5555/login $ export FLOWER_GITLAB_AUTH_ALLOWED_GROUPS=group1,group2/subgroup $ celery flower –auth_provider=flower.views.auth.GitLabLoginHandler –auth=.*@example.com

Running behind reverse proxy¶

To run Flower behind a reverse proxy, remember to set the correct Host header to the request to make sure Flower can generate correct URLs. The following is a minimal nginx configuration:

server {
    listen 80;
    server_name flower.example.com;
    charset utf-8;

    location / {
        proxy_pass http://localhost:5555;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Note that you should not expose this site to the public internet without any sort of authentication! If you have a htpasswd file with user credentials you can make nginx use this file by adding the following lines to the location block:

auth_basic "Restricted";
auth_basic_user_file htpasswd;

Docker Usage¶

To run Flower via Docker, you’ll need a broker running. If you don’t have one, you can fire up a simple Redis instance with Docker from the official Redis image.

$ docker run –name localredis -p 6379:6379 –rm -d redis

Now, clone this repository, build flower from the Dockerfile, start the container and open http://localhost:49555

$ docker build -t "flower" .
$ docker run -d -p=49555:5555 --rm --name flower -e CELERY_BROKER_URL=redis://0.0.0.0:6379/0 flower flower --port=5555

For more information about running with Docker see https://docs.docker.com/

Prometheus Integration¶

Flower exports several celery worker and task metrics in Prometheus’ format. The /metrics endpoint is available from the get go after you have installed Flower.

By default on your local machine Flower’s metrics are available at: localhost:5555/metrics.

Read further for more information about configuration and available metrics please.

Complete guide on integration of Celery, Flower, Prometheus and Grafana is here: Celery -> Flower -> Prometheus -> Grafana Integration Guide.

Configure Prometheus to scrape Flower metrics¶

To integrate with Prometheus you have to add Flower as the target in Prometheus’s configuration. In this example we are assuming your Flower and Prometheus are installed on your local machine with their defaults and available at localhost:<port number>.

To add Flower’s metrics to Prometheus go to its config file prometheus.yml which initially will look like this:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

and alter the scrape_configs definition to be:

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: flower
    static_configs:
      - targets: ['localhost:5555']

You can also just point Prometheus at the example prometheus.yml file in the root of the Flower repository <https://github.com/mher/flower/prometheus.yml> when you start it from the command line (note that you would have to set flower to point at localhost in your etc/hosts config for the DNS to resolve correctly):

./prometheus --config.file=prometheus.yml

Available Metrics¶

Below you will find a table of available Prometheus metrics exposed by Flower.

Name	Description	Labels	Instrument Type
flower_events_total	Number of times a celery task event was registered by Flower.	task, type, worker	counter
flower_task_prefetch_time_seconds	The time the task spent waiting at the celery worker to be executed.	task, worker	gauge
flower_worker_prefetched_tasks	Number of tasks of given type prefetched at a worker.	task, worker	gauge
flower_task_runtime_seconds	The time it took to run the task.	task, worker	histogram
flower_worker_online	Shows celery worker’s online status.	worker	gauge
flower_worker_number_of_currently_executing_tasks	Number of tasks currently executing at this worker.	worker	gauge

Using Metric Labels¶

You can filter received data in prometheus using promql syntax to present information only for selected labels. We have the following labels available:

task - task name, i.e. tasks.add, tasks.multiply.
type - task event type, i.e. task-started, task-succeeded. Note that worker related events will not be counted. For more info on task event types see: task events in celery.
worker - celery worker name, i.e celery@<your computer name>.

Example Prometheus Alerts¶

See example Prometheus alerts. Add the rules to your alertmanager.yml config as in the alert manager’s documentation.

Example Grafana Dashboard¶

See example Grafana dashboard. You can import it easily in Grafana. Hover over the + button in the side bar menu -> Import -> Upload JSON file. The dashboard should give you a nice starting point for monitoring of your celery cluster.

Celery -> Flower -> Prometheus -> Grafana Integration Guide¶

In this guide you will learn how to setup each part of the stack to make it talk to the next one and achieve Celery monitoring solution with help of Flower.

Same as above we assume localhost usage and for ease of deployment I will use Pycharm configurations to start docker containers with necessary images. If you do not have docker installed on your system: download and install it please.

Start Celery Broker¶

Easiest is to use Redis Pycharm run configuration.

Or run:

docker run --name redis -d -p 6379:6379 redis

Set Up Your Celery Application¶

We are assuming that your Celery application has tasks in tasks.py file. The -E argument makes Celery send events which are required to produce Prometheus metrics.

Create celeryconfig.py in root of your Celery app. We are setting Celery to use Redis DB as the broker/backend in this example. Skip this if you configure your broker/backend already in another way (make sure to adjust further steps to that).

broker_url = 'redis://localhost:6379/0'
celery_result_backend = 'redis://localhost:6379/0'

Or download it from here.

Start your Celery app:

celery -A tasks worker -l INFO -E

When the app starts you should see this line:

-- ******* ---- .> task events: ON

Start Flower Monitoring¶

In your Celery application folder run this command (Flower needs to be installed):

celery flower -A tasks --broker=redis://localhost:6379/0

Configure and Start Prometheus¶

Create prometheus.yml file. Note its absolute path - we will use it to start the Prometheus docker image. For ease of use put it in the root of your Celery project (so that you can use Pycharm configuration below without any changes).

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: flower
    static_configs:
      - targets: ['localhost:5555']

Run Prometheus inside docker:

You can use Prometheus Pycharm run configuration (may need to adjust the prometheus.yml path if it is not in root of your Celery project).

Or just start it via command line:

docker run --name Prometheus -v <ABSOLUTE PATH TO YOUR prometheus.yml FILE>:/etc/prometheus/prometheus.yml -p 9090:9090 --network host prom/prometheus

Now go to localhost:9090 and check that Prometheus is running. If everything so far was set up and started correctly, you should be able to see metrics provided by Flower in your Prometheus’s GUI. Go to Graph tab and start typing flower - the autocomplete should show you all available metrics.

_images/flower-metrics-in-prometheus.png

Start Grafana¶

You can use Grafana Pycharm run configuration.

Or run it from the terminal:

docker run --name Grafana -d -v grafana-storage:/var/lib/grafana -p 3000:3000 --network host grafana/grafana

try to access its web GUI now by going to localhost:3000, use admin/admin for credentials. It will ask you to set up a new password - you may click skip for now.

Add Prometheus As a Data Source In Grafana¶

Click Configuration (settings icon) in the left side-bar. Then the blue Add data source button.

Search for Prometheus data source and click it (it should be at the top).

_images/grafana-add-prometheus-data-source.png

Once in Prometheus data source configuration, use all defaults and enter the HTTP/URL parameter as below (which is the placeholder by the way):

http://localhost:9090

_images/grafana-configure-prometheus-data-source.png

Scroll down and click Save & Test, if all is good a green banner will pop up saying Data source is working

_images/grafana-test-prometheus-data-source.png

Import The Celery Monitoring Dashboard In Grafana¶

Download Grafana dashboard.

Hover over the + icon in the left side-bar and click Import button.

Click Upload JSON file button and select the celery-monitoring-grafana-dashboard.json you have just downloaded.

_images/grafana-import-celery-monitoring-dashboard.png

Click on the Prometheus field and select a Prometheus data source.

_images/grafana-configure-imported-dashboard.png

Click Import to finish the process.

You should see a dashboard as on the image below. Congratulations!

Flower is Open Source and licensed under the BSD License.