Supervise internals

Sometimes you need somebody to look after your process :)

supervisor

In python2 world it's common tool to use supervisor to look after your application processes.

Supervisor in version 4.0 will support python 3.

Supervisor consists of daemon - called supervisord and control application - called supervisorctl which talks to daemon over socket.

Some supervisor configuration file looks like that:

[supervisord]
logfile = /var/log/python-application/supervisor/supervisord.log
logfile_maxbytes = 10MB
logfile_backups = 10
loglevel = info
pidfile = /tmp/python-application-supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200

[unix_http_server]
file = /tmp/python-application-supervisord.sock
chmod = 0770

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl = unix:///tmp/python-application-supervisord.sock

[group:pyapps]
programs=gunicorn,celery

[program:gunicorn]
directory = /opt/python-application/src
environment = DJANGO_SETTINGS_MODULE="settings.production"
command = gunicorn wsgi:application -b 0.0.0.0:%(ENV_APPLICATION_PORT)s --reload
stdout_logfile = /var/log/python-application/supervisor/gunicorn.log
stderr_logfile = /var/log/python-application/supervisor/gunicorn.log
autostart = True
autorestart = True
stopsignal = KILL
stopasgroup = true

[program:celery]
directory = /opt/python-application/src
environment = DJANGO_SETTINGS_MODULE="settings.production",C_FORCE_ROOT=true
command = celery -A pyapp worker -c 1 -E
stdout_logfile = /var/log/python-application/supervisor/celery.log
stderr_logfile = /var/log/python-application/supervisor/celery.log
autostart = True
autorestart = True
stopsignal = INT
stopwaitsecs = 60
stopasgroup = true

To connect to daemon with that configuration you should run:

supervisorctl -c configuration_file.ini

Now you can start, stop, restart, reload programs or all groups of programs. When you change configuration file you should run reread command to reload configuration on daemon side.

Configuration describes group of processes called pyapps which contains two applications or programs: gunicorn and celery - this names can be different than running command, but this way it’s more clear. Every program has directory with source code, command which supervisor should run when starting application. You can define stdout and stderr log files. You can autostart program when it crashes. You can declare some environment variables (comma separated). One unusual entry: %(ENV_APPLICATION_PORT)s allows you to load environment variable named APPLICATION_PORT and substitute it in supervisor configuration file.

circus

Supervisor currently doesn't work with python3 so you need some other tool like circus from mozilla to deal with.

Like supervisor, circus has two main parts: daemon - circusd and control application - circusctl.

Some example configuration file for circus:

[circus]
debug = True

[watcher:gunicorn]
cmd = gunicorn wsgi:application -t 1800 -b 0.0.0.0:80 --reload --threads=3
numprocesses = 1
working_dir = /opt/python-application/src
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/python-application/circus/gunicorn.log
stdout_stream.refresh_time = 0.5
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/python-application/circus/gunicorn.log
stderr_stream.refresh_time = 0.5

To control circus daemon you use circusctl command.

Cookbooks debugging in kitchen

Sometimes you just need debug your cookbooks

chef-shell

Easiest way to debug execution of your cookbook is using chef-shell.

To do so you run chef-zero as daemon on provisioned kitchen instance (note: every command runs from /tmp/kitchen directory):

root@ubuntu-1404:/tmp/kitchen# /opt/chef/embedded/bin/chef-zero -d

Then you need to upload your cookbook to new instance of chef-zero:

root@ubuntu-1404:/tmp/kitchen# knife cookbook upload -a -c client.rb
Uploading debugbook    [0.1.0]
Uploaded all cookbooks.

Then just run chef-shell:

root@ubuntu-1404:/tmp/kitchen# chef-shell -z -c client.rb -o 'debugbook'
loading configuration: client.rb
Session type: client
Loading..[2016-08-24T23:12:35+00:00] WARN: Run List override has been provided.
[2016-08-24T23:12:35+00:00] WARN: Original Run List: []
[2016-08-24T23:12:35+00:00] WARN: Overridden Run List: [recipe[debugbook]]
resolving cookbooks for run list: ["debugbook"]
.Synchronizing Cookbooks:
  - debugbook (0.1.0)
done.

This is the chef-shell.
 Chef Version: 12.13.37
 http://www.chef.io/
 http://docs.chef.io/

run `help' for help, `exit' or ^D to quit.

Ohai2u vagrant@default-ubuntu-1404!
chef (12.13.37)>

pry-remote

Or you can use harder way with pry-remote.

Define private network and port forwarding in kitchen so we can connect to pry server. In .kitchen.yml for your cookbook add network parameters:

driver:
  name: vagrant
  network:
    - ['private_network', {ip: '33.33.33.33'}]
    - ['forwarded_port', {guest: 9876, host: 9876}]

In your recipe add directives to install pry-remote and set break point:

chef_gem 'pry-remote'
require 'pry-remote'

binding.remote_pry '0.0.0.0'

Run your kitchen converge or test and you get something like this:

[...]
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Recipe: debugbook::default
         * chef_gem[pry-remote] action install (up to date)
       [pry-remote] Waiting for client on druby://0.0.0.0:9876

Now in other shell just run:

pry-remote -s 33.33.33.33 -p 9876

And you’re connected to pry session.

Example cookbook can be find here: debugbook

Creating decoder for heka

Extending Heka with lua encoders

tbd

Source:

Chef HA

Chef HA

tbd

Source:

Spell Checking in Sublime Text

I need it so TIL

I thought that it’d be nice to have spell checking in sublime for writing this posts in English. So Sublime Text uses Hunspell for spell checking and you need to install it in your os:

sudo apt-get install hunspell

Let’s say that i want two languages: Polish and English (us & gb):

Polish dictionary can be found in repository by titoBouzout or you can download some from openoffice.

Create dir in ~/.config/sublime-text-3/Packages/Language - Polish - for my example it was ubuntu, on mac osx it could be: ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/Language - Polish. Put your dictionary (i want extra Polish.dic) there.

After that just add those lines to your sublime text user settings:

{
    "dictionary": "Packages/Language - English/en_US.dic",
    "dictionary": "Packages/Language - Polish/Polish.dic",
    "spell_check": true
}

To change language in sublime-text: View > Dictionary > Language - English > en_US

Source:

spell_checking 4 sublime-text