YALP

Distributed log parsing and collection.

YALP is a log parsing pipeline written in python. It utilized Celery for stable and scalable distributed processing, is easy to configure, and customize and extend.

Install and Configure

Brief install guide:

$ sudo apt-get install rabbitmq-server mongodb
$ virtualenv /srv/yalp_env
$ source /srv/yalp_env/bin/activate
(yalp_env) $ pip install yalp

/srv/yalp.yml:

# Celery configuration
broker_url: amqp://guest:guest@localhost:5672//
inputs:
  - 'file':
      path: '/var/log/syslog'
      type: messages
parsers:
  - 'passthrough':
      type: messages
outputs:
  - 'mongo':
      uri: 'mongodb://localhost:27017/yalp'
      database: yalp
      collection: logs

(yalp_env) $ yalp-inputs -c /srv/yalp.yml
(yalp_env) $ yalp-parsers -c /srv/yalp.yml
(yalp_env) $ yalp-outputs -c /srv/yalp.yml

Full Installation Guide