YALP¶
Distributed log parsing and collection.
YALP is a log parsing pipeline written in python. It utilized Celery for stable and scalable distributed processing, is easy to configure, and customize and extend.
Install and Configure¶
Brief install guide:
$ sudo apt-get install rabbitmq-server mongodb
$ virtualenv /srv/yalp_env
$ source /srv/yalp_env/bin/activate
(yalp_env) $ pip install yalp
/srv/yalp.yml
:
# Celery configuration
broker_url: amqp://guest:guest@localhost:5672//
inputs:
- 'file':
path: '/var/log/syslog'
type: messages
parsers:
- 'passthrough':
type: messages
outputs:
- 'mongo':
uri: 'mongodb://localhost:27017/yalp'
database: yalp
collection: logs
(yalp_env) $ yalp-inputs -c /srv/yalp.yml
(yalp_env) $ yalp-parsers -c /srv/yalp.yml
(yalp_env) $ yalp-outputs -c /srv/yalp.yml