Shipping your Nginx logs to Elasticsearch using Logstash

First of all create a new log format in /etc/nginx/nginx.conf and add/replace an access log directive: log_format logstash ‘$http_host ‘ ‘$remote_addr [$time_local] ‘ ‘”$request” $status $body_bytes_sent ‘ ‘”$http_referer” “$http_user_agent” ‘ ‘$request_time ‘ ‘$upstream_response_time’; access_log /var/log/nginx/access.log logstash; Install Elasticsearch like described here. Install Logstash like described here. Add the following pattern to logstash in /opt/logstash/paggerns/nginx: NGUSERNAME […]

Setting up Logstash on Debian/Ubuntu

Install Logstash: cd /tmp wget tar -xvpf logstash-1.4.2.tar.gz mv logstash-1.4.2 /opt/logstash Create an init script /etc/init.d/logstash (this one worked fine for me on Ubuntu 14.04): #! /bin/sh ### BEGIN INIT INFO # Provides: logstash # Required-Start: $network $remote_fs $named $elasticsearch # Required-Stop: $network $remote_fs $named $elasticsearch # Default-Start: 2 3 4 5 # Default-Stop: […]

Install Elasticsearch 1.4 on Debian/Ubuntu

Install Java: apt-get install openjdk-7-jre openjdk-7-jre-headless Add the repository and install Elasticsearch: wget -qO – | sudo apt-key add – add-apt-repository "deb stable main" apt-get update apt-get install elasticsearch Edit /etc/default/elasticsearch and change ES_HEAP_SIZE to limit memory usage depending on your machine. Start Elasticsearch: /etc/init.d/elasticsearch start Configure Elasticsearch to automatically start during bootup : […]

Installing Nginx 1.7.9 with Google Pagespeed on Ubuntu 14.04 (Trusty)

Basically we're going to rebuild .deb packages, adding the Pagespeed module. So first we're adding the repo and preparing some dependencies: add-apt-repository -s -y ppa:nginx/development apt-get update apt-get -y install devscripts apt-get -y build-dep nginx=1.7.9   We're building the packages using an unprivileged user: adduser bob −−shell /bin/bash \ –disabled-password –gecos bob sudo su – […]

InnoDB Error Table mysql.innodb_table_stats not found

Follow these steps to recover from this innodb error "InnoDB: Error: Table mysql.innodb_table_stats not found.". First try to drop these tables from a mysql console: USE mysql; DROP TABLE innodb_index_stats; DROP TABLE innodb_table_stats; DROP TABLE slave_master_info; DROP TABLE slave_relay_log_info; DROP TABLE slave_worker_info; This may throw some errors. If your mysql data folder is somewhere different than the […]

Install Nginx 1.6 on Ubuntu 14.04 Trusty

If you are missing the command add-apt-repository, have a look here. To install Nginx then on Ubuntu, run these as root: add-apt-repository ppa:nginx/stable apt-get update apt-get install nginx

Installing a 3 node Zookeeper Ensemble

First install the packages: apt-get install zookeeper zookeeper-bin zookeeperd As Zookeeper is launched using Java, you may want to update the Java options in /etc/default/zookeeper, for example: JAVA_OPTS="-Xms1024M -Xmx1024M -noverify -Duser.timezone=GMT+1" This would assign 1GB of RAM to the JVM for Zookeeper and set the timezone in Java to GMT+1. There are more options, this […]

Installing Redis-CLI without installing the whole server

On Debian actually there is only one package redis-server in order to install the server and its tools. If you would like to have the client only, you can achieve this as follows. As a prerequesite you need to install these packages: apt-get install libjemalloc1 libjemalloc-dev gcc make Then proceed with the actual installation: cd /tmp wget […]

Installing Redis 2.8 Master/Slave on Debian Wheezy

First activate the dotdeb repository: echo "deb wheezy all" > /etc/apt/sources.list.d/dotdeb.list echo "deb-src wheezy all" >> /etc/apt/sources.list.d/dotdeb.list cd /tmp wget apt-key add dotdeb.gpg apt-get update apt-get install redis-server Here are a few lines that you might want to have a look at in /etc/redis/redis.conf: If redis is supposed to be reachable only […]