Naveen Gurram

Hi there I’m Naveen Gurram. I’m a software engineer primarily work on backend. You can read more about me in the About me section

4 minute read

Standing up own Elastic Search Cluster

Elastic Search is the most popular full text search engine out there which supports cluster with multiple nodes working together providing scalability and reliability without any additional infrastructure.

At the time of writing Amazon managed service is available but with limited functionality like encryption at rest and inter node encrypted communication was not available and they deemed to be mandatory for our solution.

We decided to stand up our own ES cluster in AWS, its not without challenges.

  • Clustered ES cluster means auto discoverable when a node goes down
  • Encrypting data at rest
  • Encrypting internode communication
  • Snapshots
  • Restoring cluster from a existing snapshot

Auto discoverable nodes

Due to auto scaling policy new nodes will be joining the cluster at any time, but that new node needs to be attached to the existing ES cluster is the challenge and it is solved with use of tags. When we bootstrap a EC2 we find other nodes with certain tags so it can be attached to a cluster.

Encrypting data at rest

Data at rest encryption is achieved using EC2 types which has ephemeral drive and using linux LUKS to encrypt the root drives. Encrypted EBS volumes is also possible except ES get good performance with ephemeral drives.

Encrypting inter-node communication

We used SearchGuard plugin with ES to help with the encryption. One more challenge we encountered was with the certificate authority. We created our own certificate authority to generate certs with our CA and use that to sign the payload. More details in plugin page.

Snapshots

We used snapshot api to store snapshot in S3. While bootstraping ES node we configure S3 key for storing snapshots.

Restoring cluster from a existing Snapshot

Again while bootstraping after registering snapshot location, cluster is rebuilt from last snapshot that way when a new node is added to the cluster we load latest snapshot and then let ES rebalance.

Assumptions

  • Every node is master eligible and marked as data node.

if you have different architecture you might have to change to suit your requirement.

Bootstrap Script

#!/bin/sh
# this script is run on new instances
# to install the required software.

# Install Elastic Search # 2.4.1 
echo "Install ElasticSearch2.4.1"
rpm -ivh --httpproxy xxx --httpport xxx https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.4.1/elasticsearch-2.4.1.rpm
echo "Instal json query package (jq)"
yum -y install jq
# register elasticsearch as service
chkconfig --add elasticsearch
# find out instanceId
INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
echo "InstanceId=$INSTANCE_ID"
# find out host_ip
HOST_IP=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} | jq .Reservations[].Instances[].PrivateIpAddress | tr -d '"')
echo "HostIp=$HOST_IP"
# find out role from ec2-tags
ROLE_NAME=$(cat /etc/aws-tags.conf | grep 'Role=' | awk -F\= '{print $2}')
echo "RoleName=$ROLE_NAME"
# find out all other ips in that role which are in running state
aws ec2 describe-instances --filters Name=tag:Role,Values=${ROLE_NAME} | jq '.Reservations[].Instances[] | {status: .State.Name, IP: .PrivateIpAddress} | select(.status == "running") | .IP' | tr -d '"' >> ipsInAgs.txt
# now do a loop on ipsInAgs.txt file and they are the nodes in the cluster.
# XM26q.nJ5{`M^Yq] 
UNICAST_HOSTS=""
while read p; do
  if [ $p != $HOST_IP ]; then
     if [ ${#UNICAST_HOSTS} != 0 ]; then
        UNICAST_HOSTS="$UNICAST_HOSTS ," 
     fi 
     UNICAST_HOSTS="$UNICAST_HOSTS \"$p\"" 
  fi  
done <ipsInAgs.txt

# Create secured mount disk 
echo $RANDOM > /root/passphrase

if [ -e /dev/xvdb ]; then
    #mkfs -t ext4 /dev/xvdb
    #mount /dev/xvdb /app
    # LUKS encryption
    echo Setting up LUKS on /dev/xvdb
    cryptsetup luksFormat --key-file /root/passphrase /dev/xvdb -q
    cryptsetup luksOpen /dev/xvdb --key-file /root/passphrase app
elif [ -e /dev/xvdc ]; then
    echo Setting up LUKS on /dev/xvdc
    cryptsetup luksFormat --key-file /root/passphrase /dev/xvdc -q
    cryptsetup luksOpen /dev/xvdc --key-file /root/passphrase app
fi

if [ -e /dev/mapper/app ]; then
    echo Formatting and mounting LUKS device on /app
    mkfs.ext4 -m 0 /dev/mapper/app
    mount /dev/mapper/app /app
fi


# create directories for logs and data
mkdir -p /app/proj/data /app/proj/logs
# change owner and group to elasticsearch 
chown elasticsearch:elasticsearch -R /app/proj/data /app/proj/logs
# install elastic search
cd /tmp
wget https://xxxx/artifactory/repo1-cache/com/floragunn/search-guard-ssl/2.4.1.16/search-guard-ssl-2.4.1.16.zip

echo "Y" | /usr/share/elasticsearch/bin/plugin install file:///tmp/search-guard-ssl-2.4.1.16.zip


# create self signed certs
cp /staging/ca.tar.gz /tmp

tar xzvf /tmp/ca.tar.gz -C /tmp
chown root:root -R /tmp/ca/
chmod 755 -R /tmp/ca
chmod +x /tmp/ca/gen_node_cert.sh
# generate cert
cd /tmp/ca
/bin/bash gen_node_cert.sh changeit capass

cp "/tmp/ca/$HOSTNAME-keystore.jks" /etc/elasticsearch/keystore.jks

# trust store
cp /tmp/ca/truststore.jks /etc/elasticsearch/

# take a backup of elasticsearch.yml template
cp /staging/elasticsearch.yml /staging/elasticsearch.yml.bkp

sed -i -e "s/UNICAST_HOSTS/${UNICAST_HOSTS}/g" /staging/elasticsearch.yml
sed -i -e "s/HOST_IP/${HOST_IP}/g" /staging/elasticsearch.yml
sed -i -e "s/ROLE_NAME/${ROLE_NAME}/g" /staging/elasticsearch.yml

cat /staging/elasticsearch.yml

chown root:elasticsearch /etc/elasticsearch/*.jks
chmod 750 /etc/elasticsearch/*.jks

# copy elasticsearch to /etc/elasticsearch folder
cp /staging/elasticsearch.yml /etc/elasticsearch/
cp /staging/logging.yml /etc/elasticsearch/

# increase memlock limit 
echo "#increasing memlock limit to unlimited" >> /etc/security/limits.conf
echo "elasticsearch soft memlock unlimited" >> /etc/security/limits.conf
echo "elasticsearch hard memlock unlimited" >> /etc/security/limits.conf

# find out how much memory is there and we want to assign half of it, and not more than 32g
TOTAL_MEMORY=$(awk '/MemTotal/ {print int(($2/1024/1024/2)+1)}' /proc/meminfo)
echo "Total Available memory $TOTAL_MEMORY"
# not more than 32 g
if [ $TOTAL_MEMORY -gt 32 ]; then
   TOTAL_MEMORY=32
fi

# set es_heap
echo "ES_HEAP_SIZE=${TOTAL_MEMORY}g " >> /etc/sysconfig/elasticsearch
echo "MAX_LOCKED_MEMORY=unlimited " >> /etc/sysconfig/elasticsearch

# start elasticsearch
service elasticsearch start


exit 0

Conclusion

Elastic Search itself has rich feature set which allowed to configure to our needs. I hope this is helpful for people who wants to build their own ES cluster.

You May Also Enjoy

Calling Graphql mutation using WebClient

1 minute read

I had a requirement to call a graphql mutation endpoint from java and even with multiple graphql tools, requirement was to use a standard REST client to make...

How I used Intellij HttpClient CLI in Jenkins

1 minute read

Like many organizations we are into building scalable microservices in cloud, and we have variety of protocols like REST, Graphql etc using both server and s...

Augment Spring boot app with external jars

2 minute read

Spring Boot packaging to run as a standalone jar has it advantages, like you can pack all the dependencies into single uber jar and run it on the command lin...