Qbox to join forces with Instaclustr. Read about it on our blog post here.

Elasticsearch recently published two new releases, and both are now available for Qbox users to upgrade their clusters. Version 1.6.1 contains bug fixes and small enhancements. Along with some security fixes, release 1.7.0 adds two new features to Elasticsearch: delayed shard allocation and prioritization of index recovery. We recommend that you upgrade as soon as you can accommodate it in your environment.

Qbox customers can contact support to request an upgrade to either release. Read this article for a distillation of the release notes and the benefits of these upgrades.

On July 16, Elastic published two releases for Elasticsearch, Versions 1.6.1 and 1.7.0. Version 1.6.1 contains a number of bug fixes.

Both Elasticsearch 1.6.1 and 1.7.0 include these two security fixes:

  • Remote code execution vulnerability — Elasticsearch versions prior to 1.6.1 are vulnerable to attacks on its transport protocol in which remote code execution is enabled. This protocol facilitates communication between ES nodes and Java clients. This security fix mitigates this vulnerability.
  • Directory traversal vulnerability — Another security fix address the fact that most previous Elasticsearch versions are also vulnerable to a directory traversal attack that permits retrieval of files that are accessible by the Elasticsearch JVM process. Users who not ready to upgrade to this release may use a firewall, reverse proxy, or Shield to block Snapshot-Restore API calls from unknown sources.

In addition to the security fixes, Version 1.7.0 also adds two new features to Elasticsearch: delayed shard allocation and prioritization of index recovery. To help you see the need for these upgrades, we revise and clarify the feature overviews from the release notes below.

Delayed Shard Allocation

Elasticsearch 1.6.0 introduced synced flushing to dramatically speed up the recovery of idle shards when restarting a node. This change functions well only in controlled environments in which shard allocation is disabled. Of course, there’s no benefit when a node disconnects from the cluster or when it incurs an surprise reboot.

Consider this scenario:

  1. A node suddenly shuts down.
  2. The master node reassigns the shards of this node to other nodes.
  3. A copy of each shard goes over the network to their new locations.
  4. The downed node simultaneously attempts to rejoin the cluster.
  5. The master redistributes shards to take advantage of the previously downed node, which is now seen as a new node. Potentially, none of the existing shards on the new node are reused.

Even though we throttle concurrent recoveries both at the node level and at the cluster level, this reshuffling of shards is likely to induce strain on the cluster. This is quite avoidable if the cluster is made to wait for the previously downed node to rejoin.

In ES 1.7.0, users can now specify the index.unassigned.node_left.delayed_timeout setting, which defaults to one minute. When a node departs from the cluster, the master node will wait for one minute before reallocating the orphan shards to other nodes. If the node returns within the timeout interval, the master will recognize and reallocate its shards (that already reside on that node).

A duration of one minute should be sufficient for a node to recover and rejoin the cluster, but it still means that reallocation will proceed if the node doesn’t rejoin. You may choose to modify this setting, according to how frequently you monitor your cluster.

For new indices, users can specify a default value for this setting in the config/elasticsearch.yml file, but you can update it dynamically on live indices using the update index settings API.

Prioritization of Index Recovery

Release 1.7.0 also adds the capability to prioritize the recovery order for indices.

Consider a power failure takes your entire logging cluster down. As the cluster recovers and reinstates hundreds of indices, the recovery order is automatic. However, you may have an urgent interest in accessing an index that is far down in the automatic priority that ES assigns during the recovery operation. Of course, you cannot reindex until the most recent index recovers.

In 1.7.0, index recovery is done with prioritization, according to these specifications:

  • The optional index.priority setting — a higher value indicates higher priority.
  • The index creation date — later is given a higher priority.
  • The index name — alphabetical ranking.

Without making any changes to your cluster, recent indices will recover before older ones. If you find it necessary to increment the priority of an older index, simply set the dynamic index.priority setting to a number higher than zero.

To get all the details, read more in the Elastic announcement.

Helpful Resources from the Qbox Blog

Have a look at these other Qbox resources that can help you optimize your work with Elasticsearch: