Nomad allocations, groups and tasks explained!

Code samples are here

When you first start working with Hashicorp’s Nomad it can be challenging to face a lot of new terminology all at once.

Allocations? Jobs? Groups? Tasks? What do these all mean? And how do they relate to docker containers?

Before we start, here is a list of software versions used in this post:

• Docker 24
• Hashicorp Nomad 1.9.2 (standalone or cluster)

Tasks

Let’s look at this very simple Nomad job:

job "basic-job" {
  group "service" {
    count = 1
    task "service" {
      driver = "docker"
      config {
        image = "davidlublink/devopsgeneration:pretend-work"
      }
    }
  }
}

Starting from the task, we see we invoked the docker driver. In this stanza, we are telling Nomad to create a new docker container with the image provided.

In this case, since there is only one task defined and there is no count set on the group stanza, the group, task, and job all mean the same thing.

A task really is the smallest unit of work inside of your Nomad job!

Groups

Groups, also called task groups, are primarily used for segmenting your multi-task Nomad job into logical atomic units of work that are deployed based on the requirements of the job writer

job "fourth-job" {
  group "service1" {
    count = 1
    task "service" {
      driver = "docker"
      config {
        image = "davidlublink/devopsgeneration:pretend-work"
      }
      env {
        THE_VAR="the values you are looking for"
      }
    }

    task "service2" {
      driver = "docker"
      config {
        image = "davidlublink/devopsgeneration:pretend-work"
      }
      env {
        THE_VAR="not the values you are looking for"
      }
    }
  }
}

Fourth-job contains 2 tasks within the same group. The group tells Nomad that the two tasks must be deployed *together* as a group. This means that both tasks are deployed on the same Nomad client together. If either task fails, it’ll take down the entire group. When scaling, both tasks will scale together(1 of each, 5 of each, 100 of each ).

A big advantage of having 2 tasks in the same group is the ability to share resources such as storage(/alloc/data is shared), network(bridge/cni modes share network with entire group), etc…

Now consider this job, second-job:

job "second-job" {
  group "service1" {
    count=1
    task "service" {
      driver = "docker"
      config {
        image = "davidlublink/devopsgeneration:pretend-work"
      }
      env {
        THE_VAR="the values you are looking for"
      }
    }
  }
  group "service2" {
    count = 1
    task "service" {
      driver = "docker"
      config {
        image = "davidlublink/devopsgeneration:pretend-work"
      }
      env {
        THE_VAR="not the values you are looking for"
      }
    }
  }
}

Each task is in it’s own group, and this tells Nomad to deploy each group separately. If one group fails, the other group is unaffected. Scaling happens at the group level, so the two services can be deployed on the same or different Nomad nodes. Of course with separate groups, we lose the ability to easily share resources.

	Shared group	Separate groups
File sharing	Shared /alloc/data/	Added layer of complexity to share files. CSI or other plugin needed
Network sharing	Bridge: Yes CNI: Yes	Added layer of complexity to share networking
Failure handling	Failed task brings down group, one failed tasks causes all other in the group to restart	Tasks are separated, if one fails, the other keeps going.
Deployment strategy	All tasks within the group must be allocated together on the same Nomad client	Nomad can deploy each group as it sees fit, possibly on the same Nomad client, possibly different
Scaling	All tasks scale together, with 2 tasks in the group and count=5, that’s 10 tasks running	Each group scales separately, so if one has count 7 the other has count 4, that’s 11 tasks
Change	Any change that causes any task within the group to need to be destroyed/created causes the ENTIRE group to be destroyed/created	Each group is destroyed/created individually, so a change in one group doesn’t affect the other

Job

Think of a job as 1 or more groups, there isn’t really any limit on the number of groups in the same job, though it may become cumbersome to manage a job with too many groups!

Allocations

Allocation is the term used by Nomad to refer to an instance of a group that is deployed/being deployed/dead.

Many different events can cause allocations to be destroyed and replaced, changes to the job definition it self, unstable service, operator intervention etc…

Consider this screenshot from the web UI:

You can see I am looking at the group ‘service1’, there have been 2 allocations. One is complete and the other currently running.

Conclusion

In the web UI, the breadcrumb shows organization of the different concepts here.

Job(fourth-job) is first since it is what contains different groups.

Next we have ‘Task Group’ which we see is service1.

Allocation d45dbb6d is the allocation of service1 task group that was being inspected.

Finally we have the task itself, service, and it is the basic unit of work.