You are visiting relatives, staying overnight. How do you leave your room once you are leaving? Or AirBnB apartment? Do you leave garbage laying around, water running, door open and stove on? Probably not. At least I hope you do not. The same should apply to your applications. Applications should shutdown gracefully (and preferably pretty fast). Graceful shutdowns are even more important nowadays when underlying orchestration service such as Kubernetes is handling the lifecycle of your app. Neglecting graceful shutdown causes a lot of headaches to operations:

• Scaling is slower
• Resources leak
• Data stores are left in inconsistent state and you might lose data
• Operations are having harder time managing the app

Follow the simple instructions layed out in this post and keep your operations smooth and operators (or your own team) happy!

Do not swallow OS signals

Your app should be PID 1 in the container. Run your application directly at the ENTRYPOINT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

FROM golang:alpine as builder
ENV USER=app
ENV UID=12345
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"
WORKDIR /app
COPY . .
RUN go mod download
RUN GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o /app/app

FROM scratch
WORKDIR /app
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
COPY --from=builder /app/app /app/app
USER app:app
EXPOSE 3000
ENTRYPOINT ["/app/app"]

Avoid shell-wrappers since those can swallow signals. If you are absolutely sure you need such tool make sure you use one that is capable of forwarding any OS signals it receives to your application. (Note: running job-management tools is not good reason. It usually indicates that you are trying to dynamically run parts of your application inside container implying multiple critical processes. That is huge antipattern. You should always have only one critical process inside your application to keep things simple and easy to reason about.)

Handle the SIGINT and SIGTERM

You should listen and act on the SIGINT and SIGTERM signals. This is the way how underlying runtime notifies your app that it is time to stop whatever application is doing, release resources and quit. Furthermore your shutdown shouldn’t take very long. Finish in-flight request, do not accept more and release any locks and connections there might be allocated.

This is minimal example with Go how you can be sure that you are playing nice with your runtime:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

const graceful_shutdown_timeout = 10

func main() {
	http.HandleFunc("/ping", handlePingPong)

	httpServer := http.Server{
		Addr: ":3000",
	}

	go func() {
		log.Printf("Starting HTTP server on port 3000")
		if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
			log.Fatalf("HTTP server ListenAndServe Error: %v", err)
		}
		log.Println("Stopped serving new connections.")
	}()

	// The graceful magic is here
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	sig := <-sigChan

	log.Printf("Received signal %s", sig)

	// Shut down server gracefully
	shutdownCtx, shutdownFunction := context.WithTimeout(context.Background(), graceful_shutdown_timeout*time.Second)
	defer shutdownFunction()

	if err := httpServer.Shutdown(shutdownCtx); err != nil {
		log.Fatalf("Shutdown error: %v", err)
	}
	log.Printf("Graceful shutdown complete.")
}

func handlePingPong(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	w.Write([]byte(`{"ping": "pong"}`))
}

Catch those signals! There is no reason to not implement proper signal handling.

Respect timeouts

When pod needs to be terminated Kubernetes sends a SIGTERM signal to the main container process (PID 1) and waits for the pod to terminate. This is the reason why you want your application to be PID 1. If the pod does not terminate within the 30-second grace period Kubernetes sends a SIGKILL signal to force termination. Note, there is no way to handle SIGKILL, your pod will die and anything you have ongoing/open is lost. Having to resort to SIGKILL should be absolute worst case scenario and you never want your pods to be killed that way.

In order to not becoming forcefully terminated your app should cleanup and quit as fast as possible. Even though the grace period can be increased you shouldn’t need to. If you have operations running inside the pod that require over 30s to complete (let alone more if you need to increase grace period) you are probably doing something that smells fishy. This is especially true for any application that servers as frontend. HTTP request running >30s? You are doing too much, capture the work, store it somewhere and return token which can be used to query job status. Do not block frontend server with long running jobs. Only place where longer running jobs are valid are backend job-queues but even then try to make jobs small so they execute quickly.

On the example above there is 10s grace period after which the application will close. Implement shutdown timeout for your app as well.