server : refactor middleware and /health endpoint

[ngxson]

/health endpoint

In the beginning, /health endpoint was used to retrieve slots state. That was because at the time, /completions endpoint returns an error if there is no slot available. Therefore, /health was used to allow the application to wait until one slot is available.

Nowadays, the server now can queue (defer) the request if no slots is available. /health is used by docker for health checking. This is now become a problem when the server is busy doing a long task, /health can timeout. On HF inference endpoint, this causes the container to be in unhealthy state, which triggers a force restart.

Therefore, I propose a cleaner usage:

• GET /health is now purely used to report actual health
• GET /slots can be used as a replacement to get slot state

As a consequence, /health?fail_on_no_slot=1 is also moved to /slots?fail_on_no_slot=1 (for compatibility, we keep this option)

Refactor middleware

Some repeated code blocks, for example setting Access-Control-Allow-Origin, is now moved to middleware.

Middleware now also responsible to return error if the server is not yet ready:

When the server starts, if the model is being loaded, accessing to any endpoint will result in 503 error code:

Behavior on loading model failed

If model fails to load (for example, file does not exist), the server will simply exit with status code 1. This resolves #7787 where user reports that loading invalid model causes the server to crash.

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[mcharytoniuk]

I started a discussion thread related to this issue, please take a look: #9276