In the first of our series on designing the Clarifai API we mentioned our transition from Django, to Goji, to grpc-gateway. Today, we would love to share some of what we have learned by running grpc-gateway in production.
Grpc-gateway, a server for gRPC and RESTful styles
Grpc-gateway is a server that routes HTTP/1.1 request with JSON bodies to gRPC handlers with protobuf bodies. This means you can define your entire API as gRPC methods with requests and responses defined as protobuf, and then implement those gRPC handlers on your API server. Grpc-gateway can then transform your gRPC service into a RESTful API.
Even better, you can mix the gRPC and HTTP requests into the same hostport of your server. We will walk through a simple API server setup that you can try for yourself (so that you don't have to spend three months banging your head trying to get things right).
First, let’s define a simple gRPC method called TestRPC that we will implement in our API server:
| syntax = "proto3"; | |
| import "google/api/annotations.proto"; | |
| // Here we tell the proto compiler that we want the results accessible in an "api" package. | |
| option go_package = "api"; | |
| // This will be our request. | |
| message TestRequest { | |
| // A string that we will use in the url path. | |
| string url_path_field = 1; | |
| // A integer in the body of the request. | |
| int32 body_field = 2; | |
| // Request with a nested message. | |
| NestedMessage nested = 3; | |
| } | |
| // An example of a nested message. | |
| message NestedMessage { | |
| // Another field. | |
| string something = 1; | |
| } | |
| // This will be our response. | |
| message TestResponse { | |
| // A string that we will use in the url path. | |
| string url_path_echo = 1; | |
| // A integer in the body of the request. | |
| int32 body_echo = 2; | |
| // Respond with the nested message. | |
| NestedMessage nested_response = 3; | |
| } | |
| // Here is the overall service where we define all our endpoints. | |
| service V2 { | |
| // Here is our TestRPC method to use as an example. | |
| // It wi | |
| rpc TestRPC (TestMessage) returns (TestMessage) { | |
| option (google.api.http) = { | |
| post: "/v2/test/{url_path_field}" | |
| additional_bindings { | |
| post: "/v2/test/another/{url_path_field}" | |
| body: "*" | |
| } | |
| }; | |
| } | |
| } | |
You’ll see some atypical fields in the .proto that are leveraged by grpc-gateway. One of the most important of these fields is the option (google.api.http) where we define what HTTP URL(s) will be used to handle our request. We also specify POST as the HTTP method we will accept. Finally, if you have a request body that you expect (typical for POST requests and others), you must use the body field. If you don’t, then the request won’t be passed along to the handler.
In this example, we show a few of the features of protobufs and grpc-gateway:
- You can use fields from the request proto (in this example TestRequest) in the URL template so that multiple URLs can be parsed together.
- You can type the fields easily.
- You can share objects between multiple requests and responses to make it an object-oriented interface.
In this example we will simply echo back the request fields to the response, so TestResponse is setup with similar fields to the request. There is in-depth documentation for the google.api.http proto here which is very helpful in understanding the various options and how handling happens with grpc-gateway.
Getting started with protos in golang
To use the protos in golang, you need to compile them into their generated code. The easiest way to get the protoc compiler we’ve found is to use Python’s pip installer:
pip install grpcio
Installation for the grpc-gateway in golang is done using:
go get github.com/grpc-ecosystem/grpc-gateway
Then you can compile your proto files with a command-line something like:
| python -m grpc.tools.protoc \ | |
| -grpc-gateway_out=allow_delete_body=true,request_context=true,logtostderr=true:$GOSOURCEDIR \ | |
| service.proto |
This will result in two new files:
example.pb.go
example.pb.gw.go
The first being the typical golang compiled proto and the second being the grpc-gateway specific compile proto which will contain all the translation layers between HTTP and gRPC that grpc-gateway uses.
The main.go file
Next, let’s look at the example main.go file that will serve on a given port and listening at the endpoint we defined above.
At the top of the file, we define the handlers for our gRPC methods. Typically you would do this in a different package, but for simplicity, it was done here.
| package server | |
| import ( | |
| "context" | |
| "fmt" | |
| "net" | |
| "os" | |
| "clarifai/proto/clarifai/api" | |
| "github.com/cockroachdb/cmux" | |
| "github.com/grpc-ecosystem/grpc-gateway/runtime" | |
| "github.com/zenazn/goji/bind" | |
| "github.com/zenazn/goji/graceful" | |
| "google.golang.org/grpc" | |
| ) | |
| // This is the struct that we will implement all the handlers on. | |
| type ServiceHandlers struct { | |
| } | |
| var ( | |
| // Here we enforce that the ServiceHandler implements all the methods of our compiled api.V2Server. | |
| H api.V2Server = &ServiceHandlers{} | |
| ) | |
| func (m *ServiceHandlers) TestRPC(ctx context.Context, req *api.TestRequest) (*api.TestResponse, error) { | |
| // This is where you implement the handler for the TestRPC method. | |
| // Nested messages in protos are pointers, so let's make sure something was sent in a request. | |
| if req.Nested != nil { | |
| fmt.Println(req.Nested.Something) | |
| } | |
| // Here we just echo back the request fields in the response to show how it works. | |
| return &api.TestResponse{ | |
| UrlPathEcho: req.UrlPathField, | |
| BodyEcho: req.BodyField, | |
| NestedResponse: req.Nested, | |
| }, nil | |
| } | |
| func ServeAndWait(handlers *ServiceHandlers, hostport string) { | |
| // Start by setting up a port. | |
| l, err := net.Listen("tcp", hostport) | |
| if err != nil { | |
| panic(err) | |
| } | |
| // Create the cmux object that will multiplex 2 protocols on the same port. | |
| // The two following listeners will be served on the same port below gracefully. | |
| m := cmux.New(l) | |
| // Match gRPC requests here | |
| grpcL := m.Match(cmux.HTTP2HeaderField("content-type", "application/grpc")) | |
| // Otherwise match regular http requests. | |
| httpL := m.Match(cmux.Any()) | |
| // Now create the grpc server with those options. | |
| gSvrOpts := setupGrpcServerOptions() | |
| grpcS := grpc.NewServer(gSvrOpts...) | |
| // The api package is our compiled proto package (proto/clarifai/api) which we now register so that grpc knows | |
| // That the handlers are implemented on "handlers". | |
| api.RegisterV2Server(grpcS, handlers) | |
| // This is where you register the handlers for HTTP requests through grpc-gateway. | |
| svrMuxOpts := setupServeMuxOptions() | |
| // Now make a new grpc-gateway server mux to handle the HTTP requests. | |
| grpcGatewayMux := runtime.NewServeMux(svrMuxOpts...) | |
| dialOpts := setupGrpcDialOptions() | |
| // Now leveraging the compiled grpc-gateway output (also in the proto/clarifai/api package) we register the grpc handlers | |
| // Which handles the translation from HTTP requests to calling grpc at the port specified. This uses a grpc client connection | |
| // The compiled code in RegisterV2HandlerFromEndpoint handles each endpoint defined in the .proto API spec. | |
| if err := api.RegisterV2HandlerFromEndpoint(ctx, grpcGatewayMux, port, dialOpts); err != nil { | |
| panic(err) | |
| } | |
| // Finally, tell the grpcGatewayMux that now knows about our endpoints to route any traffic over HTTP | |
| // From "/" down to those endpoints. | |
| httpMux := runtime.ServeMux{} | |
| httpMux.Handle("/", grpcGatewayMux) | |
| // Here we handle some signals so that we can stop the server when running. | |
| graceful.HandleSignals() | |
| // Handle gracefully stopping the server on signals. | |
| var gracefullyStopped bool | |
| bind.Ready() | |
| graceful.PreHook(func() { | |
| gracefullyStopped = true | |
| fmt.Println("Server received signal, gracefully stopping.") | |
| }) | |
| graceful.PostHook(func() { | |
| fmt.Println("Server stopped") | |
| }) | |
| // Collect the exits of each protocol's .Serve() call on this channel | |
| eps := make(chan error, 2) | |
| // Start the listeners for each protocol | |
| go func() { eps <- grpcS.Serve(grpcL) }() | |
| // We use graceful as the server here to serve normal mux | |
| go func() { eps <- graceful.Serve(httpL, httpMux) }() | |
| // cmux starts all the servers for us when we call Serve() (grpcS and httpS) | |
| fmt.Printf("listening and serving (multiplexed) on: %d\n", port) | |
| err = m.Serve() | |
| // The rest of the code handles exit errors of the muxes | |
| var failed bool | |
| if err != nil { | |
| // cmux server error | |
| if !gracefullyStopped { | |
| fmt.Println(err) | |
| failed = true | |
| } | |
| } | |
| // Handle exiting like they do here: https://github.com/gdm85/grpc-go-multiplex/blob/master/greeter_multiplex_server/greeter_multiplex_server.go | |
| var i int | |
| for err := range eps { | |
| if err != nil { | |
| // Protocol server error | |
| if !gracefullyStopped { | |
| fmt.Println(err) | |
| failed = true | |
| } | |
| } | |
| i++ | |
| if i == cap(eps) { | |
| close(eps) | |
| break | |
| } | |
| } | |
| if failed { | |
| os.Exit(1) | |
| } | |
| graceful.Wait() | |
| } | |
| func main() { | |
| // H is a struct with implementations of the handlers for our service. | |
| // We will start a server around it. | |
| hostport := "0.0.0.0:8000" | |
| ServeAndWait(H, hostport) | |
| } | |
| func setupGrpcServerOptions() []grpc.ServerOption { | |
| // This is where you can setup custom options for the grpc server | |
| // https://godoc.org/google.golang.org/grpc#ServerOption | |
| return nil | |
| } | |
| func setupGrpcDialOptions() *[]grpc.DialOption { | |
| // This is where you can set up your dial options. | |
| // https://godoc.org/google.golang.org/grpc#DialOption | |
| return nil | |
| } | |
| func setupServeMuxOptions() []runtime.ServeMuxOption { | |
| // https://godoc.org/github.com/grpc-ecosystem/grpc-gateway/runtime#ServeMuxOption | |
| return nil | |
| } | |
Here we listen on a port and then pass that listener into cmux which is a high-performance mux written by Cockroach Labs. This cmux allows us to register more than one type of handler. From here we register both the gRPC handler to accept binary gRPC requests directly and the HTTP handler to allow grpc-gateway to translate HTTP requests to gRPC for us.
We also show how to gracefully stop the process, so that if a signal is received by your server, you can do some cleanup before exiting. Finally, each of the two listeners are started using goroutines, and then errors are handled from the channel that is created.
Two years and counting
This type of server has been handling our requests for over two years behind an nginx proxy and has been great for many reasons:
-
Our API is fully defined in protocal buffers which are easy to read, extensible and object oriented.
-
We support HTTP RESTful API endpoints as well as direct binary gRPC requests, while implementing only gRPC handles for both.
-
We can automatically generate gRPC based API clients.
-
We have built a variety of tools around gRPC handles such as authorization scopes in our fine-grained API keys.
-
We have option to generate swagger files defining the API from the protobuf API spec.
All these things and more have allowed us to now add over 140 endpoints to our API in the last couple years. Check out these endpoints by signing up at clarifai.com.
