Problem Description

We believe that after the prevailing from DevOps, the operation and maintenance will start writing code (it should be said more than the previously written more), the birth of the operation and maintenance development (more and more). For example, developing Prometheus Exporter has become a work and maintenance (in fact, it is existing, work is done, it is not necessary to develop, or you can implement it).

Now, we need to develop Prometheus Exporter for our own application, but the application provides a GRPC interface, so the operation and maintenance should learn how to use the GRPC framework.

This note will record: GRPC's quick start method, learning notes, and related issues (this note is our study notes for GRPC, so only the foundation entry related content).

solution

We understand how it works by learning GRPC usage processes:
1) Write .proto file: This file contains information about RPC methods, data structures, etc.
2) Generate Pile Code: Use the command protoc to generate a pile code (can generate a variety of languages, such as Go C ++ Java Python C #, etc.). "Pile code" is similar to "class library".
3) Write the server: In the program, use these class libraries, write the server, implement the RPC method. Then, run the server, listen to the TCP port, wait for the request;
4) Write the client: In the program, use these class libraries, write the client, the client connects the server, and calls these RPC methods, and the receiving the server returns the result;

In the call, the relationship between the various parts can be explained below:

We use the Python language, so we focus on using the Python example. However, using other languages ​​is also possible, and as long as the GRPC framework is used, in view of GRPC is also network communication, multiple languages ​​are interoperable.

Learning road map

The official document has provided detailed examples and is the best tutorial. Follow the official document, learn, while practicing, it is enough.

For Python, use the GRPC framework, if you need to get started quickly, it is recommended to read the document in the following order:
1) Introduction to GRPC:Introduction to gRPC | gRPC
2) Getting started with the GRPC framework: Python:Quick start | Python | gRPC
3) Use the full example of the GRPC framework (Python):Basics tutorial | Python | gRPC

Common document (Python):
1）Welcome to gRPC Python’s documentation! — gRPC Python 1.36.1 documentation
2）Language Guide  |  Protocol Buffers  |  Google Developers
3) About GRPC concepts and terminology:Core concepts, architecture and lifecycle | gRPC

Entry case

Because we use the Python language, the example here uses the Python environment. Although it is different from other languages, the process is similar.

Preparation (installation environment)

# In order not to affect the host environment, we choose to operate in Virtual Environment
 # Since everyone has already begun to learn the GRPC framework, we believe that tools such as virtual environments should be credible, so the notes are not further introduced in the virtual environment.
mkvirtualenv "grpc"
workon "grpc"

 # Install GRPC and Tools
pip install grpcio grpcio-tools

First stage, definition service

Define helpoWorld.Proto files:

syntax = "proto3";

package helloworld;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}

This file defines the "Communication Protocol of the client and the server":
1) At the point of the server: At that time the server will implement Greter service, the service provides a SayHello method, the parameter is a HelloreQuest structure, returns a HellorePly structure;
2) Standing at the client: At that time, the client will call the Greter service Sayello method, incoming the HelloreQuest structure, get the response of the HellorePly structure;

In the second phase, create a pile code

HelloWorld.Proto file definition protocol, but cannot be used directly, you need to generate a pile code (class library).

Use the existing protocal buffers definition to create a pile code:

# Create a pile code
python -m grpc_tools.protoc \
    --proto_path=./ \
    --python_out=. --grpc_python_out=. \
    ./helloworld.proto

When executed as the above command, the following file will be generated:
1) HelloWorld_pb2.py: This file contains the data structure, service information, etc., which we defined in the Protocol, etc. (mostly used for client).
2) HelloWorld_PB2_GRPC.PY: This file contains GRPC-related methods (mostly used for server)

Description of command line parameters:
1) - Proto_path: Specifies the .proto lookup path (similar to Java's ClassPath, Python PythonPath). The code is usually written in different files, then references (import, include), so you need to specify the search path, which is the role of this option. Of course, our example only uses HelloWorld.Proto, not using other files, so it is excessive.
2) - Python_out: HelloWorld_PB2.PY output path
3) - GRPC_PYTHON_OUT: File HelloWorld_PB2_GRPC.PY output path
4) HelloWorld.Proto: Our protocol file

In the third stage, write the server

Define the HelloWorld_Server.py file:

from concurrent import futures

import grpc

import helloworld_pb2
import helloworld_pb2_grpc


class Greeter(helloworld_pb2_grpc.GreeterServicer):

    def SayHello(self, request, context):
        return helloworld_pb2.HelloReply(message='Hello, %s!' % request.name)


if __name__ == '__main__':
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
    server.add_insecure_port('[::]:50051')
    server.start()
    server.wait_for_termination()

class Greeter(helloworld_pb2_grpc.GreeterServicer):
1) The server should provide services, so it is to implement GREETER service, inheriting from the GreeServiceer class (this is basically fixed). Then, implement the SayHello method.
2) The Request parameter of the SAYHELLO method is the HelloreQuest structure, and the NAME attribute is obtained via the request.name;
3) As defined by the protocol, the method returns the HellorePly structure that contains the Message parameter. From the code: HellorePly appears in the form of a method; its properties appear in the form of method parameters; rather than directly incoming the HellorePly structure.

Then, the main method launches the GRPC service, listens 50051 port, waiting for the client access.

Finally, we need to launch this server, otherwise the following client cannot connect: python ./helloworld_server.py

In the fourth stage, write clients

Define the helloworld_client.py file:

from __future__ import print_function

import grpc

import helloworld_pb2
import helloworld_pb2_grpc
from google.protobuf.json_format import MessageToJson
        
if __name__ == '__main__':
    with grpc.insecure_channel('localhost:50051') as channel:
        stub = helloworld_pb2_grpc.GreeterStub(channel)
        response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
        print("Greeter client received: " + response.message)
		print(MessageToJson(response, including_default_value_fields=True))

MessageToJson(): Turn back to JSON string type
including_default_value_fields=True: If the field is the default value, it is not passed between the client and the server. This property is needed to display the default field.

The flow of this sample code is basically such a process):
1) Connect the GRPC service;
2) Set the pile code to use this connection;
3) Send a request, call the Sayhello method, and incorporate the HelloreQuest structure (of course, here is still in the form of a data structure);

Run the client, will get the output of the server:

# python3 helloworld_client.py
Greeter client received: Hello, you!
{
  "message": "Hello, you!"
}

At this point, a simple GRPC example is successful.

Frequently asked questions

ModuleNotFoundError: No module named 'common_pb2'

ImportError: No module named 'InternalMessage_pb2' · Issue #3430 · protocolbuffers/protobuf

Problem Description：

Traceback (most recent call last):                                    
  File "agent.py", line 22, in <module>                 
    import controller_pb2 as Controller                                                    
  File "/srv/http/agent/controller_pb2.py", line 14, in <module>
    import common_pb2 as common__pb2                                                    
ModuleNotFoundError: No module named 'common_pb2'

problem causes: All .Proto is involved in the generation of the pile code.

solution: Note We use * .proto when creating a pile code to define all .Proto to participate in the generation of the pile code.

# Create a pile code
python -m grpc_tools.protoc \
    --proto_path=./ \
    --python_out=. --grpc_python_out=. \
    ./*.proto

related articles

"GRPC" - common code
"GRPC" - debugging tool
「gRPC」- 4.Packages and Build Tools
「gRPC」- 05.DEBUGING

references

Introduction to gRPC | gRPC
Quick start | Python | gRPC
Protobuf to json in python - Stack Overflow
python - Protobuf doesn't serialize default values - Stack Overflow
python - Getting all field names from a protocol buffer? - Stack Overflow