The Deep Learning Lecture Hall is dedicated to pushing the latest technologies, products and activities in artificial intelligence and deep learning.

I believe that many of the small partners in the community have used Caffe's deep learning framework for a long time, and I also hope to understand Caffe's implementation from the code level to customize the new features. This article will analyze the Caffe source code from the perspective of the overall architecture and the underlying implementation.

1.Caffe overall architecture

The Caffe framework has five main components, Blob, Solver, Net, Layer, Proto, and its structure is shown in Figure 1. Solver is responsible for the training of deep networks. Each Solver contains a training network object and a test network object. Each network consists of several layers. Each Layer's input and output Feature maps are represented as Input Blobs and Output Blobs. Blob is the structure of Caffe's actual stored data. It is a matrix of indefinite dimensions. It is generally used to represent a straightened four-dimensional matrix in Caffe. The four dimensions correspond to the Batch Size(N) and the number of channels of the Feature Map (C). Feature Map height (H) and width (W). Proto is based on Google's Protobuf open source project. It is an XML-like data exchange format. Users only need to define the data members of the object according to the format. The serialization and deserialization of objects can be implemented in multiple languages. Structure definition, storage, and reading of the network model.

2.Blob analysis

The basic data storage class blob in Caffe is described below. Blob uses the SynchedMemory class for data storage. The data member data_ points to the memory or memory block that actually stores the data. The shape_ stores the dimension information of the current blob, and the diff_ holds the gradient information when the reverse is passed. In the Blob, there is actually only a four-dimensional form of num, channel, height, and width. It is a data structure of indefinite dimensions. The data is expanded and stored, and the dimension exists in a shape_variable of type vector<int>, so that each Each dimension can be arbitrarily changed.

Let's take a look at the key functions of Blob. The data_at function can read the data stored in this class. diff_at can be used to read the error sent back. By the way, give a hint, try to use data_at (const vector<int>& index) to find the data. The Reshape function can modify the storage size of the blob, and count is used to return the amount of stored data. The BlobProto class is responsible for packetizing and serializing Blob data into Caffe's model.

3. Factory mode description

Next, we introduce a design pattern Factory Pattern. The Solver and Layer objects in Caffe are created using this pattern. First, look at the UML class diagram in factory mode:

Just as the Factory generates the same function but different models, these products implement the same Operation. Many people read the code of the factory mode, which will cause such a question. Why doesn't the new one come out, so it seems that there is nothing wrong with the new one. Imagine the following situation, because the name of the code refactoring class has changed, or the constructor parameters have changed (increasing or decreasing parameters). And there are N new classes in your code. If you don't use the factory again, you can only find one by one. The role of the factory model is to reduce the user's understanding of the product itself and reduce the difficulty of use. If you use a factory, you only need to modify the implementation of the concrete object method of the factory class, and other code will not be affected.

For example, write code should not be hungry to work overtime to eat supper, McDonald's chicken wings and KFC's chicken wings are MM love to eat, although the taste is different, but whether you bring MM to McDonald's or KFC, just tell the waiter "Take four chicken wings" will do. McDonald's and KFC are factories that produce chicken wings.

4.Solver analysis

Let's move back to the topic, let's see how Solver is optimized in Caffe. The SolverRegistry class is the factory class we saw above. It is responsible for giving us a product that optimizes the algorithm. The external only needs to define the data and network structure, and it can optimize itself.

Solver<Dtype>* CreateSolver(const SolverParameter& param) This function is the operation of CreateProduct in factory mode. This SolverRegistry factory class in Caffe can provide us with 6 kinds of products (optimization algorithm):

The functions of these six products are to implement the parameter update of the network, but the implementation is different. Then let's take a look at their use process. Of course, these products are similar to the Operation in the Product class above. Each Solver inherits the Solve and Step functions. The only thing that is unique in each Solver is that the function in the ApplyUpdate function is different, the interface is the same, and this is also with us. The products produced by the factory mentioned before have the same function and different details. For example, most rice cookers have the function of cooking rice, but each type of rice cooker can be heated differently, and the chassis is heated by stereo heating. Wait. Next, let's take a look at the key functions in Solver.

The flow chart of the Solve function in Solver is as follows:

Step function flow chart in the Solver class:

 

The key to Solver is the process of calling the Sovle function and the Step function. You only need to compare the implementation of the two functions in the Solver class. You can understand the process of Caffe training execution by understanding the above two flowcharts.

http://5.NetClass resolution

After analyzing Solver, we will analyze some key operations of the Net class. This is the deep network object we created with Proto, which is responsible for the forward and reverse delivery of deep networks. The following is the initialization method of the Net class NetInit function call flow:

A simple analysis of key functions in Net's class

1).ForwardBackward: Forward and Backward are called in order.

2).ForwardFromTo(int start, int end): Performs forward transfer from the start layer to the end layer, using a simple for loop call.

3).BackwardFromTo(int start, int end): Similar to the previous ForwardFromTo function, calling the reverse pass from the start layer to the end layer.

4).The ToProto function completes the serialization of the network into a file, looping through the ToProto function of each layer.

6.Layer analysis

Layer is the basic building block of Net, such as a convolutional layer or a Pooling layer. This section describes the implementation of the Layer class.

6.1Layer's inheritance structure

6.2 Layer creation

Similar to how Solver was created, Layer was created using the factory pattern. Here are a few macro functions:

REGISTER_LAYER_CREATOR is responsible for putting the function that creates the layer into the LayerRegistry.

Let's take a look at the generated macro REGISTER_LAYER_CLASS for most of the functions created by the layer. You can see that the macro function is relatively simple, and the type is used as part of the function name, so that you can create a create function and put the create function into the LayerRegistry.

 

REGISTER_LAYER_CREATOR(type, Creator_##type##Layer)

This code is in the split_layer.cpp file

REGISTER_LAYER_CLASS(Split)。

So we will replace the type after giving you an example, refer to the code below.

 

Of course, the creation function here seems to be called directly, without some problems related to our previous factory pattern. Are all the classes of the layer like this? Of course not, let's take a closer look at the convolution class.

How does the convolution layer create a function? Of course, the creation function of the convolution layer is in LayerFactory.cpp. The screenshot is for everyone to see. The specific code is as follows:

Both of these types of Layer's creation functions have corresponding declarations. Here is a direct description of the layer in addition to the cudnn implementation, the other layers are created in the first way, and the layer with cudnn implementation is the second way to create the function.

6.3 Layer initialization

After the introduction, let's see how several functions in the layer are called.

Key Function Setup This function is called in NetInit in the previous flowchart. The code is as follows:

In this way, the entire Layer initialization process, CheckBlobCounts is called first, then LayerSetUp, followed by Reshape, and finally SetLossWeights. In this way, the life cycle of the Layer initialization is known to everyone.

6.4 Introduction to other functions of Layer

Layer's Forward function and Backward function complete the forward and reverse transfer of the network. These two functions must be implemented in their own implementation of the new layer. Among them, Backward will modify the diff_ of the blob in the bottom, thus completing the direction of error.

7.Protobuf introduction

The Caffe.proto file in Caffe is responsible for the construction of the entire Caffe network, and is responsible for the storage and reading of Caffemodel. Here's an example of how Protobuf works:

Use the protobuffer tool to store 512-dimensional image features:

1).Message Write: Create a new txt file with the suffix name changed to proto, write your own message as follows, and put it into the unpacked protobuff folder;

Among them, dwFaceFeatSize represents the number of feature points; pfFaceFeat represents the face features.

2).Open the windows command window (cmd.exe)---->cd space, copy and paste the file path of the protobuff into the ------>enter;

3).Enter the command protoc *.proto --cpp_out=. --------->enter

4).You can see that the files "*.pb.h" and "*.pb.cpp" are generated in the folder, indicating that it is successful.

5).The following can be integrated with your own code:

(1) Create your own project, add "*.pb.h" and "*.pb.cpp" files to your project and write #include" *.pb.h"

(2) According to the tutorial of the distribution library, you can configure the library.

Protobuf's library method under VS:

Solution---->Right-click on the project name---->Attribute

(1)c/c++--->General--->Additional include directory--->

($your protobuffer include path)\protobuffer 

(2)c/c++--->Linker-->General--->Additional Library Directory-->

($your protobuffer lib path)\protobuffer

(3) c/c++--->Linker-->Input--->Additional Dependencies-->

Libprotobufd.lib; (with d for debug mode)

Or libprotobuf.lib; (without d, in release mode)

The method of packaging using protobuf is as follows:

7.1 Modeling of Caffe's Model

BlobProto is actually a class that Blob is serialized into Proto. The Caffe model file uses this class. Net calls the Toproto method of each layer. The Toproto method of each layer calls the ToProto method of the Blob class, so that the complete model is serialized into the proto. Finally, just serialize the object that this proto inherits from the message class into the file to complete the model write file. When Caffe packages the model, it simply calls the WriteProtoToBinaryFile function, and the contents of this function are as follows:

So far, Caffe's serialization model has been completed.

7.2 A brief description of Proto.txt

The construction of the Caffe network and the parameter definition of Solver are done by this type file. ReadProtoFromTextFile is called during the Net build process to read all network parameters. Then call the above process to build the entire caffe network. This file determines how to use each blob in the caffe model. If there is no file, the caffe model file will not be available, because only a variety of blob data is stored in the model. The float value, and how to split the data is determined by the prototxt file.

Caffe's architecture uses a reflection mechanism to dynamically create layers to build Net. Protobuf essentially defines graph. The reflection mechanism is formed by the macro with the map structure, and then uses the factory pattern to create various layers. Of course, unlike the general definition configuration, xml or json is used. The project uses the proto file to assemble the components.

to sum up

The above is a general introduction to the Caffe code architecture, hoping to help the community's partners find the key to open the custom Caffe door. The authors hope to use this to learn more about peer-to-peer communication with Caffe and the underlying implementation of the deep learning framework.