This article mainly introduces the AsyncMessenger code framework structure and the main useddata structure

The figure above shows the connections between the key classes in Ceph's AsyncMessenger module. There are 14 main classes used in the AsyncMessenger module. The role of each class, as well as the main member variables and methods contained in it, are introduced below one by one.

---

1. AsyncMessenger class, SimplePolicyMessenger class and Messenger class

The AsyncMessenger class, SimplePolicyMessenger class, and Messenger class are the relationship of inheritance and inheritance. Messenger is an abstract message manager, and its main interface is implemented in the derived class AsyncMessenger. The SimplePolicyMessenger class is some connection strategy to the message manager. For the defined settings, the relevant member variables and methods of the message manager are defined and implemented in AsyncMessenger.

The key member variables and class methods of an AsyncMessenger instance are shown in the following table (including parent class member variables and class methods inherited by this class). AsyncMessenger contains a WorkerPool object, a Processor instance, and a list of 3 AsyncConnectionRef objects and a list of ConnectionRef objects.

---

Member variables in the AsyncMessenger class:

Member variable name	Return value type	description
*pool	WorkerPool	Use pool->get_worker() to get worker threads from the thread pool to work
processor	Processor	Processor instance, mainly used to monitor connections, bind sockets, accept connection requests, etc., equivalent to the processing center of AsyncMessenger
listen_sd	int	Defined listen socket
conns	ceph::unordered_map(entity_addr_t, AsyncConnectionRef)	Address and connection map table, add the connection and address information to this map table when creating a new connection, first search the map according to the address when sending a message, if a return connection is found, if no creation is found A new connection.
accepting_conns	set(AsyncConnectionRef)	Receive a collection of connections, this collection mainly stores those connections that have been received.
deleted_conns	set(AsyncConnectionRef)	A collection of connections that have been closed and need to be cleaned
local_connection	ConnectionRef	Locally connected counter
did_bind	bool	The initial value is false, set to true after the binding address, and set to false again when stop

---

Member methods in the AsynsMessenger class:

Member method name	Return value type	description
bind (const entity_addr_t& bind_addr)	int	Bind the socket, the specific binding process is completed by the processor's bind() function
start()	int	After registering an instance of AsyncMessenger, start the instance. The specific execution process is completed by WokerPool's start() function.
wait()	void	Wait for the stop signal, if you receive the stop message, call the processor's stop() function, then set did_bind to false, and finally delete the established connection
send_message (Message *m, const entity_inst_t& dest)	int	Add a lock, and then call _send_message(m, dest) to send the message to the destination address
get_connection (const entity_inst_t& dest)	ConnectionRef	The function is used to establish a connection, determine whether it is a local connection, otherwise continue to find out whether the connection already exists, and create a connection if it does not exist
ready()	void	The registered AsyncMessenger is ready, start the event processing center, start work, start the worker thread
create_connect(const entity_addr_t& addr, int type)	AsyncConnectionRef	Create a connection and add it to conns
submit_message(Message *m, AsyncConnectionRef con,const entity_addr_t& dest_addr, int dest_type)	void	It will be used when sending messages. According to the destination address, determine whether the connection that needs to send the message exists and whether the connection is a local connection. If it is a local connection, dispatch the message directly. If the connection does not exist, you need to create a new one according to the message type. connection
_send_message(Message *m, const entity_inst_t& dest)	int	Find the destination address from the connection, then call submit_message() to send the message
add_accept(int sd)	AsyncConnectionRef	Create a new connection and add it to accepting_conns

---

2. Processor class, WorkerPool class and Worker class

---

• The Processor class is equivalent to a processor in the AsyncMessenger module. AsyncMessenger needs to complete many operations (start, ready, bind, etc.) through the Processor. When Messenger completes address binding, the Processor starts and then listens for upcoming connections. That is to say, some operations of AsyncMessenger module such as starting, binding, and ready are encapsulated on the basis of the corresponding operation of Processor.

• The Processor class defines an AsyncMessenger object, an instance of NetHandler, and a Worker object.

Member variables (methods) of the Processor class:

Member variable (method) name	Return value type	description
*msgr	AsyncMessenger	The pointer instance of AsyncMessenger is used to call member variables (methods) in AsyncMessenger. The most used is the address information obtained during binding.
net	NetHandler	After binding the socket, set it to non-blocking, then this is the socket option.
*worker	Worker	Worker thread
listen_sd	int	Get the value of the socket description word, non-negative means that the socket was created successfully, -1 means an error
nonce	uint64_t	The unique ID for entity_addr_t in the constructor
bind(const entity_addr_t &bind_addr, const set& avoid_ports)	int	Perform the specific process of binding sockets
start(Worker *w)	int	Execute the start of the message module, specifically start the thread and keep it in working state
accept()	void	The process of establishing a connection. If the connection is established successfully, the connection is added to the accepting_conns set through the add_accept() function
stop()	void	Close socket
rebind(const set& avoid_port)	int	If the binding is not successful for the first time or the binding fails due to other reasons, perform rebinding

---

• The WorkerPool class is a thread pool, the main role is to create worker threads, and then put it into its own worker container, each time the worker thread is created according to the parameters of the configuration file ms_async_op_threads to specify the number of worker threads, the creation is in WorkerPool Made in the constructor.

• A worker collection is defined in the WorkerPool class, which is used to store worker threads, and a coreids is also defined. The user stores a collection of CPU IDs to provide the role of specifying a CPU to run a single thread. One parameter in the configuration file is ms_async_affinity_cores, which binds the created worker to the specified CPU core. If two threads are created, the bound cpu core is 0 and 1. The default ms_async_affinity_cores value is empty, that is, all cpu resources are used. If the cpu resources are not enough, you can specify the worker cpu core.

Member variables (methods) of WorkerPool class

Member variable (method) name	Return value type	description
coreids	vector	Used to store the collection of CPU id
WorkerPool(CephContext *c)	Constructor	The constructor of WorkerPool creates a corresponding number of worker threads according to the value of ms_async_op_threads, and completes the binding of worker and CPU core.
start()	void	Create a worker thread in the worker collection, start the thread and start working
*get_worker()	Worker	Get worker threads in the worker collection
get_cpuid(int id)	int	Get cpu id
workers	Worker*	The collection of Worker threads, the worker threads created by WorkerPool in the constructor are put into this collection

---

• The Worker class is a specific worker thread. The main job of the Worker thread is a loop. Call epoll_wait to get the event that needs to be processed. Use the loop to process this event. When there is an external operation, such as reading a message, registering a callback class, creating a File events, and then initiate the callback operation to process the request. When the message module starts, a WorkerPool object and an instance of EventCenter are defined in the Worker class with a thread.

Member variables (methods) of the Worker class

Member variable (method) name	Return value type	description
*pool	WorkerPool	WorkerPool instance, used to get the CPU id in the entry() function
done	bool	Set to true if the thread's work is completed, otherwise false
center	EventCenter	An instance of EventCenter, perform the initialization of EventCenter in the constructor of Worker
*entry()	void	The entry function of the worker thread starts a while loop to perform event processing, and this worker thread is used in the entire message module
stop()	void	Set done to true, then call the wakeup function of EventCenter to stop the socket

---

3. AsyncConnection class

• AsyncConnection is the core of the entire Async message module. Connection creation and deletion, data read and write instructions, connection reconstruction, and message processing are all performed in this class. This section focuses on the analysis of important member variables and 24 member functions.

  Member variables of AsyncConnection class

Member variable name	Return value type	description
*async_msgr	AsyncMessenger	AsyncMessenger object, call some environment variables, etc.
out_q	map（int, list（pair（bufferlist, Message*）） ）	Data structure for storing messages and message map information
sent	list（Message*）	Store the messages that need to be sent
local_messages	list（Message*）	Store locally transmitted messages
outcoming_bl	bufferlist	Bl to temporarily store messages
read_handler	EventCallbackRef	Callback instructions to handle read requests
write_handler	EventCallbackRef	Callback instructions to handle write requests
connect_handler	EventCallbackRef	Callback instruction to handle connection request
local_deliver_handler	EventCallbackRef	Callback instruction to handle local connection request
data_buf	bufferlist	Bl to store data
data_blp	bufferlist::iterator	pointer to data_buf
front, middle, data	bufferlist	Header, middle part and data part
connect_msg	ceph_msg_connect	Message connection
net	NetHandler	An instance of NetHandler, handling network connections
*center	EventCenter	EventCenter object, used to call the operation of the event center
*recv_buf	char	Buf for receiving messages from socket

---

Member methods of AsyncConnection class

Numbering	Member method name	Return value type	description
1	do_sendmsg(struct msghdr &msg, int len, bool more)	int	What is returned is the length of the message that needs to be sent
2	try_send(bufferlist &bl, bool send=true)	int	Add a write_lock, and then call _try_send to actually send the message
3	_try_send(bufferlist &bl, bool send=true)	int	If the value of send is false, bl will be added to the send buffer. The purpose of this is to avoid errors outside the messenger thread
4	prepare_send_message(uint64_t features, Message *m, bufferlist &bl)	void	Encode and copy the data in m to bl
5	read_until(uint64_t needed, char *p)	int	Loop reading, call read_bulk, if r value is not 0, keep looping
6	_process_connection()	int	Handle connections, perform different operations according to different state, the key point is that the value of state is different
7	_connect()	void	First assign the value of STATE_CONNECTING to state, then call dispatch_event_external to add the read_handler event to the external_events collection
8	_stop()	void	Log off the connection, then assign STATE_CLOSED to state, close the socket, and clean up the event
9	handle_connect_reply(ceph_msg_connect &connect, ceph_msg_connect_reply &r)	int	Perform different operations based on the value of reply.tag
10	handle_connect_msg(ceph_msg_connect &m, bufferlist &aubl, bufferlist &bl)	int	Process the connection of the message, if successful, receive the connection
11	discard_out_queue()	void	Clear AsyncConnection's message queue
12	requeue_sent()	void	Re-enter the send queue
13	handle_ack(uint64_t seq)	void	Process the confirmation message and delete the message in the send queue
14	write_message(Message *m, bufferlist& bl)	int	Write the message to complete_bl, call _try_send to send the message
15	_reply_accept(char tag, ceph_msg_connect &connect, ceph_msg_connect_reply &replybufferlist authorizer_reply)	int	There is a reply_bl of bufferlist structure, call try_send to send reply_bl out
16	is_queued()	bool	Determine whether to enter the queue, mainly the two queues out_q and outcoming_bl
17	shutdown_socket()	void	Close socket
18	connect(const entity_addr_t& addr, int type)	void	Used when the AsyncConnection is first constructed, and then call the _connect() function
19	accept(int sd)	void	Set the value of state to STATE_ACCEPTING, then call the create_file_event function to create a file event, and call the dispatch_event_external function to distribute the callback instruction
20	send_message(Message *m)	int	Generally, when you need to send a message, this function will be called for specific sending operations, before the connection has been completed
21	handle_write()	void	Use a while loop to call write_message to write data to m
22	process()	void	Or do different processing according to different state values
23	local_deliver()	void	This function is mainly used to handle local messaging
24	cleanup_handler()	void	Clean up the event processing assistant and reset it

---

4. EventCenter class and EventCallback class

• The AsyncMessenger message module is based on epoll's event-driven approach, instead of establishing a Pipe for each connection between them, and then creating two threads, one to handle the reception of the message and the other to handle the sending of the message. Thread way. Unlike the SimpleMessenger message module, the AsyncMessenger message module uses events, so it needs a data structure to handle events, namely EventCenter, with a thread dedicated to cyclic processing, all operations are performed through callback functions, avoiding a large number of threads usage of. In EventCenter, a FileEvent data structure and a TimeEvent data structure are defined. Most events are FileEvents. The following describes the main member variables and methods in EventCenter.

Member variable (method) name	Return value type	description
FileEvent	struct	File event class
TimeEvent	struct	Time event
external_events	deque(EventCallbackRef)	Queue for storing external events
*file_events	FileEvent	FileEvent instance
*driver	EventDriver	Example of EventDriver
time_events	map(utime_t, list(TimeEvent))	Event event container
net	NetHandler	NetHandler examples
process_time_events()	int	Handle time events
*_get_file_event(int fd)	FileEvent	Get file events
init(int nevent)	int	Create different event handlers according to different macros; call create_file_event to create events.
create_file_event(int fd, int mask, EventCallbackRef ctxt)	int	Create file events based on fd and mask, call add_event function to add the created event to the event handler to process
create_time_event(uint64_t milliseconds, EventCallbackRef ctxt)	uint64_t	Create a time event, and then add it to time_events
delete_file_event(int fd, int mask)	void	Delete file event
delete_time_event(uint64_t id)	void	Delete time event
process_events(int timeout_microseconds)	int	If the event is read_cb or write_cb, the corresponding callback function is called for processing (completed by the do_request function); if it is not the two events, the events in the external_events queue are taken into cur_process, and a while loop is called to process .
dispatch_event_external(EventCallbackRef e)	void	Put the created external events into the external_events queue

• EventCallback is an interface class, and its subclasses will be defined according to different operations. The use method uses a virtual function do_request() to call back to handle different events. The specific processing is performed in do_request().

---

5. EventDriver class, EpollDriver class, KqueueDriver class and SelectDriver class

• The event center is equivalent to a container for event processing. It does not really handle events by itself, and completes the processing of events by means of callback functions. Similarly, how to get the events that need to be processed is not completed by the event center, it is only responsible for processing, the specific acquisition of the events that need to be processed is completed by EventDriver, EventDriver is an interface class, its implementation is mainly by EpollDriver, KqueueDriver And SelectDriver three types of operations. Ceph supports the use of multiple operating systems. If you are using the Linux operating system, use EpollDriver. If you are using BSD, use KqueueDriver. If you are not, use SelectDriver (the system is defined as the worst case). Event-driven execution mainly depends on the epoll method, which mainly has three functions: epoll_create (a file node is created in the epoll file system, and epoll's own kernel high-speed cache area is created, a red and black tree is established, and the desired size is allocated. Memory object, create a list linked list, used to store ready events); epoll_ctl (put the socket to be monitored on the corresponding red-black tree, and register the kernel interrupt handler A callback function notifies the kernel that if the data for this handle arrives, put it in the ready list); epoll_wait (observe whether there is data in the ready list, and extract and clear the ready list, very efficient). Because this project runs in Linux, the following uses EpollDriver as an example to describe Ceph's underlying event driver.

EpollDriver member variables (methods)

Member variable (method) name	Return value type	description
epfd	int	epoll file descriptor
*events	struct epoll_event	an object of epoll_event
size	int	Get the number of files when performing initialization
init(int nevent)	int	Perform the initialization of EpollDriver, mainly call epoll_create to create epoll object
add_event(int fd, int cur_mask, int add_mask)	int	Perform different operations according to the mask of the event. If it is EVENT_READABLE, it means that the corresponding file descriptor is readable. If it is EVENT_WRITABLE, it means that the file descriptor is writable, and then call epoll_ctl to add the event
del_event(int fd, int cur_mask, int del_mask)	int	Call epoll_ctl to modify or delete events
resize_events(int newsize)	int	Number of empty events
event_wait(vector &fired_events, struct timeval *tp)	int	Call epoll_wait loop to process events

---

6. NetHandler

• NetHandler is a class for network processing in the AsyncMessenger module, which defines six key member methods, of which NetHandler::generic_connect() is needed for each connection, creating a socket, setting the socket to non-blocking, setting Socket options, etc. are frequently used methods, the table below analyzes them in detail.

Member method name	Return value type	description
create_socket(int domain, bool reuse_addr=false)	int	Create socket
generic_connect(const entity_addr_t& addr, bool nonblock)	int	The communication parties use this function to create a connection, first call create_socket() to create a socket, then set the created socket to non-blocking, and then call system socket:: connect() to establish a connection after completion
set_nonblock(int sd)	int	Set Socket to non-blocking
set_socket_options(int sd)	void	Call the system's socket::setsockopt function to set some key options of the socket
connect(const entity_addr_t &addr)	int	A simple encapsulation of NetHandler::generic_connect()
nonblock_connect(const entity_addr_t &addr)	int	Interface function, set up non-blocking connection

---

The basic data structure and framework of AsyncMessenger are described above, and the code flow is described in the next chapter.

Transfer from: http://blog.csdn.net/zhq5515/article/details/54234893