===============================================
Inter Process Communication(IPC) Message Router
===============================================
Introduction
============
Inter Process Communication(IPC) Message Router
IPC Router provides a connectionless message routing service between
multiple processes in a MSM setup. The communicating processes can
run either in the same processor or in a different processor within
the MSM setup. The IPC Router has been designed to::
1) Route messages of any types
2) Support a broader network of processors
The IPC Router follows the same protocol as the existing RPC Router
in the kernel while communicating with its peer IPC Routers.
Hardware description
====================
The IPC Router doesn't implement any specific hardware driver.
The IPC Router uses the existing hardware drivers to transport messages
across to the peer router. IPC Router contains a XPRT interface layer to
handle the different types of transports/links. This interface layer
abstracts the underlying transport complexities from the router and
provides a packet/message interface with the transports.
Software description
====================
The IPC Router is designed to support a client-server model. The
clients and servers communicate with one another by exchanging data
units known as messages. A message is a byte string from 1 to 64K bytes
long. The IPC Router provides a connectionless message routing service
between the clients and servers i.e. any client/server can communicate
with any other client/server in the network of processors.
Network Topology Overview:
--------------------------
The system is organized as a network of nodes. Each processor in the
network is the most fundamental element called node. The complete
network is hierarchically structured i.e. the network is divided into
tiers and each tier is fully-meshed. The following figure shows an
example network topology.::
----N1--- ----N4---
| | | |
| | | |
N2-----N3-------N5-------N6
| |
| |
----N7----
| |
| |
N8------N9
Each node in the complete network is identified using a unique node id
(Nx in the example network). In the example network, nodes N1, N2 & N3
are fully-meshed to form a tier 1 network. Similarly nodes N4, N5 & N6
form another tier 1 network and nodes N7, N8 & N9 form third tier 1
network. These three tier 1 networks are fully-meshed to form a tier 2
network.
Each transport/link in the network is identified using a unique name/id
called XPRT id. This XPRT id is used by the nodes to identify the
link to be used while sending message to a specific destination.
In addition, each transport/link in the network is assigned a link id.
This link id is used to identify the tier to which the link belongs to.
This link marking is used to avoid the routing loops while forwarding
the broadcast messages. The incoming messages are only forwarded onto an
egress link which has a link id different from that of an ingress link.
IPC Router Addressing Overview:
-------------------------------
Each client/server in the network is identified using a unique
<Node_id:Port_id> combination. Node_id identifies the processor on which
a client/server is running. Port_id is a unique id within a node. This
Port_id is assigned by the IPC Router in that node when a client/server
comes up. The Node_id & Port_id are 32 bits each.
Port_id 0xFFFFFFFE is reserved for Router &
0xFFFFFFFF is reserved for broadcast messages.
Each server in the network can also be addressed using a service name.
The service name is of the form <service(32 bits):instance(32 bits)>.
When a server comes up, it binds itself with a service name. This name
information along with the <Node_id:Port_id> is broadcast onto the
entire network.
Control Path:
-------------
IPC Router uses the control messages to communicate and propagate the
system wide events to the other routers in the system. Some of the
events include::
1) Node Status
2) Server Status
3) Client Status
4) Flow Control Request/Confirmation
Message Header:
---------------
IPC Router prepends a header to every message that it communicates with
other IPC Router. The receiving IPC Routers use the header to identify
the source and destination of the message, size of the message, type
of the message and handle any flow control requests. The IPC Router
header format is as follows::
0 31
-------------------------------------------------
| Version |
-------------------------------------------------
| Message Type |
-------------------------------------------------
| Source Node ID |
-------------------------------------------------
| Source Port ID |
-------------------------------------------------
| Confirm RX |
-------------------------------------------------
| Payload Length |
-------------------------------------------------
| Destination Node ID |
-------------------------------------------------
| Destination Port ID |
-------------------------------------------------
Message Header v2(Optimized):
-----------------------------
The following optimization has been done to the IPC Router header to
make it concise, align with the system requirement and enable future
expansion::
0 8 16 24 31
-----------------------------------------------------------------
| Version | Message Type | Control Flag |
-----------------------------------------------------------------
| Payload Length |
-----------------------------------------------------------------
| Source Node ID | Source Port ID |
-----------------------------------------------------------------
| Destination Node ID | Destination Port ID |
-----------------------------------------------------------------
Control Flag::
0 14 15
------------------------------------------------------------------
| Reserved | Opt. Hdr. | Confirm RX |
------------------------------------------------------------------
IPC Router identifies and processes the header depending on the version
field. The Confirm RX field in message header v1 becomes part of the
control flag. All the other fields are reduced in size to align with the
system requirement.
Optional Header:
An optional header bit field is introduced in the control flag to handle
any unforeseen future requirement that this header cannot handle. When
that bit is set, an optional header follows the current header. The
optional header format is as follows::
0 8 16 31
-----------------------------------------------------------------
| Length(words) | Type | Control Flag |
-----------------------------------------------------------------
| |
| Optional Header Contents |
| |
-----------------------------------------------------------------
Design
======
The IPC Router is organized into 2 layers::
1) Router Core layer
2) Router - XPRT Interface layer
This organization allows the router to abstract the XPRT's complexities
from that of the core router functionalities. The Router Core layer
performs the following core functions::
1) Message Routing
2) Distributed name service
3) Flow control between ports
The Router core layer contains the following important data structures
to perform the core functions in their respective order::
1) Routing Table
2) Table of Active Servers
3) Table of Active Remote ports
All these data structures get updated based on the events passed through
the control path.
The Router - XPRT Interface layer hides the underlying transport
complexities and provides and abstracted packet interface to the
Router Core layer. The Router - XPRT Interface layer registers itself
with the Router Core layer upon complete initialization of the XPRT.
The Router - XPRT Interface layer upon registration exports the
following functionalities to the Router Core::
1) Read from the XPRT
2) # of bytes of data available to read
3) Write to the XPRT
4) Space available to write to the XPRT
5) Close the XPRT
The user behavioral model of the IPC Router should be::
1) Create a port
2) If server, register a name to the port
3) If remote port not known, lookup through the name
4) Send messages over the port to the remote port
5) Receive messages along with the source info from the port
6) Repeat steps 3, 4 & 5 as required
7) If server, unregister the name from the port
8) Close the port
Power Management
================
IPC Message Router uses wakelocks to ensure that the system does not go
into suspend mode while there are pending messages to be handled. Once all
the messages are handled, IPC Message Router releases the wakelocks to
allow the system to go into suspend mode and comply with the system power
management requirement.
SMP/multi-core
==============
The IPC Router uses mutexes & spinlocks to protect the shared data
structures to be SMP/multi-core safe.
Security
========
None
Performance
===========
None
Interface
=========
Kernel-space APIs:
------------------
::
/*
* msm_ipc_router_create_port - Create a IPC Router port
*
* @msm_ipc_port_notify: notification function which will notify events
* like READ_DATA_AVAIL, WRITE_DONE etc.
* @priv: caller private context pointer, passed to the notify callback.
*
* @return: a valid port pointer on success, NULL on failure
*
*/
struct msm_ipc_port * msm_ipc_router_create_port(
void (*msm_ipc_port_notify)(unsigned event, void *data,
void *addr, void *priv),
void *priv);
::
/*
* msm_ipc_router_close_port - Close down the port
*
* @port: Port to be closed
*
* @return: 0 on success, -ve value on failure
*
*/
int msm_ipc_router_close_port(struct msm_ipc_port *port);
::
/*
* msm_ipc_router_send_to - Send data to a remote port
*
* @from_port: Source port of the message
* @data: Data to be sent
* @to_addr: Destination port name or address
*
* @return: number of bytes sent on success, -ve value on failure
*
*/
int msm_ipc_router_send_to(struct msm_ipc_port *from_port,
struct sk_buff_head *data,
struct msm_ipc_addr *to_addr);
::
/*
* msm_ipc_router_recv_from - Receive data over a port
*
* @port: Port from which the data has to be read
* @data: Pointer to the data
* @src_addr: If valid, filled with the source address of the data
* @timeout: Time to wait for the data, if already not present
*
* @return: number of bytes received on success, -ve value on failure
*
*/
int msm_ipc_router_recv_from(struct msm_ipc_port *port,
struct sk_buff_head **data,
struct msm_ipc_addr *src_addr,
unsigned long timeout);
::
/*
* msm_ipc_router_register_server - Bind a local port with a service
* name
*
* @server_port: Port to be bound with a service name
* @name: Name to bind with the port
*
* @return: 0 on success, -ve value on failure
*
*/
int msm_ipc_router_register_server(struct msm_ipc_port *server_port,
struct msm_ipc_addr *name);
::
/*
* msm_ipc_router_unregister_server - Unbind the local port from its
* service name
*
* @server_port: Port to be unbound from its service name
*
* @return: 0 on success, -ve value on failure
*
*/
int msm_ipc_router_unregister_server(struct msm_ipc_port *server_port);
::
/*
* msm_ipc_router_lookup_server - Lookup port address for the port name
*
* @name: Name to be looked up for
*
* @return: Port address corresponding to the service name on success,
* NULL on failure
*
*/
struct msm_ipc_addr * msm_ipc_router_lookup_server(
struct msm_ipc_addr *name);
User-space APIs:
----------------
User-space applications/utilities can use the socket APIs to interface
with the IPC Router. IPC Router, in order to support the socket APIs,
will register a new socket Address/Protocol Family with the kernel
Socket layer. The identity of the new Address/Protocol Family will be
defined using the macro AF_MSM_IPC/PF_MSM_IPC (hardcoded to 38) in
include/linux/socket.h file. Since IPC Router supports only message
oriented transfer, only SOCK_DGRAM type of sockets will be supported
by the IPC Router.
Driver parameters
=================
debug_mask - This module parameter is used to enable/disable Router
log messages in the kernel logs. This parameter can take any value
from 0 to 255.
Dependencies
============
Drivers in this project:
------------------------
The following drivers are present in this project, listed in the
bottom - up order of the stack.
1a) Router - SMD XPRT Interface driver. This driver is used to interface
the Router with the SMD transport.
1b) Router - HSIC XPRT Interface driver. This driver is used to interface
the Router with the HSIC_IPC Bridge transport for off-chip communication.
2) Core Router driver. This driver performs the core functionalities
3) Socket - Router Interface driver. This driver enables the socket
interface to be used with the IPC Router.
In the write/send direction, these drivers interact by invoking the
exported APIs from the underlying drivers. In the read/receive
directions, these drivers interact by passing messages/events.
Drivers Needed:
---------------
::
1) SMD
2) Kernel Socket Layer
3) Platform Driver
4) HSIC IPC Bridge Driver
To do
=====
Improvements:
-------------
The IPC Router is designed to route any messages, as long as the system
follows the network architecture and addressing schemes. But the
implementation in progress will route only QMI messages. With few
additional enhancements, it can route existing RPC messages too.