文章目錄

引用

前言

在一套完整的分佈式系統中，client端向server端發起一個請求，然後client等待此請求被server端處理完畢，然後接受到serve的返回結果。自此一個請求就算作是被處理完了。這種block等待處理結果的請求處理行爲在我們日常的系統中十分的常見。但是這種處理方式的一個明顯弊端是，未處理完成的請求勢必會佔住server端的處理資源。因此一般常見的改進做法是提高server端的Handler數量，來提高服務端的請求併發處理能力，這種做法是比較簡單直接的。但其實這裏還有另外一個方向點的優化，是否能夠提高單個請求的處理時間來做優化呢？比如一些已經被處理完畢的請求，但是正處於返回response結果的，這也是會佔着Handler資源的。因爲返回response操作也是在請求被處理環節的一部分。假設說我們能將回復請求的階段從處理請求方法中拆分出去，通過延時返回的方式，毫無疑問，這也會在一定程度上提高server端的throughput。本文筆者來聊聊關於RPC Server請求的回覆延時處理以及Hadoop RPC Server是如何做這部分優化的。

Server端延時請求回覆的優劣勢

按照前面我們所說的，如果將server端的<請求即刻處理->請求即時回覆>變爲<請求即刻處理->請求延時恢復>，它會給整個系統帶來怎樣的變化呢？

以下是其所帶來的好的一面和不好的一面：

優勢：
增大系統整體的throughput，因爲請求結果回覆變爲了延時異步的方式，這相當於提早釋放了Handler的資源，讓Handler能夠馬上接下來處理其它客戶端的請求。

弊端：

請求回覆的延時處理意味着server端於client的open connection會增多，因爲client的請求回覆還沒有被執行，這些連接並沒有被關閉。
Server端因爲此改動接受了過多的請求，導致請求回覆階段成爲bottleneck。

Hadoop RPC Server的請求延時回覆處理

下面我們通過Hadoop RPC Server內部目前已經優化了的請求回覆處理例子，來具體瞭解下這部分的處理邏輯。

樣例一：增加延時請求計數的處理

這個改動源自Hadoop社區JIRA：HADOOP-10300：Allowed deferred sending of call responses。

它的一個主要思路是這樣的：

1）在每個PRC call裏面多添加了一個請求等待的計數值，初始值爲1
2）正常情況下，Server端在處理完請求後，會執行sendResponse方法，然後會將上述計數值做減1操作，然後執行請求回覆操作。
3）但是，如果我們想要做請求的延時回覆處理，我們可以額外執行一個postponeResponse的方法來增大請求回覆等待的計數值。這樣的話，在正常邏輯中的sendResponse則不會實際執行請求回覆操作，它只做計數值的減操作。只有再第二次Server被觸發執行了sendResponse後，纔會執行請求回覆操作。

相關代碼如下：

/** 請求回覆等待計數值 */
private AtomicInteger responseWaitCount = new AtomicInteger(1);

...

 
/**
 * Allow a IPC response to be postponed instead of sent immediately
 * after the handler returns from the proxy method.  The intended use
 * case is freeing up the handler thread when the response is known,
 * but an expensive pre-condition must be satisfied before it's sent
 * to the client.
 */
@InterfaceStability.Unstable
@InterfaceAudience.LimitedPrivate({"HDFS"})
public void postponeResponse() {
  // 執行延時回覆處理，將計數值加1
  int count = responseWaitCount.incrementAndGet();
  assert count > 0 : "response has already been sent";
}

@InterfaceStability.Unstable
@InterfaceAudience.LimitedPrivate({"HDFS"})
public void sendResponse() throws IOException {
  // 執行請求回覆操作時，減小計數值
  int count = responseWaitCount.decrementAndGet();
  assert count >= 0 : "response has already been sent";
  // 如果計數值爲0了，則進行實際回覆返回操作，否則不進行response信息的返回
  if (count == 0) {
    assert rpcResponse != null : "response has not been set";
    connection.sendResponse(this);
  }
}

下面是對應的testcase：

// Test that IPC calls can be marked for a deferred response.
// call 0: immediate
// call 1: immediate
// call 2: delayed with wait for 1 sendResponse, check if blocked
// call 3: immediate, proves handler is freed
// call 4: delayed with wait for 2 sendResponses, check if blocked
// call 2: sendResponse, should return
// call 4: sendResponse, should remain blocked
// call 5: immediate, prove handler is still free
// call 4: sendResponse, expect it to return
@Test(timeout=10000)
public void testDeferResponse() throws IOException, InterruptedException {
  final AtomicReference<Call> deferredCall = new AtomicReference<Call>();
  final AtomicInteger count = new AtomicInteger();
  final Writable wait0 = new IntWritable(0);
  final Writable wait1 = new IntWritable(1);
  final Writable wait2 = new IntWritable(2);
  
  // use only 1 handler to prove it's freed after every call
  Server server = new Server(ADDRESS, 0, IntWritable.class, 1, conf){
    @Override
    public Writable call(RPC.RpcKind rpcKind, String protocol,
        Writable waitCount, long receiveTime) throws IOException {
      Call call = Server.getCurCall().get();
      int wait = ((IntWritable)waitCount).get();
      // 根據傳入的wait次數值，做postponeResponse的處理，意爲這個call需要做額外對應次數的sendResponse方法纔會有結果返回
      while (wait-- > 0) {
        call.postponeResponse();
        deferredCall.set(call);
      }
      return new IntWritable(count.getAndIncrement());
    }
  };
  server.start();
  
  final InetSocketAddress address = NetUtils.getConnectAddress(server);
  final Client client = new Client(IntWritable.class, conf);
  Call[] waitingCalls = new Call[2];
      
  // calls should return immediately, check the sequence number is
  // increasing
  assertEquals(0,
      ((IntWritable)client.call(wait0, address)).get());
  assertEquals(1,
      ((IntWritable)client.call(wait0, address)).get());
  
  // do a call in the background that will have a deferred response
  final ExecutorService exec = Executors.newCachedThreadPool();
  Future<Integer> future1 = exec.submit(new Callable<Integer>() {
    @Override
    public Integer call() throws IOException {
      return ((IntWritable)client.call(wait1, address)).get();
    }
  });
  // make sure it blocked 
  try {
    future1.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future1.isDone());
  waitingCalls[0] = deferredCall.get();
  assertNotNull(waitingCalls[0]);  // proves the handler isn't tied up, and that the prior sequence number
  // was consumed
  assertEquals(3,
      ((IntWritable)client.call(wait0, address)).get());  // another call with wait count of 2
  Future<Integer> future2 = exec.submit(new Callable<Integer>() {
    @Override
    public Integer call() throws IOException {
      return ((IntWritable)client.call(wait2, address)).get();
    }
  });
  // make sure it blocked 
  try {
    future2.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future2.isDone());
  waitingCalls[1] = deferredCall.get();
  assertNotNull(waitingCalls[1]);  // the background calls should still be blocked
  assertFalse(future1.isDone());
  assertFalse(future2.isDone());  // trigger responses
  waitingCalls[0].sendResponse();
  waitingCalls[1].sendResponse();
  try {
    int val = future1.get(1, TimeUnit.SECONDS);
    assertEquals(2, val);
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }  // make sure it's still blocked 
  try {
    future2.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future2.isDone());  // call should return immediately
  assertEquals(5,
      ((IntWritable)client.call(wait0, address)).get());  // trigger last waiting call
  waitingCalls[1].sendResponse();
  try {
    int val = future2.get(1, TimeUnit.SECONDS);
    assertEquals(4, val);
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  
  server.stop();
}

樣例二：在RPC Call內添加延時回覆標記

第二種改動方法相對就比較直接了，直接在RPC call內標明是是否需要將此RPC call的請求回覆行爲變爲延時回覆模式。

此優化源自Hadoop社區JIRA HADOOP-11552：Allow handoff on the server side for RPC requests。

相關核心改動如下：

/** A generic call queued for handling. */
public static class Call implements Schedulable,
PrivilegedExceptionAction<Void> {
...
// 是否需要做請求延時回覆的標記
rivate boolean deferredResponse = false;
...

@InterfaceStability.Unstable
public void deferResponse() {
  this.deferredResponse = true;
}

@InterfaceStability.Unstable
public boolean isResponseDeferred() {
  return this.deferredResponse;
}

// 以下兩個方法在實際使用中需要被覆寫
// 延時返回正常結果方法
public void setDeferredResponse(Writable response) {
}

// 延時返回錯誤response結果
public void setDeferredError(Throwable t) {
}

請求call的處理方法

    @Override
    public Void run() throws Exception {
      ...
      try {
        // 1）執行請求處理操作，並得到結果值
        value = call(
            rpcKind, connection.protocolName, rpcRequest, timestampNanos);
      } catch (Throwable e) {
        populateResponseParamsOnError(e, responseParams);
      }
      // 2）如果不需要做延時回覆處理的話
      if (!isResponseDeferred()) {
        long deltaNanos = Time.monotonicNowNanos() - startNanos;
        ProcessingDetails details = getProcessingDetails();

        details.set(Timing.PROCESSING, deltaNanos, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKWAIT, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKSHARED, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKEXCLUSIVE, TimeUnit.NANOSECONDS);
        details.set(Timing.LOCKFREE, deltaNanos, TimeUnit.NANOSECONDS);
        startNanos = Time.monotonicNowNanos();

        // 3）設置請求response信息並返回response
        setResponseFields(value, responseParams);
        sendResponse();

        deltaNanos = Time.monotonicNowNanos() - startNanos;
        details.set(Timing.RESPONSE, deltaNanos, TimeUnit.NANOSECONDS);
      } else {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Deferring response for callId: " + this.callId);
        }
      }
      // 否則是延時返回處理，直接返回此方法
      return null;
    }

以下是一個簡單的延時回覆處理Server的testcase：

public class TestRpcServerHandoff {

  public static final Log LOG =
      LogFactory.getLog(TestRpcServerHandoff.class);

  private static final String BIND_ADDRESS = "0.0.0.0";
  private static final Configuration conf = new Configuration();


  public static class ServerForHandoffTest extends Server {

    private final AtomicBoolean invoked = new AtomicBoolean(false);
    private final ReentrantLock lock = new ReentrantLock();
    private final Condition invokedCondition = lock.newCondition();

    private volatile Writable request;
    private volatile Call deferredCall;

    protected ServerForHandoffTest(int handlerCount) throws IOException {
      super(BIND_ADDRESS, 0, BytesWritable.class, handlerCount, conf);
    }

    @Override
    public Writable call(RPC.RpcKind rpcKind, String protocol, Writable param,
                         long receiveTime) throws Exception {
      request = param;
      deferredCall = Server.getCurCall().get();
      Server.getCurCall().get().deferResponse();
      lock.lock();
      try {
        invoked.set(true);
        invokedCondition.signal();
      } finally {
        lock.unlock();
      }
      return null;
    }

    void awaitInvocation() throws InterruptedException {
      lock.lock();
      try {
        while (!invoked.get()) {
          invokedCondition.await();
        }
      } finally {
        lock.unlock();
      }
    }

    void sendResponse() {
      deferredCall.setDeferredResponse(request);
    }

    void sendError() {
      deferredCall.setDeferredError(new IOException("DeferredError"));
    }
  }

  @Test(timeout = 10000)
  public void testDeferredResponse() throws IOException, InterruptedException,
      ExecutionException {


    ServerForHandoffTest server = new ServerForHandoffTest(2);
    server.start();
    try {
      InetSocketAddress serverAddress = NetUtils.getConnectAddress(server);
      byte[] requestBytes = generateRandomBytes(1024);
      ClientCallable clientCallable =
          new ClientCallable(serverAddress, conf, requestBytes);

      FutureTask<Writable> future = new FutureTask<Writable>(clientCallable);
      Thread clientThread = new Thread(future);
      clientThread.start();

      server.awaitInvocation();
      awaitResponseTimeout(future);

      server.sendResponse();
      BytesWritable response = (BytesWritable) future.get();

      Assert.assertEquals(new BytesWritable(requestBytes), response);
    } finally {
      if (server != null) {
        server.stop();
      }
    }
  }

  @Test(timeout = 10000)
  public void testDeferredException() throws IOException, InterruptedException,
      ExecutionException {
    ServerForHandoffTest server = new ServerForHandoffTest(2);
    server.start();
    try {
      InetSocketAddress serverAddress = NetUtils.getConnectAddress(server);
      byte[] requestBytes = generateRandomBytes(1024);
      ClientCallable clientCallable =
          new ClientCallable(serverAddress, conf, requestBytes);

      FutureTask<Writable> future = new FutureTask<Writable>(clientCallable);
      Thread clientThread = new Thread(future);
      clientThread.start();

      server.awaitInvocation();
      awaitResponseTimeout(future);

      server.sendError();
      try {
        future.get();
        Assert.fail("Call succeeded. Was expecting an exception");
      } catch (ExecutionException e) {
        Throwable cause = e.getCause();
        Assert.assertTrue(cause instanceof RemoteException);
        RemoteException re = (RemoteException) cause;
        Assert.assertTrue(re.toString().contains("DeferredError"));
      }
    } finally {
      if (server != null) {
        server.stop();
      }
    }
  }

  private void awaitResponseTimeout(FutureTask<Writable> future) throws
      ExecutionException,
      InterruptedException {
    long sleepTime = 3000L;
    while (sleepTime > 0) {
      try {
        future.get(200L, TimeUnit.MILLISECONDS);
        Assert.fail("Expected to timeout since" +
            " the deferred response hasn't been registered");
      } catch (TimeoutException e) {
        // Ignoring. Expected to time out.
      }
      sleepTime -= 200L;
    }
    LOG.info("Done sleeping");
  }

  private static class ClientCallable implements Callable<Writable> {

    private final InetSocketAddress address;
    private final Configuration conf;
    final byte[] requestBytes;


    private ClientCallable(InetSocketAddress address, Configuration conf,
                           byte[] requestBytes) {
      this.address = address;
      this.conf = conf;
      this.requestBytes = requestBytes;
    }

    @Override
    public Writable call() throws Exception {
      Client client = new Client(BytesWritable.class, conf);
      Writable param = new BytesWritable(requestBytes);
      final Client.ConnectionId remoteId =
          Client.ConnectionId.getConnectionId(address, null,
              null, 0, null, conf);
      Writable result = client.call(RPC.RpcKind.RPC_BUILTIN, param, remoteId,
          new AtomicBoolean(false));
      return result;
    }
  }

  private byte[] generateRandomBytes(int length) {
    Random random = new Random();
    byte[] bytes = new byte[length];
    for (int i = 0; i < length; i++) {
      bytes[i] = (byte) ('a' + random.nextInt(26));
    }
    return bytes;
  }
}

Server端的請求延時回覆需要根據實際的場景進行運用，並不是說異步延時回覆的方式就比RPC同步等待response結果的方式好。上述相關代碼的改動感興趣的同學可閱讀下文對應JIRA的鏈接。

引用

[1].https://issues.apache.org/jira/browse/HADOOP-10300
[2].https://issues.apache.org/jira/browse/HADOOP-11552

提高RPC Server throughput的請求延時回覆處理

文章目錄

前言

Server端延時請求回覆的優劣勢

Hadoop RPC Server的請求延時回覆處理

樣例一：增加延時請求計數的處理

樣例二：在RPC Call內添加延時回覆標記

引用

釘釘打卡速度慢

Nginx R31 doc 官方文檔-01-nginx 如何安裝

Qt/C++音視頻開發74-合併標籤圖形/生成yolo運算結果圖形/文字和圖形合併成一個/水印濾鏡

挑戰程序設計競賽 2.2章習題 POJ - 3617 Best Cow Line 貪心

字節面試：MySQL什麼時候鎖表？如何防止鎖表？

.NET8連接SQL SERVER 2008 R2 報：證書鏈是由不受信任的頒發機構頒發的

golang開發環境搭建(win10)

python計算機視覺學習筆記——PIL庫的用法

Golang初學：獲取程序內存使用情況，std runtime

HDFS Rolling Upgrade的實現要點分析

Alluxio基於冷熱數據分離的元數據管理策略

存儲系統元數據管理演變升級

Ozone的Erasure Coding方案設計

Ozone數據寫入過程分析

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結