Secret Sharing in a Pub/Sub Overlay

One of our major concerns is that Byzantine brokers may arbitrarily process the encrypted publication messages using a compromised decryption key. Therefore, it is imperative that the key should be protected from the Byzantine brokers. Here, we employ Shamir’s secret sharing scheme [23]. With this technique, a secret can be split into n shares in such a way that at least k shares are needed to reconstruct the original secret. In other words, even if up to k-1 shares are compromised, the original secret cannot be re-generated. This is called the (k, n) threshold scheme with the constraint that k > 1 and n ≥ k. In our context, the secret is the key necessary for subscribers to decrypt the publication messages that are encrypted by publishers. In this section, we focus on the case where the Byzantine brokers reside along the end-to-end path between publishers and subscribers. We show how the original decryption key should be split by a publisher assuming that there is up to f number of Byzantine brokers at the next immediate hop. Then, we explain how the split secret shares should be propagated towards the interested subscribers.

Initial Key Split.

Suppose the publisher P₁ sends out a publication p₁ via broker B₁ as shown in Fig 2. Assume that B₁ is the Byzantine broker and drops p₁. In order to prevent the loss of messages, a redundant path can be established via replica . Duplicate publications can be sent through the redundant paths so that at least one message is guaranteed to be forwarded towards the subscribers. With the replica, the pub/sub overlay becomes tolerant to a single failure of reliable delivery. As a generalization, if there are f failures at each hop on an overlay, at least f + 1 replicas are needed on that hop. These replicas form a group that we call a virtual node.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. A simple broker replication example for handling the case where a Byzantine broker violates the reliable publication delivery requirement.

https://doi.org/10.1371/journal.pone.0158516.g002

For decryption keys, we use the secure in-band key delivery strategy. Similar to the previous case, we need to add broker replicas in order to tolerate node failures. However, the simple replication technique we used in the previous example is not sufficiently safe for the case of transferring the decryption key. The difficulty stems from the fact that it is impossible to perfectly detect whether a broker is Byzantine or not. In order to prevent the Byzantine brokers from obtaining the decryption keys, we choose to employ the secret sharing technique to safely split the decryption keys into multiple shares, initially at the publishers. Assume that there are r replicas in the virtual node V which is the next hop of publisher P. The brokers in V to which the publishers are directly connected are referred to as publisher-edge brokers. A secret can be split into r shares by a publisher, and these shares can be evenly distributed among the r replicas at V. Having only one secret share, each replica cannot reconstruct the original secret. However, we cannot rule out the possibility of multiple Byzantine brokers colluding to collect a sufficient number of shares required for the reconstruction of the original secret. Also, assuming that (k, n) threshold scheme is used by the publishers, the f Byzantine brokers among the publisher-edge brokers may drop k secret shares. Even if other non-Byzantine brokers correctly deliver k-1 secret shares to the authorized subscribers, those secret shares are not sufficient for reconstructing the secret decryption key. Based on this observation, we have to first assume that the number of Byzantine brokers should be less than k in order to prevent these brokers from breaking the (k, n) threshold scheme and the requirement of reliably delivering the secret decryption key to the authorized subscribers.

This assumption is more formally stated in Assumption 1 as follows.

Assumption 1 At every virtual node V, there are brokers that are non-Byzantine.

For example, if there are 5 brokers in a virtual node, we assume that there are up to 2 () Byzantine brokers and there are at least 3 () non-Byzantine brokers.

Assumption 1 reflects the maximum fault-tolerance we aim to achieve. This is a reasonable assumption given the following threat model. Each server running a broker replica follows independent authentication and authorization process, thus a security breach on a particular server does not immediately and/or automatically lead to another security breach on other servers.

Given Assumption 1 and the (k, n) threshold scheme, we have to ensure that k secret shares among the n secret shares must be delivered only to k non-Byzantine brokers. In other words, if there are f Byzantine brokers, there have to be at least f + 1 additional replicas that are non-Byzantine brokers. Therefore, the threshold-scheme to be used at the publisher can be expressed as (f + 1, 2f + 1). Alternatively, we can express the threshold scheme as (, n) where n is the number of replicas in the next-hop virtual node. For example, if there are 5 publisher-edge brokers at the next hop of a publisher P, then a (3, 5) threshold scheme should be used by P.

However, the initial key split alone does not guarantee that the secrete shares can be safely delivered to the subscribers beyond the publisher-edge brokers. We articulate this problem further in the following subsection.

Propagation of Secret Shares.

Before we present the problem of reliably propagating the secret shares to the subscribers, we enlist key notations as follows.

• V: A virtual node
• V_f: A virtual node with forwarding brokers
• V_r: A virtual node with receiving brokers
• prec(V): A virtual node that precedes V, e.g., V_f precedes V_r
• |V|: The number of brokers in virtual node V
• reconstruct(S): A predicate that returns true if a currently received set of split secret shares S can be used to reconstruct the secret split at the preceding virtual node V_p.
• nByz: A non-Byzantine broker
• Byz: A Byzantine broker
• nByz(V): A set of non-Byzantine brokers in V
• Byz(V): A set of Byzantine brokers in V
• B(V): A set of all brokers in V

The first challenge is to prevent Byzantine brokers from tampering with the secret shares it received from the previous hop. Such tampering can be trivially prevented with a well-known security measure such as digital signature for checking the integrity of the message on the subscriber side.

A more challenging task is to ensure that no more than k shares end up at any single broker down the publication delivery path when (k, n) threshold scheme is enforced. If a broker that received k shares happens to be Byzantine, then it can drop all the shares, leaving only k-1 shares at the virtual node. In such a case, the original key cannot be reliably reconstructed on the subscriber side. If multiple Byzantine brokers collude each other to collect at least k shares, then these Byzantine brokers can reconstruct the secret and abuse it. Therefore, it is important to make sure that Byzantine brokers at every hop do not collectively receive more than k shares.

As the very first step, we impose a basic secret share propagation scheme called (k, n) threshold propagation. The implementation of this propagation scheme is provided in Algorithm 1. This algorithm is designed in such a way that no broker on the next-hop virtual node receives more than k shares among the n split secret shares.

Algorithm 1: Deterministic (k, n) threshold propagation scheme

 /* Input: n initial shares generated with (k, n) secret sharing scheme */

1 foreach Forwarding broker B_f ∈ V_f do

2  x = total secret shares B_f has;

3  foreach share i ∈ x do

4   foreach Receiving broker B_r ∈ V_r do

5    t = total received secret shares of B_r;

6    if t < k then

7     send i to B_r;

8     t = t + 1;

Algorithm 1 has a serious limitation since a forwarding broker B_f can violate the scheme and arbitrarily send its share to a receiving broker at the next hop, which breaks the requirement that a receiving broker must not receive more than k shares out of n original split shares. Consider the following example in Fig 3. Publisher P₁ sends out secret shares to brokers B₁, B₂ and B₃ at the virtual node V₁. Now suppose the secret shares have to be relayed to the succeeding virtual node V₂ that also has three replicas. In an ideal case, each broker in V₂ should receive just one share. However, assume that B₃ of V₁ and B₂ of V₂ happen to be Byzantine brokers, as shown in Fig 3(A). B₃ of V₁ may ignore the secret share propagation policy and forward its shares to B₂ of V₂. Upon the receipt of the two shares, B₂ of V₂ may either reconstruct the secret key or intentionally drop the keys. In order to prevent this case, we may consider strengthening the secret sharing scheme at V₁ as follows. The threshold is increased from (2, 3) to (3, 5), and two more replicas (B₄ and B₅) are deployed at V₁ as shown in Fig 3(A). In this way, B₂ at V₁ cannot reconstruct the secret, and other non-Byzantine brokers can safely forward the three shares that are sufficient for reconstructing the original secret. However, we can trivially come up with a case where this new threshold scheme also fails. As shown in Fig 3(B), suppose that B₁ instead of B₃ at V₂ turns out to be the Byzantine broker. Also, suppose that another Byzantine broker B₃ of V₁ forwards its share to B₁ of V₂. These examples show that it is not possible to prevent the situation where k shares arrive at an arbitrary single broker at any hop. There are two reasons for this. First, the brokers at the forwarding node cannot detect with certainty whether a replica in the next hop is Byzantine. Second, Byzantine brokers can yield an arbitrary behavior such as sending shares to any replicas on the next hop.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The issues with the propagation of secret shares through multiple hops.

https://doi.org/10.1371/journal.pone.0158516.g003

As an alternative, we considered splitting the original secret using linear network coding [25]. With linear network coding, a secret is split into n encoded blocks. We distribute the n encoded blocks according to the propagation scheme in Algorithm 1. This is seemingly a stronger mechanism for preventing the Byzantine brokers to illegally reconstruct the original secret, as the Byzantine brokers need the entire n encoded blocks to decode the original secret. If (k, n) threshold scheme is used on the other hand, then only k+1 shares are needed for the Byzantine brokers to reconstruct the original secret. If we assume that the majority of the forwarding brokers on a virtual node is non-Byzantine, then it is impossible for a Byzantine broker to receive all n encoded blocks as long as the non-Byzantine brokers abide by the propagation scheme described in Algorithm 1. The linear network coding paired with Algorithm 1 may exhibit a higher fault-tolerance. However, this version of secret propagation can also fail. For example, as shown in Fig 4(A), suppose a decryption key is encoded into three blocks, c₁, c₂ and c₃. Assume that c₂ and c₃ reach B₂ of V₂ as the Byzantine broker B₃ of V₁ arbitrarily sends his blocks of code to B₂ instead of B₃ of V₂. There is no concern that B₂ of V₂ will be able to reconstruct the original decryption key. However, this broker may arbitrarily forward all the encoded blocks to B₁ of V₃ at the next hop, which leads to a situation where all the necessary encoded blocks are collected. If B₁ of V₃ turns out to be Byzantine, then this broker can reconstruct the key and use it for any malicious intent.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. An example of failed delivery of encoded publication messages and an example of guaranteed reliable propagation of secret shares.

https://doi.org/10.1371/journal.pone.0158516.g004

Note that the aforementioned propagation scheme above splits the original secret only once at the publisher. We now opt for splitting a secret share further down the path, and we prove that this is the most viable solution. For example, as shown in Fig 4(B), after B₁ of V₁ receives a secret share s₁ from publisher P₁, it further splits the share into three sub-shares as s₁₁, s₁₂ and s₁₃. Then, B₁ of V₁ relays those further split secret shares to the next hop V₂.

This propagation scheme called iterative secret propagation is implemented in Algorithm 2.

Algorithm 2: Iterative secret share propagation scheme

 /* Input: n initial shares generated with (k, n) secret sharing scheme */

1 foreach Forwarding broker B_f ∈ V_f do

2  x = total secret shares B_f has;

3  foreach share i ∈ x do

4   S(i) = List of secret shares by splitting i with (|V_r|—|Byz(V_r)|,|V_r|) threshold scheme;

5   x = first index of V_r;

6   foreach i′ ∈ S(i) do

7    send i′ to x’th broker in V_r;

8    x = x+1;

We prove that under Algorithm 2 Byzantine brokers in a virtual node cannot receive a sufficient number of secret shares to reconstruct the original secret. Before we proceed with the proof, we set a few additional key assumptions.

Assumption 2 Publishers behave correctly.

In contrary to Assumption 2, publishers can attack publish/subscribe overlay. For example, Wun et al. presented the possibility of publishers participating in the denial-of-service attack [26]. However, note that this paper is focused on handling the issues with Byzantine brokers, and devising the security measures against the malicious publishers is not in the scope of this paper.

Assumption 3 Non-Byzantine brokers abide by the message propagation rules.

Now we prove that Theorem 1 holds if Algorithm 2 is enforced by the brokers,

Theorem 1 A broker B in V_r cannot receive more than sets of secret shares from V_f such that, for every set S B received, reconstruct(S) holds.

Proof 1 Suppose B in V_r received sets of secret shares from V_f, such that, for every set S B received, reconstruct(S) holds. This occurs only if Byzantine brokers in V_f violate the protocol in Algorithm 2 that a broker must distribute its split shares S′ in such a way that reconstruct(S′) does not hold (as enforced in Algorithm 2: 8–10). This implies that the majority of the brokers in V_f are not non-Byzantine. Therefore, it contradicts Assumption 1.

Theorem 1 states that Byzantine brokers cannot reconstruct a secret unless they receive all split secret shares, which is not possible given our assumptions.

Theorem 2 Non-Byzantine brokers in V_r receive secret shares from the non-Byzantine brokers in V_f that precedes V_r which are sufficient for reconstructing original secret S generated by a publisher PUB.

We prove Theorem 2 by mathematical induction as follows.

Proof 2 Basis: There is only one virtual node between PUB and the interested subscribers.

Assume that the number of secret shares the non-Byzantine brokers in V_r receive is less than . This implies that the publisher did not generate a sufficient number of secret shares. Hence, this contradicts Assumption 2.

Inductive Step: Assume that non-Byzantine brokers in i’th consecutive virtual nodes from PUB receive m secret shares in total that are sufficient for reconstructing the original secret S. We show that in the subsequent virtual node V_i+1, non-Byzantine brokers receive a sufficient number of secret shares to reconstruct S.

Assume that the total number of secret shares the nByz(V_i+1) received is less than ()() from the Byz(V_i). Note that every non-Byzantine broker nByz in V_i must generate a total of |V_i+1| secret shares for every secret nByz received from the B(V_{i − 1}). We assumed that a correct number of secret shares are received by the non-Byzantine brokers up to i’th virtual node as the inductive step. Therefore, the assumption that the non-Byzantine brokers in V_i+1 received less than ()() implies that at least one non-Byzantine broker in V_i generated less than for one of the shares it received from B(V_{i − 1}). This also means that the non-Byzantine broker violated the rule specified in Algorithm 2, and therefore it contradicts Assumption 3.

Theorem 2 states that non-Byzantine brokers are guaranteed to always forward a set of secret shares that are sufficient for reconstructing the original secret at the authorized receiver. Finally, we can derive Theorem 3.

Theorem 3 A non-authorized subscriber (SUB2) that is not entitled to the messages published by a publisher PUB cannot receive a sufficient number of secret shares from the brokers to reconstruct the original key S generated by PUB. An authorized subscriber (SUB1) that is entitled to the messages published by PUB must receive a sufficient number of secret shares to re-construct the original key generated by PUB.

Proof 3 Basis: There is only one virtual node V between PUB and the two subscribers, SUB1 and SUB2.

Assume that the number of secret shares SUB1 receives is insufficient to re-construct S. This assumption implies that non-Byzantine brokers in V failed to send sufficient number of secret shares. Therefore, this assumption contradicts Theorem 2.

Assume that the number of secret shares SUB2 receives is sufficient to re-construct S. This assumption implies that Byzantine brokers in V were able to collude each other to send a sufficient number of secret shares to SUB2 for re-constructing S. However, the number of secret shares PUB sent to Byz(V) is less than . Therefore this assumption contradicts Assumption 2 and Assumption 1.

Inductive step: Assume that Theorem 3 holds when there are i consecutive virtual nodes between PUB and the two subscribers, SUB1 and SUB2. Show that Theorem 3 holds when there are i + 1 virtual nodes between PUB and the two subscribers, SUB1 and SUB2.

Assume that the number of secret shares SUB1 receives via V_i+1 is insufficient to reconstruct S. This assumption implies that non-Byzantine brokers in V_i+1 failed to forward a sufficient number of secret shares to SUB1. This contradicts with Theorem 2.

Assume that the number of secret shares SUB2 receives is sufficient to reconstruct S. This can occur only when the brokers in V_i+1 sent their shares to SUB2. This indicates that the majority of the brokers in V_i+1 is Byzantine, which contradicts Assumption 1.

In order to reconstruct the original key, a subscriber should know how many virtual nodes the secret shares traversed and how many replicas are allocated at each virtual node. The number of virtual nodes corresponds to the number of reconstructions to apply on the received secret shares. The number of replicas at every virtual node gives a subscriber the necessary information about what threshold scheme to apply when executing the reconstruction. Publishers and brokers tag these pieces of information to the secret shares while the brokers forward them down the end-to-end path towards the subscribers.

Propagation of Encrypted Content

So far, we have introduced a decryption key propagation method that is applied to a pub/sub overlay with replicated brokers. Note that replicas introduce multiple alternative routes through which the encrypted content can be forwarded. Typically alternative routes offer opportunities for traffic load-balancing. However, in our context, those routes entail a new issue with the reliable delivery of publication content itself. Given the next hop virtual node with n replicas, a forwarding virtual node can prepare n duplicates of the encrypted content in order to guarantee the reliable delivery. Replicating the content in such a way down the path towards the interested subscribers can significantly increase network traffic. To avoid this problem, we can opt for sending only one publication to one of the replicas and re-transmit the publication in case it gets lost. However, re-transmitting the publication may incur non-negligible delay. In order to reduce the redundant traffic and delay caused by the re-transmission, we can encode and split the file into multiple blocks and then send them out at the same time through multiple paths. Only the missing blocks need to be re-transmitted. There can be a situation where all the n blocks end up in the hands of Byzantine brokers at a certain virtual node. This is still safe because the blocks are encrypted and can only be decrypted with the keys that are transferred securely through the secret share propagation method that we devised in the previous section. For the case of the Byzantine brokers corrupting the blocks, publishers and subscribers can use a digital signature mechanism to check the integrity of each block. Overhead of this approach is measured in the evaluation section of this paper.

Replica Placement Framework

In practice, fully replicating every node in a pub/sub overlay may not be feasible due to cost and limited budget. Therefore, we devise a framework that directs the placements of replicas strategically on the most appropriate locations in a pub/sub overlay for the efficient usage of resources. In our framework, we allow administrators to explicitly specify the criteria for the replica placements. These criteria are mainly broken into two categories. The first criteria specifies the reliability factor (R). The second one specifies the performance factor (P). With the placement of additional replicas, a pub/sub overlay becomes more fault-tolerant. On the other hand, the addition of the replicas can degrade performance since secret sharing at the replicas increases the latency and traffic, which potentially leads to congestion. To strike the balance between the two contradicting problems above (i.e., reliability versus performance), we have devised a 3-phase allocation method. The input to this method is the set of the end-to-end paths between all publisher and subscriber pairs. Given n nodes in an overlay, there can be at most n(n − 1) end-to-end paths. In the first phase, our framework allocates replicas based on the reliability criteria. A priority is assigned to every end-to-end path. The priority (ρ) is measured as a product of the following metrics on the end-to-end path: (1) path length measured as the number of hops; (2) failure frequency ratio over a fixed period of time (γ) and (3) user-defined weight (ω). The failure frequency ratio (γ) is the fraction of the number of failures that have occurred on an end-to-end path over the total number of failures occurred on all end-to-end paths. The weight (ω) indicates the importance of an end-to-end path, and the user (the administrator) can freely assign a numeric value to it. The replicas are allocated proportionally to ρ.

In the second phase, the replicas allotted for each end-to-path are now distributed among the nodes that constitute the end-to-end path. The replicas are distributed proportionally to the failure frequency ratio within the end-to-end path. This frequency ratio of a node on the end-to-end path is measured as the fraction of the number of failures by the node over the total number of failures among all the nodes within the end-to-end path. Fig 5 illustrates a sample placement of replicas after the completion of the second phase for the end-to-end paths between publisher P and the subscribers S₁, S₂ and S₃. Assume that the ρ values for the paths, P − S₁, P − S₂ and P − S₃ are 2, 3 and 5, respectively, are given. Given 10 available replicas in total, the number of replicas for each path is determined in the first phase, as shown in the table in Fig 5. In the second phase, replicas are assigned to the nodes based on their individual failure frequency ratio. We observed a couple of interesting things about this phase. First, there can be cases where a virtual node consists of only two replicas. In such cases, secret sharing cannot be enforced because we cannot assure that a majority of the nodes will be non-Byzantine. Second, all 3 end-to-end paths intersect at B₁ and B₂. Thus, those two brokers receive a batch of replicas more than once during the execution of the second phase. A possible variation of the second phase is to assign a pack of replicas only once to a node. For example, the 2 packs of replicas can be removed from B₁ and be re-assigned to any under-provisioned nodes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. An example of replica placement for the end-to-end path from publisher P to subscribers S₁, S₂ and S₃.

Bs represent brokers. With the replicas, Bs form a virtual node.

https://doi.org/10.1371/journal.pone.0158516.g005

In the third phase, replicas on each end-to-end path are de-allocated based on how the currently allocated replicas degrade the performance. The main QoS metric we focus on is the latency incurred by the secret sharing scheme. The latency can grow proportionally to the number of replicas. Therefore, the administrators can set the maximum number of replicas that can be assigned per disjoint end-to-end path. If the number of replicas assigned to an end-to-end path in the first phase exceeds the maximum number of replicas allowed, then the replicas can be incrementally removed. Here, we remove the replicas from the node with the least failure frequency ratio first. Note that, this is based on a basic performance model that reflects the correlation between the addition of replicas and the latency metric. Our framework can be extended to employ more sophisticated analytical performance models such as queueing network models [28].

Dynamic Replica Deployment Protocol

In this section, we provide a protocol for flexibly re-deploying brokers across the pub/sub broker overlay at runtime. This re-deployment protocol involves attachment and/or detachment of brokers. This protocol is designed in such a way to prevent disruptions to the publication delivery service. Upon the attachment or the detachment of replicas, the threshold scheme for secret sharing gets updated among the broker replicas on the virtual nodes.

Before we articulate the re-deployment protocol, we describe the extended broker architecture of the reference pub/sub overlay implementation [20, 21]. As shown in Fig 6, each broker has a single input queue and multiple output queues. Output queues are grouped to be associated with each virtual node in the next hop. Each output queue is designated to a broker replica in the next-hop virtual node. A broker receives secret shares from the previous virtual node through its input queue. When a secret share from the previous hop gets dequeued from the input queue, the broker runs a topic-based matching in order to determine where the secret shares and publications should be forwarded to. The topic does not need to be encrypted as long as it does not reveal private information. However, if the topic has to be encrypted as well, then homomorphic matching techniques have to be used as introduced in [29], which is the subject for future work. Upon the detection of the next virtual node to forward the secret, a broker first splits the received secret share once again.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The architecture of the extended pub/sub broker for secret forwarding.

https://doi.org/10.1371/journal.pone.0158516.g006

Algorithm 3: Broker replica detachment

 /* For a detaching broker B_d: */

1 Sends its own ID to the all forwarding virtual nodes for B_d ();

2 while Not received ACK from every V ∈ and input queue not empty do

3  Keep processing messages in the input queue;

4  Enqueue output messages into appropriate output queues;

5 Flush all output queues;

6 Disconnect from all forwarding and receiving virtual nodes for B_d;

Algorithm 4: Broker replica attachment

 /* For an attaching broker B_a: */

1 Initialize an input queue;

2 Replicate routing state;

3 Configure output group and output queues;

4 Connects with forwarding virtual nodes for B_a;

5 Notifies the forwarding virtual nodes the ID of B_a and a new threshold scheme;

Algorithm 5: Updates at a broker in the forwarding virtual node for a broker B

 /* When received a notification message */

1 if Received detachment notification then

2  Change threshold scheme;

3  Flush the output queue mapped to B;

4  Remove the output queue mapped to B;

5  Send ACK to the next hop;

6 else

 /* When attach notification received:                       */

7  Create and map an output queue to B;

8  Change threshold scheme;

Now we explain the protocol for re-deploying brokers as specified in Algorithm 3, Algorithm 4 and Algorithm 5. Here we provide a couple of definitions that specify a relationship between virtual nodes. A virtual node V_x is a forwarding virtual node for a virtual node V_y if V_y sends publications to V_x. On the other hand, a virtual node V_x is a receiving virtual node for a virtual node V_y if V_x receives publications from V_x. Acknowledgements exchanged between the brokers are denoted as ACK.

For the detachment of a broker B_d, the brokers in the forwarding virtual nodes for B_d have to update the threshold scheme for secret sharing. If B_d is detached, the number of brokers in the virtual node B_d belongs to (denoted as V_Bd) gets decremented by 1. Suppose the threshold scheme running at the forwarding virtual nodes for B_d was originally (, r) assuming that the number of broker replicas is r at V_Bd. Upon the detachment of B_d, the brokers in the forwarding virtual nodes should newly enforce a (, r − 1) threshold scheme. The only disruption is caused when the brokers in the forwarding virtual nodes cease to process incoming messages when the threshold scheme is updated. However, the update process is executed instantly, thus the disruption is negligible. This update task is highly critical to the confidential delivery of secret keys. If the brokers at the forwarding virtual nodes do not pause the processing of incoming messages, then an incoming message may be split with the old threshold scheme. This can cause a case where the majority of secret shares can reach a single broker, which may result in the reconstruction of the secret in case the broker is Byzantine. After the threshold scheme is updated, the brokers at the forwarding virtual nodes continue to process the incoming messages as well as flush the output queue mapped to the detaching broker replica.

Likewise, there is a very brief pause at the forwarding virtual nodes for attaching broker B_a. Similarly, the threshold scheme must change from (, r) to (, r+1). As stated in Algorithm 4:2, the attaching broker B_a has to replicate one of the brokers at the virtual node to which B_a newly belongs (denoted as ). We do not employ any popular VM cloning tools such as REMUS [30] and Snowflock [31] for promptly replicating the VM where the broker to replicate might reside. This is because VM cloning replicates the VM state including the security-sensitive information such as the secret share. Because of the security hole in the VM cloning techniques, we resort to adapting the on-demand replication technique of constructing the routing state of the newly attached broker [24]. Subscription and advertisement topics are the ingredients for constructing the complete routing state at B_a. B_a can ask any broker in the neighboring virtual nodes to forward their subscriptions and advertisements. However, there can be Byzantine brokers at these neighboring virtual nodes. The Byzantine broker may arbitrarily drop or corrupt the advertisements and subscriptions, thus sabotaging the replication effort by B_a. Hence, B_a has to accept only the subscriptions and the advertisements that are sent by the non-Byzantine brokers. Note that B_a cannot identify which broker at is Byzantine. However, we assume that the majority of r brokers are non-Byzantine. Therefore, B_a accepts a subscription or an advertisement only if it is received at least times where r is the number of replicas at . This procedure of checking the validity of subscription and advertisement topics may contribute to the delay in replicating the complete routing state for B_a. However, we employ the technique that allows the publications to be delivered at the neighboring brokers of B_a promptly after matching subscriptions are added at B_a. Thus, publication delivery resumes very quickly. The non-disruptive nature of our dynamic replica deployment protocol is based on the technique developed in [24]. It allows the broker placements to be revised dynamically if necessary, as explained in the previous section.

The Effect of Secret Sharing on Latency and Traffic

Our Java implementation of Shamir’s secret sharing scheme [23] is integrated into the PADRES pub/sub broker. We ran this broker on a machine with Intel Core2 Duo CPU T5550 at 1.83 GHz and 3GB memory. We first measured the number of secret shares as the path length between publishers and subscribers increase. The number of secret shares increases exponentially as the path length and the node fanout increase, as shown in (Fig 7(A) and 7(B)). As mentioned earlier, to reduce the secret shares, publishers can refresh the decryption key for a bulk of publications instead of generating one for every publication. For example, as shown in (Fig 7(C) and 7(D)), the traffic increase can be approximately 10 times less when keys are refreshed every 1GB as oppose to refreshing the keys every 200MB of data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The effect of secret forwarding on latency and traffic with varying node fanout and path length.

https://doi.org/10.1371/journal.pone.0158516.g007

We also measured the latency in splitting a secret into 3 to 10 shares. We measured the latency in reconstructing the secret as well. The secret is either an AES-128 or an AES-256 key. We set the threshold scheme (k, n) where . As shown in Fig 8, the time it took to split the secret was well under 1 ms. The time it took to reconstruct the secret increased proportionally to the number of shares. It took longer to reconstruct than to split the keys. For example, at the maximum of 10 shares, it took just 5.2 ms on average. However, the reconstruction is done only once by the subscriber, and the brokers along the end-to-end path do not involve in the reconstruction. Our scheme requires the secret splitting at every virtual hop. Thus, the secret splitting overhead is incurred at the broker replicas at every virtual hop. This causes the overall end-to-end latency to increase with the number of hops. However, the end-to-end latency does not grow with the number of replicas at every virtual hop, because the secret splitting is done concurrently among the broker replicas.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Performance overhead of secret splitting and reconstruction.

https://doi.org/10.1371/journal.pone.0158516.g008

The Effect of Friends Dispersion

Facebook users may have friends who are dispersed on many geographical locations. That is, some user may have friends scattered all around the globe, and some user may only have friends from the local region. We conducted a set of experiments on the effect of scattered friends on the overhead of our scheme. We generated a fully connected pub/sub broker overlay topology that does not contain any redundant path. This overlay was assumed to be deployed on a wide area network. Given the overlay, we randomly assigned Facebook users to brokers according to a Zipf distribution. The degree of skewness is controlled by the variable α. With a high α value, Facebook users are clustered close together. On the other hand, with a low α value, Facebook users are disperse and relatively far from each other. We first generated a 400-node overlay with the average node fanout of 2. At every node we assigned 3 replicas that follow the (2, 3) threshold scheme. Fig 9(A) shows the cumulative distribution function of the total secret shares that are delivered during an interaction between a publisher and a subscriber via brokers, over a one-month period. With α = 0.5, the median secret shares generated was approximately 80,000. However, the average was 5 million. The pairs of publishers and subscribers that were far apart contributed significantly to this high average. On average, the publishers and the subscribers were 10 hops away from each other with α = 0.5. With a higher α value of 2, the average number of secret shares per interaction dropped sharply to 860 as the publishers and the subscribers were apart 2.6 hops on average. From this result, we affirmed that the end-to-end path length between the publishers and the subscribers affects the overhead of our scheme. This can guide the administrator of pub/sub broker overlays to reduce the number of hops by consolidating the brokers along the publication delivery paths, so that the number of secret shares is reduced. Also, the node fanout can be controlled to adjust the structure of the overlay. For example, the node fanout of the previous overlay were changed to 5, and the number of brokers are kept the same. Fig 9(B) shows that the number of secret shares per interaction was significantly decreased compared to the case in Fig 9(A). For the same α value of 0.5, the case in Fig 9(B) exhibited 939 secret shares generated on average for each interaction. This was a 99% decrease of secret shares compared to the case where the 400-node overlay had a node fanout of 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The CDF of secret shares generated per interaction that randomly takes place on a 400-node pub/sub broker overlay.

https://doi.org/10.1371/journal.pone.0158516.g009

We also measured the proportion of the secret shares in the data received at the subscriber end. Since we did not know exactly what content was exchanged during the interactions, we could only assume the content was of a certain size. Suppose the average volume of the content per interaction was 1MB. There were a total of 1.4 million interactions over the month in the previous example. Therefore, the total throughput over the month was 1.3 TB/month when our scheme was not applied. With our scheme enabled on a network of 400 nodes with α = 0.5 and node fanout of 5, the monthly throughput of secret shares would be approximately 144 GB/month. Hence, the secret share traffic would constitute 11% of the total throughput. This proportion can be a useful indicator of how costly our scheme can be. In order to reduce the proportion of the secret shares, we can refresh the secret key less frequently than refreshing the key for every single message. In Fig 10, the key refresh rate was set to where m is the number of messages ranging from 1 to 10. With the key refresh rate of 0.1 and all other settings kept same as the previous example, the monthly secret share traffic throughput was reduced to 39 GB/month. The traffic was reduced by 79% compared to the case where the key refresh rate was 1. With a lower refresh rate, the overlay becomes more vulnerable to the compromise of secret keys. However, the performance overhead could be greatly reduced. This is a simple approach of dealing with the trade-off between the performance and reliability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. The effect of varying secret key refresh rate.

https://doi.org/10.1371/journal.pone.0158516.g010

Another adaptation technique is to place an out-of-band repository for the secret shares used by multiple subscribers. By using this technique, the secret shares could be significantly reduced compared to the scheme of sending the secret shares in-band, as shown in Fig 10. However, this adaptation technique has different security and reliability implication as mentioned earlier.

The effect of Byzantine brokers

In this section, we show how many re-transmissions can occur under the presence of Byzantine brokers that are randomly chosen from the nodes on the broker overlay network according to the Zipf distribution. We varied the degree of clustering of the Byzantine brokers by varying α which determines the skewness of the Zipf distribution. The higher the α gets, the closer the Byzantine brokers are clustered together. Similar to the previous test cases, we randomly placed the Facebook users on the pub/sub broker overlays according to the Zipf distribution controlled by the value c. We assumed that Byzantine brokers drop messages in order to violate the reliable delivery requirement. We also assume that the failure detection and failover mechanism was not enabled. Given this setting, we replayed the interactions on a 400-node broker overlay with a node fanout of 5, and the result is shown in Fig 11. The first thing we observed is that the increase of re-transmission is not proportional to the increase of the number of Byzantine brokers. This is because a broker may sit on the intersection of multiple end-to-end paths between Facebook users. Therefore, even a small number of Byzantine brokers can affect the pub/sub interactions significantly. When the Facebook users are far apart from each other, there is a higher chance of Byzantine brokers intersecting on different end-to-end paths. However, in a number of cases, re-transmission occurred the most frequently when the Byzantine brokers were moderately dispersed at α = 1. This was because the brokers closer to the core of the overlay network were chosen, thus affecting relatively larger numbers of intersections. The total number of re-transmissions was higher than the number of total interactions we replayed. For example, with α = 1, c = 0.5 and f = 15, the number of required re-transmissions exceeded 2.5 million. This indicates that more than one broker along the end-to-end paths between the Facebook users failed during an interaction. From this experiment, we can see that a small fraction of the brokers can affect the pub/sub overlay and the running services significantly. In order to prevent this, a prompt detection of the Byzantine brokers and failover mechanism should be devised. However, perfectly detecting a Byzantine broker is not practically feasible. A viable solution is to force the brokers to replicate the received piece of content further down the path as secret share is split further down the path. However, the content can be much larger than the secret shares. Thus, the traffic increase caused by this solution can be impractical. Hence, a new solution that addresses the trade-off between an imperfect failover mechanism and the rigorous replication scheme is needed. We plan to develop this solution in the future.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. The number of re-transmissions under the presence of Byzantine brokers.

c indicates how close Facebook users are clustered together. α indicates how close Byzantine brokers are clustered together. f is the number of randomly chosen Byzantine brokers in the overlay network.

https://doi.org/10.1371/journal.pone.0158516.g011