A network formation game for the emergence of hierarchies

We propose a novel network formation game that explains the emergence of various hierarchical structures in groups where self-interested or utility-maximizing individuals decide to establish or severe relationships of authority or collaboration among themselves. We consider two settings: we first consider individuals who do not seek the other party’s consent when establishing a relationship and then individuals who do. For both settings, we formally relate the emerged hierarchical structures with the novel inclusion of well-motivated hierarchy promoting terms in the individuals’ utility functions. We first analyze the game via a static analysis and characterize all the hierarchical structures that can be formed as its solutions. We then consider the game played dynamically under stochastic interactions among individuals implementing better-response dynamics and analyze the nature of the converged networks.


Introduction
Hierarchies are ubiquitous forms of organization. Not surprisingly, early comparative studies between administrative and economic hierarchies, and other studies of hierarchies in the the context of political organization, date back to the beginning of the twentieth century [1,2]. Since then, their study has encompassed a large variety of disciplines like, for example, labor and organizational economics [3][4][5], moral and evolutionary psychology [6][7][8][9], anthropology [10] and sociology [11,12]. We remark that in the economics literature, the importance of hierarchies also comes from the study of firms: the structure of firms can be understood as hierarchical structures in production organizations (e.g., superiors having veto power over subordinates to access productive assets) [13,14]. In general, the empirical study of the emergence of hierarchical structures in social networks has benefited from advancements in the field of sociometry [15], and, theoretically, graph theoretical analysis has proven to be a useful tool for structural analysis of general hierarchical structures [16]. In this paper, we adopt a game theoretical perspective to study the emergence of hierarchical structures through what is known as network formation games. We propose a game and analyze how its play leads to the emergence of hierarchies. The importance of our approach stems from the fact that this game theoretical study allows us to investigate the underlying structures in the individual utilities that promote the emergence of hierarchies from self-interested individuals. Then, the importance of this work is that it defines micro-mechanisms on the individual level that can lead to the emergence of hierarchical structures on the macro-level.
To the best of our knowledge, there is no previous study about the emergence of hierarchical network structures from a network formation perspective. However, we briefly mention that there has been a considerable body of literature on the study of hierarchies from other game theoretical perspectives [17]. For example, there are studies in the literature about allocation rules that take into account explicit hierarchical relationships among the agents (e.g., [18]), and about the analysis of games where agents have veto power over the actions of other players as a result of an underlying hierarchical structure (e.g., [19]).
On the other hand, there is a mature body of work in the network formation game literature related to other applications. Generally speaking, in a network formation game, self-interested or utility-maximizing agents decide to form or severe links (interpreted as some sort of relationship) with other agents so that they can form network structures that maximize some utility or allocation value. The structure of connections resulting from playing the game induces an emergent network whose topology can have different interpretations according to the specific application the game is modeling. The literature on network formation games has a long trajectory which historically started from a cooperative perspective, and has later devised a huge boom from a non-cooperative perspective [17,20]. We present an overview of the literature on network formation games according to a classification of different previous approaches, this classification also introduces relevant technical terms that will be useful in this paper. For other more algorithmic perspectives to the subject in terms of solutions concepts and efficiency, not mentioned in this review, we refer to the textbook treatment of the subject in [21,Chapter 19]. Network formation games have been analyzed via a static analysis, that is, the analysis of solution concepts that describes the emergence of particular "stable" networks as a result of the agents playing the game. Additionally, they have also been analyzed via a dynamic analysis, that is, the study of the emergence of a subset (or possibly all) of the networks defined by the static analysis (also known as equilibrium selection) as the result of some defined dynamic rules that describe the evolution of the agents' actions [22,23]. For example, [24] uses Nash equilibrium as solution concept, whereas [25] uses the solution concept called pairwise stability (first introduced in [26]); and both works establish a dynamical analysis of convergence towards "stable" networks in their respective solution concept. The two solution concepts mentioned above are the ones that have been typically used in applications where the efficiency of the formed network was the main focus of study; however, in general, solution concepts are proposed (and sometimes renamed) depending on the specific application to study from the network formation perspective (whether there is only a static analysis or a dynamic one too). For example, other solution concepts exist like the generalization of pairwise stability called strong stability also studied in the context of network efficiency [27], or like the solution concept introduced in [28] which is particularly tailored for the problem of routing traffic on networks.
Broadly, the literature on network formation games can be classified according to the nature of the agents that play the game and/or the nature of the connections they can establish. We discuss first the possible nature of the agents. Traditionally, a first distinction is whether the agents are non-consensual or consensual. A non-consensual agent i can establish links unilaterally with agent j without j's consent, i.e., without j having to agree to accept such link (see, e.g., [24]); and a consensual i is the one who needs j's consent or approval to establish a link (see, e.g., [25,29,30]). Another second major distinction regarding the agents is whether they all have the same structure in their utility or payoff function, i.e., the rewards or costs of making/maintaining/severing a link is the same for any agent, which is know as the homogeneous case; or whether they have different structures in their utility function, which is the heterogeneous case [31].
Regarding the distinction on the nature of the connections that agents can establish, network formation games can be classified depending on whether the formed network is undirected (i.e., the edge or link between agents i and j has no direction) or directed (i.e., every edge or link has an intrinsic direction, so that an edge can be directed from i to j). The interpretation is that an undirected edge between i and j gives a two-way benefit of flows, i.e., the utility generated by this edge benefits both i and j. On the contrary, a directed edge from i to j has a one-way flow of benefits since the utility generated by this edge benefits i but may or may not benefit j [24,31]. Examples of undirected relationships are friendships, collaborations (e.g., coauthorship), mutual insurance, etc. Examples of directed relationships arise naturally in communication networks (e.g., someone send an email to another person), loan relationships, subordinate-authority relationships, buyer-seller relationships, etc. We refer to [32,Chapter 5] for more discussion on the use of undirected vs directed graphs in game theoretical modeling.
We stress that not all network formation games in the literature have been studied from a dynamic analysis (indeed, all the classifications presented above only have to do with the static analysis aspect). A first classification from a dynamic analysis is to consider whether the agents, at any time-step of the (discrete) evolution of the game, are myopic (e.g., [24,25,30]), i.e., they want to only maximize their current utility at any time step, or, otherwise, foresighted (e.g., [33][34][35]), i.e., they also want to maximize their (expected cumulative) future utility. Myopic agents are a more reasonable assumption in contexts where agents live in a big and/or complex network, where it is virtually impossible for a single agent to completely predict how her actions would influence over the rest of the whole network and thus how current actions will affect future utility.

Contributions
Our first main contribution is to propose, to the best of our knowledge, a first study on the emergence of hierarchical structures from the perspective of network formation games. We propose a game which allows us to study and understand how self-interested or utility-maximizing individuals lead to the emergence of networks with hierarchical structures. In particular, our work shows this through the judicious inclusion of well-motivated hierarchical promoting terms in the utility functions of the agents. We assume all agents are homogeneous, i.e., all agents have the same structure in their utility function and so that performing the same actions (making/maintaining/severing a link) incur in the same costs and rewards for any agent. We also assume that all individuals can be of two types: non-consensual or consensual. For each type, we perform a static analysis of the emergent network structures that satisfy the solution concept proposed for our game and which is closely related to other notions in the classic literature of network formation games. Then, we present a dynamic analysis where we analyze the dynamic formation of these networks as a result of a stochastic process of pairwise interactions among agents playing better-response dynamics.
Our second contribution is a complete static analysis where we show that network structures with different levels of hierarchy emerge as equilibrium networks, i.e., as solutions of the game, for any type of agents. In particular, individuals within the same level or rank of hierarchy in these structures can only have cooperative relationships. Moreover, we provide theoretical results that show the relationship between the structure or topology of the formed equilibrium network and the different parameters in the utility function of the agents. Then, we provide intuitive interpretations of these theoretical results and their implications in realworld or practical scenarios. Finally, we show that all possible equilibrium networks that can be formed by non-consensual agents in any game are a strict subset of the ones that are possible to be formed in the same game by consensual agents.
Our third contribution is to analyze our proposed network formation game when it is played dynamically starting from any fixed initial graph. In such dynamic setting, agents are stochastically chosen to perform an action and they play their better-response dynamics. We show that the networks, being formed dynamically, eventually converge to some equilibrium network in finite time. Finally, we show that for any type of agent, the topology of the converged hierarchical network, besides depending on the parameters of the utility functions, also depends on the particular realization of the underlying stochastic process (i.e., the solution path).
It is important to notice that the scope of our work, as seen in the previous contributions, does not include the analysis of the efficiency or social welfare of the resulting equilibrium networks in the game (e.g., we do not explore problems such as determining the factors that lead to a more/less efficient equilibrium network), and, as mentioned in the conclusion, we leave this analysis as future work. Instead, we focus on the full characterization and generation of equilibrium networks as hierarchical structures, from both a static and dynamic analysis.

Notation and preliminary modeling
Let N = {1, . . ., n} be the finite set of agents, and, throughout the paper, consider n � 3. Let ij ≔ (i, j)2N × N. Any element of N is a node of the directed graph (digraph) G, and G is thus defined by a set of ordered pairs between different elements of N. (Strictly speaking, a graph is a pair (N, E), where N is the node set N and E is the edge set E; however, for simplicity, we adopt a notation traditionally used in the network formation literature (e.g., see [20]) and refer to the graph by its edge set; we let the context specify the node set.) An element ij 2 G is called an edge from i to j.
We define N in ði; GÞ ¼ fj 2 N j ji 2 Gg and N out ði; GÞ ¼ fj 2 N j ij 2 Gg, with the indegree of i being d in ði; GÞ ¼ jN in ði; GÞj. Given G and i 2 N, if N in ði; GÞ ¼ ; (N out ði; GÞ ¼ ;) then i is a source (sink). If ji 2 G and ij = 2 G, then we say ij is a single edge; and, if ij, ji 2 G then we say i and j have an undirected edge. A graph is complete if there is an undirected edge between any pair of nodes.
Whenever there exists a sequence of edges ik 1 , . . ., k m−1 j that connects i with j with no edge appearing more than once and no node appearing more than in two edges, we say that there exists a path of length m between i and j. Whenever j = i in the previous path, we say that this path is a cycle containing i. We denote by C(G) the condensation graph of G, and by Γ(i, G) the connected component of i 2 N. Graph G is acyclic if it has no cycles, i.e., it has no paths of one node to itself. A directed cycle is any cycle that contains at least one single edge.
We refer to Section 6 for basic graph theoretical terminology and concepts that will be used throughout the paper.
1.2.1 New notation and definitions. Given a directed acyclic graph G 0 and a node i of G 0 , the level of i, denoted by ℓ i (G 0 ), is the largest number of arcs or edges from any source node of G 0 to i. Also, for digraph G, the level of i denoted by ℓ i (C(G)), is the largest number of arcs in C(G) from any source of C(G) to the strongly connected component containing i. See Fig 1(a).
We also define N Given an undirected connected component that contains nodes i and j, we say the undi- Finally, throughout the paper, we use the terms agent, node and individual interchangeably, as well as the terms network and graph, and the terms edge and link.

Proposed model and game
In our model, any single edge ij is a pairwise relationship that describes an unambiguous hierarchical or authority relationship where agent j has a higher hierarchy or exercises some authority over i; so we say j is a supervisor of i, and, respectively, that agent i is a subordinate of j. This simple concept has been used extensively in the game theory literature [32]. On the other hand, whenever i and j have an undirected edge, we understand that they have a collaborative or cooperative relationship, and thus, they must share the same hierarchical status and no agent is the subordinate of the other one. Accordingly, we have that N s in ðj; GÞ is the set of j's subordinates, P j (G, i) is the set of j's collaborators who also have i as a common subordinate, and P j (G) is the set of j's collaborators who have the same subordinates as j. See Fig 1(b) and 1(c).

Utility definition
We first specify the costs and rewards that any agent incurs according to our proposed model.
(i). Subordinate-collaborative reward: From an organizational perspective, this represents the positive reward an individual obtains by being the subordinate or collaborator of another competent agent. For example, consider agent i wants to pursue her own goals (e.g., as an employee), then she will seek direction and help from perhaps an equally or better skilled person (e.g., a competent boss or co-worker). This reward can also be interpreted as an exogenous reward (e.g., a salary) that an agent receives by working for or with other people inside an organization. We represent the subordinate-collaborative reward per link by the positive constant γ.
(ii). Hierarchical reward: Individuals who have a higher hierarchical status or position receive an inherent reward for being in a higher position of power. For example, in companies, political organizations, and public institutions, individuals with higher hierarchical positions can exert more influence than those in lower positions (e.g., leaders or directives have more influence over ordinary individuals in the society or over their political base). Then, the reward for these individuals comes from the strength of their influence and the reach of the consequences of their decisions. Mathematically, it is natural that if i and j both appear in some cycle, then agent i has no authority or any hierarchical power over j, and so i must have the same hierarchical reward as j. Therefore, two agents in the same strongly connected component have the same reward. In summary, we model the hierarchical reward for agent i in the graph G as a (strictly) monotonic function which takes as an argument the level of i. Formally, the hierarchical reward for an agent i situated in the network G is We call H the reward function, and we assume it is non-negative and (strictly) monotonic (e.g., we have that H(0)�0). Intuitively, the monotonic property of H means that individuals with higher positions of status are better rewarded.
(iii). Management cost: An individual i who is in charge of a group of people incurs on a management cost directly proportional to the number of her subordinates. There are two possible reasons for this. First, leading or directing more subordinates requires more managerial or logistic work, thus demanding more energy than doing it with a smaller group, since i has to respond to the demands of more people. Naturally, people who collaborate with i (i.e., any j such that ij, ji 2 G) do not incur a management cost to i. Second, from an organizational perspective, a supervisor i is responsible for the group she leads, and she also needs to report to her immediate superiors (i.e., the people, if any, to whom she is a subordinate) about how the people she is responsible for are performing. This reporting task, obviously, requires more work when i is directing larger groups.
Observe that management costs must be reduced if i is part of a team of people who also manage or direct her same subordinates since this team will collaborate with her and thus reduce the workload on i. Formally, we model the management cost as c P k2N s in ði;GÞ 1 jP i ðG;kÞj (observe that this quantity is well-defined) for some positive constant c > 0. The chosen functional form for the management cost captures the qualitative behaviors mentioned above: the larger the number of subordinates of i (i.e., the larger the set N s in ði; GÞ), the larger the cost; and the larger the number of collaborators (i.e., the larger the set P i (G, k) with k 2 N s in ði; GÞ), the smaller the cost. We conclude that the utility of an individual is the sum of her subordinate-collaborative and hierarchical rewards minus her management cost.

Definition 2.1 (Utility function). The utility that an individual i incurs given a graph G is
Clearly, if i has no connections and no other agent is her subordinate in G, then u i (G) = 0. Example 1: We present a couple of graphs for which we compute the utilities of some agents to better illustrate the calculation using Eq (2). Consider Fig 2. If G is as in (a), then If G is as in (b), then

Game formulation and solution concept
Having defined the utility that individuals obtain from their membership in the network G, we now complete the definition of our proposed network formation game in strategic (or normal) form [36]. An agent i 2 N can only perform two types of actions over the network G: establishing an edge ij so that ij 2 G, or severing an existing edge ij so that ij = 2 G. Regarding the agent's type, we say i is consensual whenever i requires j's agreement to establish the link ij; otherwise, i is non-consensual whenever she can perform such action irrespective of j's agreement.
Definition 2.2 (Hierarchical network formation game). A hierarchical network formation game G is defined by the set of homogeneous agents N = {1, . . ., n}, with n � 3, the utility function for any agent i 2 N as defined in (2). Let A i ¼ f0; 1g n be the action space of i 2 N, so that any a i ¼ ða ij Þ 2 A i is an action (or pure strategy) for agent i. Given a graph G, we say agent i has taken action a ij = 1 if ij 2 G; and, otherwise, has taken action a ij = 0 if ij = 2 G. Let (γ, H, c) be the utility parameters that define the payoff of any agent as in Definition 2.1, and a = (a 1 , . . ., a n ) be the action (or pure strategy) profile which takes values from the set A ¼ ðA 1 ; . . . ; A n Þ. Then, we represent the hierarchical network formation game G by the tuple ðN; u; A; g; H; cÞ along with the specification of whether all agents are consensual or non-consensual.
As it is usual in the literature on network formation games, we only consider pure strategies throughout the paper. We also consider that any agent i 2 N is self-interested or utility-maximizing and so she wants to chose the action profile that maximizes her utility (2). Remark 2.1 (About the game's modeling scope). Firstly, we observe that an agent can only add or remove an outgoing edge, i.e., she decides who supervises her or who she wants to work for. This captures scenarios where people can decide to join (or apply for) a specific team at work or some program, and are able to quit when desired. In the case of non-consensual agents, since an agent always accepts an incoming edge, this assumption does not capture scenarios where managers have more power selecting their employees compared to the power employees have in selecting their managers. However, this scenario is captured in the case of consensual agents: agents have the managerial power to select or hire which individuals will be their subordinates from the pool of applicants.
Secondly, we observe that an agent cannot remove an already established incoming edge. When the agents are consensual and agent i wants to form edge ij, j decides whether to accept or refuse to form the edge ij; however, once j accepts and this edge is formed, j cannot removed it at any later time. This models a sense of responsibility on the agents: an agent accepts to supervise

PLOS ONE
an individual (whether the type is non-consensual or consensual) knowing that she will not be able to arbitrarily stop her supervision later. This also models situations where an ex-ante contract protects subordinates from being indiscriminately fired, but does not model scenarios where there is more managerial power in firing employees.
We propose and use the following solution concept for hierarchical network formation games: Definition 2.3 (Equilibrium network). A network G is an equilibrium network when, for any i, j 2 N, • for consensual agents:

Remark 2.2 (Connection to other solution concepts).
The concept of an equilibrium network becomes the concept of strict Nash equilibrium whenever the agents are non-consensual, and it becomes the concept of pairwise stability when the agents are consensual (with the difference that our concept employ strict inequalities in contrast to the definitions of pairwise stability found in other works, e.g., [26]).
If there exists some edge ij with i, j 2 N that can be added (or severed) so that , so that some condition on the utilities of i as described in Definition 2.3 is violated, then we say that i has an incentive or intention to establish (or sever) the link ij respectively. Obviously, when any agent has an incentive to severe some link, then G is not an equilibrium network for any type of agents. However, if agent i has an incentive to establish a link, then G is not an equilibrium network for non-consensual agents, but may or may not be one for consensual agents (recall that for consensual agents, the addition of the edge ij also depends on j).
We conclude by providing some further interpretations and intuition on the agents' types. A non-consensual agent i can be thought of being "greedy" because i accepts any agent that intends to be her subordinate or collaborator. Intuitively, we can think of greediness from two perspectives: as a bounded rationality aspect, since i can accept more subordinates even though this would mean a negative impact on her utility; or perhaps as a foresightedness feature of i, since i may accept any subordinate with the hope of having a larger future hierarchical reward. On the other hand, we can think of a consensual agent i as being "non-greedy" since she does not want to acquire more subordinates than what she can really handle, i.e., if acquiring more subordinates will affect her utility negatively. Remark 2.3 (About the homogeneity assumption). Our network formation game assumes that all agents have the same structure in their utility function, i.e., there is an assumption of homogeneity in the nature of the agents. We recognize this assumption could be a strong one for individuals in a society; however, we provide two reasons to motivate such assumption.
Firstly, the homogeneous case is the simplest possible setting to study a new concept related to network formation-in our case, the emergence of hierarchies. Indeed, in the development of the literature on network formation games, the homogeneous case has inexorably been the first or default choice for studying a new concept without worrying about "tweaking" multiple model parameters (such as different individual costs and rewards for every agent) to obtain relevant behaviors in the game.
Secondly, since the simpler homogeneous case is a particular case of the heterogeneous case, we can expect that proving the emergence of hierarchical structures in the former case implies the emergence in the latter one. Then, future works can study the expansion of our results to the heterogeneous case, where, for example, we could study how different individual costs and rewards can be mapped to specific topologies of hierarchical structures formed by our game. For example, we could obtain results on how an agent can change her own preferences and skills (and so her utility structure) in order to change her social status inside the emerged hierarchy (e.g., individuals who aim to be in high-ranked managerial positions against ones who stay in lower-level positions of subordination).

Hierarchical structures
We introduce a useful class of networks that abstracts and represents the concept of hierarchies. Intuitively, agents with a higher level have higher positions of power, and formally, this is true, since agents with a higher level have a greater hierarchical reward (see Eq (1)). When k different connected components have the same level ℓ, we say there are k components per level ℓ. If the maximum level that any agent in the network has is ℓ, then we say the network has ℓ + 1 levels.

Definition 3.1 (Hierarchical structures). A (weakly) connected digraph G is a hierarchical structure if each connected component is an undirected subgraph, and it is additionally a sequential hierarchy if there is only one connected component per any level, and, if there is more than one level, then any node has a single edge towards any other node of a higher level.
A hierarchical structure G has the particularity that any of its connected components are composed of agents that form strong collaborative units or teams, i.e., all agents in a connected component are collaborators. Thus, G is naturally a hierarchical representation of the organization, power and/or influences among teams of people (with the understanding that a team may be composed of a single individual). Now, a sequential hierarchy G is a hierarchical representation in which there is only one team at each rank or level of the hierarchy. Observe that a particular trivial case of a hierarchical structure which is also a sequential hierarchy is when the network is complete, i.e., all agents form a unique team. See  , as an example of a digraph G which has its condensation graph C(G) represented by the graph with white-colored nodes on the left, and which has a subgraph corresponding to the nodes A and B of C(G) represented by a graph with gray-colored nodes on the right. In (a) we have a particular example of a hierarchical structure which is not a sequential hierarchy, whereas in (b) we have a hierarchical structure which is a sequential hierarchy. In both cases, all agents corresponding to node A in C(G) have level 2, whereas all agents corresponding to node B in C(G) have level 1. https://doi.org/10.1371/journal.pone.0255990.g003

PLOS ONE
Observe that a pyramidal structure is formed by a hierarchical structure with two or more levels, generally with one connected component per any level, and such that the number of agents that have a lower level is greater than the ones that have a higher one. This type or class of graphs is easily found in real life hierarchies; for example, military organizations (there are less generals than colonels, but less colonels than lieutenants, etc.) and clerical hierarchies in the Catholic church [37] (there is only one Pope, and more bishops, but there are less bishops than priests) have this type of hierarchies. Moreover, it is known that many companies naturally adopt a pyramidal structure for its management organizations [38], with the CEO or president at the top, followed by a small executive leadership or vice-presidents, followed by tiers of middle managers, and all the way down to the lowest level employees.

Solution analysis
Now, we introduce the main results for this section. All relevant proofs can be found in Section 7.

Theorem 3.1 (Non-consensual agents). Consider non-consensual agents. A network G is an equilibrium network for some game G if and only if it is a sequential hierarchy that satisfies the following conditions:
(i). If the network has k > 1 levels, then the following condition holds: (ii). If there exists some critical edge ij, ji 2 G, then Moreover, there exist sequential hierarchies that are not equilibrium networks for any hierarchical network formation game. Remark 3.2 (Interpretation of Theorem 3.1). The importance of Eqs (3) and (4) is that they relate the agents' utility parameters (γ, H, c) with the specific topology of the sequential hierarchy G resulting from the hierarchical network formation game with non-consensual agents.
We start by interpreting Eq (3) H(N − 1)). Moreover, this reflects a natural organizational phenomenon: the salary difference between higher positions in an organization are more accentuated than between lower positions. (3) is satisfied for greater values of the management cost parameter c, then it is possible to find a greater number of agents with level ℓ. This has an intuitive explanation: more agents can be found on level ℓ because this implies that any agent in this level has more collaborators with whom to reduce the management cost that otherwise would be larger due to the larger value of c.

A second observation of Eq (3) is the following: if inequality
We now interpret Eq (4). First observe that, since the equilibrium networks are sequential hierarchies, all connected components are complete subgraphs. Therefore, whenever there is a critical edge, it must belong to a connected component of two nodes. Then, Eq (4) says that agent i with level ℓ and who has a critical edge, has no incentive in severing its collaborative tie with j if, for example, her subordinate-collaborative reward γ is large enough to counteract the increase on the hierarchical reward that she would get by severing her critical edge. This condition seems restrictive because it implies an upper bound on Δ(H), however, we remark that this condition is only present whenever a critical edge exists, i.e., whenever a group of only two people exist for some level. Indeed, later in this paper, in Theorem 4.4, a result is presented where agents dynamically play the hierarchical network formation game and a network with no critical edges is formed.

Theorem 3.3 (Consensual agents). Consider consensual agents. A network G is an equilibrium network for some game G if and only if G is composed by one or multiple hierarchical structures and G satisfy the following conditions:
(i). If a hierarchical structure in G has k > 1 levels, then for any i, j 2 N belonging to this hierarchical structure and such that ji 2 G, with ℓ i (C(G)) = ℓ � k − 1 and ℓ j (C(G)) = ℓ − m for any appropriate 0 � m � ℓ.
(ii). If there exists i, j 2 N such that ℓ i (C(G)) = ℓ � 0 and ℓ j (C(G)) = ℓ − m � 0 for any appropriate 0 � m � ℓ, and such that there exists no path from i to j (whether i and j belong to the same hierarchical structure or not), then Hð' þ 1Þ À Hð' À mÞ < c: (iii). If there exists a critical edge ij, ji 2 G such that ℓ i (C(G)) = ℓ � 0 and ℓ i (C(G − ij)) = ℓ + 1, then Moreover, there exist hierarchical structures, which are possibly sequential hierarchies, that are not equilibrium networks for any hierarchical network formation game. Remark 3.4 (Interpretation of Theorem 3.3). From Theorem 3.3 we conclude that consensual agents allow the formation of a wider variety of hierarchical structures than non-consensual agents. Indeed, given a game G with non-consensual agents, any of its equilibrium networks is also an equilibrium network for the same game but with consensual agents. We also remark that consensual agents allow equilibrium networks that are not connected. See Fig 4. Therefore, we note that Eqs (5) and (7)  Eq (6) can only be present in the case of consensual agents (i.e., it has no counterpart for nonconsensual agents) for two reasons. The first reason is that it must be satisfied, for example, when the equilibrium network consists of two or more hierarchical structures, or if the equilibrium is a hierarchical structure whose condensation graph has multiple sources. The second reason is that this condition is a result of the non-consensual interaction: considering the statement (ii) of the theorem, if agent i wants to establish a connection with j (observe that i has a higher level than j), j will not consent i's intention of becoming her subordinate if the increase of management cost is larger than the increase of hierarchical reward that j would obtain from having i as a subordinate.
It is easy to construct games with consensual agents in which one or multiple sequential hierarchies (including complete networks) are equilibrium networks. The next proposition gives other non-trivial examples of equilibrium networks for a game with consensual agents. Proposition 3.5 (Examples of equilibrium networks for consensual agents). Consider any hierarchical structure G with more than one level, one complete connected component per any level whose size is greater than two, and such that the only existing edges in the graph are as follows: for any i, j 2 N with ℓ i (C(G)) = ℓ and ℓ j (C(G)) = ℓ + 1, pk 2 G for any p 2 Γ(i, G), k 2 Γ(j, G). Then, (i). G, or (ii). any network composed by two or more (isolated) hierarchical structures with the same properties as G and the additional property that |P i (G)| � |P j (G)| for any i, j 2 N with ℓ j (C(G)) = ℓ i (C(G)) − 1 > 0.
is an equilibrium network for some game with consensual agents. Remark 3.6 (Existence of hierarchical structures that are not solutions of the game). As mentioned by both Theorems 3.1 and 3.3, there exist hierarchical structures that cannot be equilibrium networks of any hierarchical network formation game, i.e., there exist hierarchical

Dynamic analysis
We now explore the emergence of hierarchical structures when agents play a hierarchical network formation game dynamically, which we call the network formation process. (iii). Then: (a). For non-consensual agents: (b). For consensual agents: let first i consider the action a � ij as if she was a non-consensual agent. If ij 2 G(t), then a ij ðt þ 1Þ ¼ a � ij . Now, if ij = 2 G(t) and a � ij ¼ 1, then since i is a consensual agent, we have that a ij (t + 1) = 1 if and only if u j (G(t) + ij)�u j (G(t)) (otherwise, obviously, a ij (t + 1) = 0).

Remark 4.1 (Agents in the network formation process).
The network formation process, as clearly seen in Definition 4.1, implements a better-response dynamics for each agent, in which each utility-maximizing agent plays the game sequentially and myopically, i.e., they maximize their current utility (given the constraint of their actions) without any foresightedness or being forward-looking. We remark that the network formation process implements a better-response dynamics because an agent i can only consider the addition or deletion of a single edge per time step. If, on the other hand, i is able to select the best pure strategy in her action space A i per time step, i.e., being able to alter multiple edges at the same time, we would say the network formation process implements a best-response dynamics.
Note, however, that when the agent's utility is the same irrespective of what action she can take in the next time step, she will prefer changing her action in the next time step (since the action space has only two states, this simply means switching her current action in the next time step). Myopic agents are a natural assumption when networks are big or complex, since in this case it is hard for an agent to trace all the past actions from every other member of the network or predict what the repercussion of her actions will be over the rest of agents. Likewise, myopic agents have the least cognitive and informational requirements about the history of the played game: they only need to remember what happened in the previous time-step. These assumptions have been widely used in the literature, e.g., [24]. We also remark that pairwise selection of agents in dynamic settings of network formation games (as in step (ii) from Definition 4.1) has been widely used in the literature of network formation games, e.g., [25].
Formally speaking, the network formation process induces a homogeneous Markov chain M whose state space is the set of all possible graphs with N nodes (defined by all possible action profiles the agents can take), and whose transition probabilities are immediately defined as an induced measure from the underlying pairwise selection process defined in step (ii) from Definition 4.1. See Fig 5 for an example of a one-step transition in the network formation process. We proceed to analyze the convergence properties of the network formation process. All relevant proofs can be found in Section 8.

Definition 4.2 (Invariant networks). Consider the network formation process. An invariant network G � is any network such that if G(t) = G � then, with probability one, G(t + 1) = G � . Proposition 4.2. A network is invariant for the network formation process if and only if it is an equilibrium network.
Proof. Consider any invariant network G � and any (i, j)2N × N (with i 6 ¼ j). Consider first non-consensual agents. If ij 2 G � then it must follow that u i (G � )>u i (G � − ij); otherwise, if u i (G)�u i (G � − ij), then item (iii) from Definition 4.1 implies that i will change her action in the next time step and thus the topology of G � , which gives a contradiction. A similar proof can be done to show that if ij = 2 G � , then u i (G � )>u i (G � + ij); and by Definition 2.3, conclude that G � is an equilibrium network. Now, for consensual agents, it only suffices to analyze what happens when ij = 2 G � and u i (G � )�u i (G � + ij), i.e., the case in which, agent i's action in the next time step would be 1 if she was non-consensual. In such case, according to item (iii) from Definition 4.1, it must follow that u j (G � + ij)<u j (G � ), so that i has no consent from j and will not change her current action 0 in the next time step. This implies, according to Definition 2.3, that G � is an equilibrium network.
Given an equilibrium network, it is easy to show it is also an invariant network from Definition 2.3 and the fact that only a pair of agents is selected per time step in the network formation process.
Formally speaking, Proposition 4.2 tells us that a network G is an absorbing state of the Markov chain M if and only if it is an equilibrium network. Fig 5. Example of a one-step transition in the network formation process. At some time t � 0, we have an emergent network G(t) with its condensation graph G C (t). Assume the pair of agents (i, j) is chosen at time t by the network formation process, with i such that Γ(i, G(t)) corresponds to node A and j such that Γ(j, G(t)) corresponds to node B in G C (t). Then G C (t + 1), which is the condensation graph of G(t + 1), is as shown in the figure for non-

PLOS ONE
convergence to an equilibrium network is proved, a second natural problem is the further exploration of the specific structures or topology of the formed equilibrium networks.

Theorem 4.4 (Convergence of the network formation process). Consider the network formation process, with an arbitrary initial network G o , for a game G with a fixed type of agents and such that H(ℓ + 1) − H(ℓ)>γ + cN for any integer
With probability one, G(t) one, converges in finite time to an equilibrium network with no critical edges and such that it is connected (according to the conditions stipulated in Theorem 3.1 or Theorem 3.3 depending on the type of agents).
The following corollary follows from the proof construction of Theorem 4.4. Remark 4.6 (Analysis of results). There are important remarks to make about the convergence results for consensual agents. In general, the final or converged network from the network formation process (for any type of agents) depends on both the initial network and the specific realization path of the underlying stochastic process, i.e., the final network is a random function of the initial conditions. This is known as path dependence [24]: depending on the initial graph, there may be a positive probability of converging to more than one invariant network, provided this is allowed by the utility parameters and the topology of the initial graph. See Example 2 for an illustration of this phenomenon. Finally, notice how starting the network formation process from an empty graph greatly reduces the final topology of the possible converged equilibrium network (e.g., compare Corollary 4.5

to Theorem 4.4).
We provide a small example to illustrate the observations made in Remark 4.6 regarding path dependence.

Conclusion
In this paper we propose, to the best of our knowledge, the first network formation game model that explains the emergence of hierarchical structures in a society. We consider both non-consensual and consensual agents, and study the different topologies of hierarchies that can emerge as solutions of a game. Moreover, we develop a network formation process in which the agents play stochastically the game sequentially and myopically, and characterize the possible final formed networks. We learn that the introduction of hierarchical rewards that increasingly benefit the agents of higher hierarchy or power, is crucial in determining both the total number of levels in the final hierarchical structure and the maximum number of individuals that can share the same level. Moreover, we learn that it is plausible that individuals within the same level cooperate with each other.
Our paper motivates future work on the analysis of hierarchical structures, like, for example, the proposal of new models that can display either a narrower or a wider variety of other forms of hierarchies than the ones presented in this paper. One way to possibly achieve this, is by the judicious incorporation of heterogeneous agents in the game that may have different structures of subordinate-collaborative or hierarchical reward; see, for example, Remark 2.3. Another possible future work is to define different dynamic processes of network formation that perhaps can drive the agents to different equilibrium selection outcomes and study why such phenomena may happen.
As pointed out in Remark 2.1, a direction of future research is to extend our proposed game to allow agents to severe incoming connections from her subordinates. Moreover, a second research direction is to study the effect of stability of the equilibrium networks of the game, and further study the conditions that could lead more or less efficient equilibrium networks in terms of the game parameters. Finally, from an organizational point of view, it is relevant to theoretically understand under what conditions are flat or steep hierarchies preferable in terms of the game's parameters.

Basic graph theoretical definitions
Consider a directed graph (digraph) G. Graph G is strongly connected whenever there exist a path between any two of its nodes; and, by convention, any graph composed by a single node is in itself strongly connected. We say that i and j are weakly connected when, if we consider all edges of G being undirected, there exists a path from one of these nodes to the other, and we say that G is (weakly) connected if any two nodes are (weakly) connected. If G is not connected, then any connected subgraph such that no other node of G is connected to any of its nodes, is called an isolated network.
A (strongly) connected component of G is a subgraph of G that is strongly connected and such that no additional edges or nodes from G can be included in the subgraph without violating its property of being strongly connected. We can partition the nodes of G according to which connected component they belong to. The condensation graph of G is the graph resulting from contracting every strongly connected component of G into a single node, and it is an acyclic graph.

Proofs for the static analysis section
We first introduce a couple of useful technical results. Lemma 7.1. For both non-consensual and consensual agents, an equilibrium network cannot have directed cycles, and thus, only has undirected connected components. In particular, for the case of non-consensual agents, all connected components are complete subgraphs.
Proof. By contradiction, consider an equilibrium network G and a single edge ij 2 G such that a directed cycle involving the node i exists with the edge ij belonging to such cycle. We analyze whether j has an incentive to start a collaboration with i. Observe that H(ℓ j (G)) = H (ℓ j (G + ji)), |P j (G, k)| � |P j (G + ji, k)| for any k 2 N s in ðj; GÞ with k 6 ¼ i. Then, Thus, j has an incentive to establish the edge ji. For non-consensual agents, this would contradict the fact that G is an equilibrium network, and so G cannot have single edges in any connected component, which implies that all connected components are undirected subgraphs. Now, assume that agents are consensual and continue the analysis to see if i has an incentive in accepting a link from j. We can easily show that Thus, i will accept collaborating with j, and this immediately implies that G must only have undirected connected components for consensual agents. Now, assume we have any i, j 2 N with ij, ji = 2 G belonging to the same undirected connected component of G and that this connected component is not a complete subgraph. Then, it is easy to show that u i (G + ij) − u i (G) = γ > 0, and so i has an incentive to form the connection ij. For non-consensual agents, this contradicts G being an equilibrium network, from which we immediately conclude that all connected components must be complete subgraphs. Now, consider consensual agents. Then, it is easy to show that u j ðG þ ijÞ À u j ðGÞ ¼ À c 1 jP j ðG;iÞj < 0, and so j has no incentive in accepting a single edge from i. Thus, the connected components of G are not necessarily complete for consensual agents.
Corollary 7.1. Consider any graph G without directed cycles. Then, no agent has an incentive to sever any single edge.
Proof. Given the graph G, consider any i, j 2 N such that ij 2 G and ji = 2 G; and observe that Now, we introduce the important proofs of our Solution Analysis section. Theorem 3.1. Throughout the proof, let T i ðGÞ ≔ jN out ði; GÞjg. We denote by i ! j the fact that there exists a path between i and j, and, with a slight abuse of notation, we will also use it to denote the set of edges that compose some path between i and j, so that clearly i ! j � G. (Note that there might be multiple paths between i and j and so there is some ambiguity as to which specific path i ! j refers to, but this will not be a problem throughout the paper).

Part I:
We first prove that any given sequential hierarchy as stipulated in items (i) and (ii) of the theorem statement, is the equilibrium network for some game G. Consider a given sequential hierarchy G. Now, we choose any c, γ > 0 and fix their values. Then, the remaining problem is to find a reward function H so that G is an equilibrium network for the game G ¼ ðN; u; A; g; H; cÞ. Then, for G to be an equilibrium network, we need to ensure that no agent is able to establish new links or severe existing ones as pointed out in Definition 2.3. Let k be the number of levels of G.
If k = 1, then the network is complete with u i (G − ij) = γ(n − 2)+ H(0) − c/(n − 1) and u i (G) = γ(n − 1)+ H(0) for any i, j 2 N; which gives u i (G − ij) − u i (G) = −γ − c/(n − 1)<0 since n � 3. Thus, there is no incentive for any agent of the network to severe a link, irrespective of the function H. Thus, any non-negative monotonic function H can be chosen as the reward function.
Let us assume now that we have k > 1. Case 1: Assume i has level 0 < ℓ � k − 1 and that she considers establishing a link with some agent j that has level ℓ − 1 (recall that ij 2 G), which we call an immediate backward , and thus Having this previous expression being less than zero would suffice for a non-consensual agent to not have an incentive to establish immediate backward links, which is the same as choosing a function H that satisfies condition (i) of the theorem statement (observe that S must be, indeed, strictly increasing since the sequence fjfj j ' j ðCðGÞÞ < 'gjg kÀ 1 '¼1 is also strictly increasing for k > 2).
Case 2: Assume i has level 0 < ℓ � k − 1 and that she considers establishing a connection with an agent j that has level ℓ − m for m > 1 (assuming, obviously, that such j exists). Let q ℓ (G) = |{j|ℓ j (C(G)) = ℓ}| be the number of agents that have rank ℓ in the graph G. Then, Now, if (3) holds, then À Hð' À pÞ À Hð' À ðp þ 1ÞÞ < À g À c q 'À p ðGÞþjfj j ' j ðCðGÞÞ<'À pgj q 'À p ðGÞðq 'À p ðGÞþ1Þ � � � for any 0 � p � ℓ − 1; from which it follows that ðHð' À pÞ À Hð' À ðp þ 1ÞÞÞ < 0; and so i has no incentive to establish such new link. In conclusion, if condition (i) from the theorem statement holds, then no agent has in incentive to establish a link with any agent that has a lower level. Case 3: Assume i has any level 0 � ℓ � k − 1 and that she considers severing any of her existing edges. From Corollary 7.1, we know that she has no incentive to sever any single edge. Now, consider any agent j such that ij 2 G belongs to some undirected edge. If the undirected edge is a critical edge, i.e., |P i (G)| = 2, then it is easy to show that Having this previous expression being less than zero would suffice for i to not have an incentive to severe the edge ij, which is the same as choosing a function H that satisfies condition (ii) from the theorem statement. On the other hand, if the undirected edge is not a critical edge, so that |P i (G)| > 2, then it is easy to show that u i ðG À ijÞ À u i ðGÞ ¼ À g À c jfj j ' j ðCðGÞÞ<'gjþjP i ðGÞj jP i ðGÞjðjP i ðGÞjÀ 1Þ < 0; and thus, i has no incentive in severing the edge ij.
In conclusion, from all the previous analyzed cases, if G is a sequential hierarchy with more than one level, then we only need to define the non-negative monotonic function H as stipulated by the conditions (i) and (ii) of the theorem statement in order to finally obtain the sought game G such that G is an equilibrium network for G.
Part II: Now, we need to prove that any equilibrium network is a sequential hierarchy as stipulated in items (i) and (ii) of the theorem statement. Consider the given equilibrium network G. From Lemma 7.1 we immediately conclude that G is a hierarchical structure with complete connected components. Note that this lemma does not specify that G must be connected. However, from the proof of the same lemma, if we assume G is not connected, then there is always an incentive for some agent to form a single edge with an agent from another complete connected component of the same level, which by contradiction implies that there is only one connected component per any level in G. Now, we claim that any agent in G must have single edges towards any other agent with a higher rank than her. Assume this is not the case and take any i, j 2 N such that ij = 2 G and ℓ i (C(G)) < ℓ j (C(G)), then it follows easily that i has an incentive to form links with j since such action would increase her utility by γ > 0, giving a contradiction. Thus, in observance of Definition 3.1, we have proved that G is a sequential hierarchy. It should be noted that, from our previous analysis in Part I, the number of agents in each complete connected component of G (when there is more than one level) is determined by the utility parameters (γ, H, c) satisfying the conditions (i) and (ii) of the theorem statement. This concludes the proof that any equilibrium network is a sequential hierarchy as stipulated in items (i) and (ii) of the theorem statement.
Finally, assume we have a sequential hierarchy G such that there exists some level ℓ > 1 such that: 1) for any i 2 N that has level ℓ, we have jfj j ' j ðCðGÞÞ < 'gj � 2jP i ðGÞj 2 2þjP i ðGÞjþjP i ðGÞj 2 ; and 2) for any p 2 N that has level ℓ − 1, we have |P p (G)| = 2. Then, we claim that G cannot be an equilibrium network for any game. To see this, observe condition 1) implies , which immediately implies both (3) and (4) cannot hold simultaneously because of condition 2), for any utility parameters (γ, H, c). Proof of Theorem 3.3. Throughout the proof, let T i ðGÞ ≔ jN out ði; GÞjg. We adopt the same notation introduced at the beginning of the proof of Theorem 3.1, and we additionally introduce the following notation: given nodes i, j 2 N, let i ! not j denote the fact that there does not exist any path between i and j in G, and let i − j denote that i and j are (weakly) connected, i.e., i ! j or j ! i. Part I: We first analyze the conditions under which any given hierarchical structure, as in items (i), (ii) and (iii) of the theorem statement, is the equilibrium network for some game G. Consider a given network G that has one or more hierarchical structures. The remaining problem is to find the utility parameters (γ, H, c) so that G is an equilibrium network for the game G ¼ ðN; u; A; g; H; cÞ. Now, we analyze the conditions that ensure no agent has an incentive and/or is able to form a new link or severe any existing link, so that G can be properly defined as an equilibrium network of G.
Case 1: Assume any agent i that has level 0 < ℓ � k − 1, with k > 1 being the number of levels in the hierarchical structure for which i belongs to, is considering establishing a link with any agent j that has rank ℓ − m for any appropriate m � 1, i.e., establish a backward connection. First, assume ji 2 G. Then, following a similar procedure as for Case 1 of Part I in the proof of Theorem 3.1, we obtain u i ðG þ ijÞ À u i ðGÞ ¼ g þ Hð' À mÞ À Hð'Þ þ c P GÞ, and thus j would accept such proposition. Then, the sufficient and necessary condition for not incentivizing a backward connection from i is the satisfaction of condition (i) from the theorem statement. Second, assume ji = 2 G and j ! i. Then, it is easy to see that . Assume the worst case, i.e., i has an incentive in making such connection. Then, we find that u j (G + ij) − u j (G) = −c < 0, and so j will not accept a new link from i. Third, assume ji = 2 G and j ! not i (notice that i and j may or may not belong to the same hierarchical structure). Then, it is easy to show that i has an incentive in establishing such link, and that for j we have u j (G + ij) − u j (G) = H(ℓ + 1) − H(ℓ − m) − c, and so j will not accept such connection if and only if Hð' þ 1Þ À Hð' À mÞ < c: Case 2: Assume any agent i that has level 0 � ℓ < k − 1, with k > 1 being the number of levels in the hierarchical structure for which i belongs to, considers making a forward connection with any agent j that has rank ℓ + m for any appropriate m � 1. We already know from Part I in the proof of Theorem 3.1 that i has an incentive to establish a link with j. If m = 1 and i ! j, then it follows that u j ðG þ ijÞ À u j ðGÞ ¼ À c jP j ðG;iÞj < 0. If m = 1 and i ! not j, or m > 1 (whether i − j or not), then u j (G + ij) − u j (G) = −c < 0. Then, we conclude that j has no incentive in accepting a connection from i. Case 3: Assume any agent i that has level 0 � ℓ � k − 1, with k � 1 being the number of levels in the hierarchical structure for which i belongs to, considers making a connection with any agent j of rank ℓ (so that ij, ji = 2 G). Then, we already know i has an incentive for making such connection. If j 2 Γ(i, G), then we have u j ðG þ ijÞ À u j ðGÞ ¼ À c jP j ðG;iÞj < 0 and thus j will not accept such link from i. If j = 2 Γ(i, G) (whether i belongs or not to the same hierarchical structure as j), then we obtain u j (G + ij) − u j (G) = H(ℓ + 1) − H(ℓ) − c and so j has no incentive in accepting such link from i if and only if Case 4: Assume any agent i that has any rank ℓ considers severing any of her existing edges. From Corollary 7.1, we know that she has no incentive to sever any single edge. Now, consider any agent j such that ij 2 G belongs to some undirected edge. If the undirected edge is a critical edge and i can increase its level by severing the edge ij, then it is easy to show that have an incentive to sever the edge ij, which is expressed in Eq (7) from condition (iii) of the theorem statement. On the other hand, if the undirected edge is not a critical edge, then |Γ(i, G)| > 2 and it is easy to show that u i ðG À ijÞ À u i ðGÞ ¼ À g À c P Notice that enforcing inequalities (9) and (10) amounts to the condition (ii) of the theorem statement. Therefore, if there exist utility parameters (γ, H, c) that satisfy the conditions (i), (ii) and (iii) from the theorem statement, we conclude that the given sequential hierarchy is an equilibrium network.
Part II: Now, consider a given equilibrium network G. From Lemma 7.1 we immediately conclude that G is a hierarchical structure or is composed of isolated hierarchical structures (since this lemma does not specify whether G is connected or not). Then, from our previous analysis in Part I, it follows that the utility parameters (γ, H, c) must be such that the conditions (i), (ii) and (iii) in the theorem statement are satisfied. This concludes the proof that any equilibrium network is composed by one or multiple hierarchical structures.
Finally, we prove the existence of hierarchical structures that are not equilibrium networks of any possible hierarchical network formation game. For the case in which the hierarchical structures are sequential hierarchies, we can use the same construction given in the proof of Theorem 3.1. Now, we focus on the case in which the hierarchical structures are not sequential hierarchies. Consider a hierarchical structure G with more than one component per any level and with complete connected components, and such that the only single edges in the graph are as follows: for any i, j 2 N with ℓ j (C(G)) = ℓ j (C(G)) − 1 � 0, it holds pk 2 G (and so kp = 2 G) for any p 2 Γ(i, G), k 2 Γ(j, G). Moreover, also assume |P i (G)| > 2 for any i 2 N, i.e., there are no critical edges in G. We immediately notice from the topology of G and our discussion above that the expression in Eq (5) from condition (i) of the theorem statement becomes for any i, j 2 N with ℓ i (C(G)) = ℓ and ℓ j (C(G)) = ℓ − 1, ℓ > 0. Assume G has more than one hierarchical structure, and that one of them has ℓ + 1 > 0 levels with i p 2 N denoting any agent of such hierarchical structure such that ℓ i (C(G)) = p 2 [0, ℓ − 1]. Consider any level 0 < ℓ � k − 1. Now, observe that condition (i) (using (11)) and condition (ii) imply Hð'Þ À Hð0Þ > 'g þ c Hð'Þ À Hð0Þ < c; respectively. We claim that if there exists some level ℓ such that 1 � P ' γ, c > 0, which makes it impossible to choose a reward function H such that conditions (12) and (13) are satisfied, which in turn violates both conditions (i) and (ii) from the theorem. Proof of Proposition 3.5. We analyze the conditions under which any given network as in (i) or (ii) is the equilibrium network for some game G with consensual agents. Given the network, the remaining problem is to find the appropriate utility parameters (γ, H, c) so that it becomes the equilibrium network for the game G ¼ ðN; u; A; g; H; cÞ.
Consider G to be a given hierarchical structure as in statement (i) with k > 1 levels. First, observe that since all connected components are complete subgraphs with more than two nodes, then there are no critical edges in G, so that we do not need to analyze the satisfaction of condition (iii) of Theorem 3.3. We immediately notice from the structure of G and the proof of Theorem 3.3 that Eq (6) of condition (ii) from Theorem 3.3 becomes Hð'Þ À Hð' À 1Þ > g þ c jP i ðGÞj þ jP j ðGÞj jP i ðGÞjðjP i ðGÞj þ 1Þ with ℓ > 0 and any i, j 2 N with ℓ i (C(G)) = ℓ > 0 and ℓ j (C(G)) = ℓ − 1. Then, it follows immediately that we can first choose any γ, c > 0 and then an appropriate non-negative monotonic function H satisfying Eq (14) to form the sought game. Now, consider G to be a given network as in statement (ii). Observe that all isolated networks in G have more than one level. First, we make the observation that, considering any agent i, j 2 N with ℓ i (C(G)) = ℓ > 0 and ℓ j (C(G)) = ℓ − 1, then, from the conditions of G, the following inequalities hold: since |P i (G)| > 2.
1. eventually makes the graph connected at some time t 1 � t o , i.e., G(t 1 ) has no isolated networks (this is easy to see for non-consensual agents, and for consensual agents, it follows from the fact that inequality (6) in item (ii) of Theorem 3.3 cannot be satisfied under the conditions on the increments of the reward function H), 2. eventually makes the graph to have no directed cycles at some time t 2 � t 1 , 3. taking G(t 2 ), it starts establishing one or multiple single edges that may result in the formation of new directed cycles (if any), and then immediately turn these edges into undirected ones according to (b) (for example, for non-consensual agents, new directed edges may appear when an agent i forms a single edge with another agent j of the same level and such that there is a path from j to i); and repeat this process until the graph has no directed cycles and cannot form new directed cycles at some time t 3 � t 2 ; notice that G(t 3 ) is a hierarchical structure, 4. severs any existing critical edge in G(t 3 ) (i.e., turns it into a single edge), and continues doing this process until no new critical edge is found in the graph at some time t 4 � t 3 (this can always be done since the condition on the increments of the reward function H implies that the inequality (7) is always not satisfied, which, from the proof of Theorem 3.3, let us conclude that any existing critical edge can always be severed).
Notice that the graph G(t 4 ) is still a hierarchical structure, since severing critical edges (as in step 4) does not introduce directed cycles, nor introduce new isolated networks. Since G(t 4 ) is a hierarchical structure and the increments of the reward function H are large enough, there is no incentive for any agent to establish an edge with any agent with a lower level. Notice that for non-consensual agents, step 3) implies all agents with the same level form a complete subgraph. Notice also that for consensual agents, if a single edge is established between two agents i and j with the same level in G(t 4 ), then, for any t � t 4 , this single edge will not be an undirected edge with probability one, nor no new directed cycle that contains i and j will be formed with probability one. Therefore, we conclude that, for any type of agents, no new undirected edges nor new directed cycles can be formed for any G(t)�G(t 4 ) with probability one.
So, given the arbitrary initial network G o , let us define T to be the random time at which G(t) becomes a hierarchical structure as characterized by G(t 4 ). Then, it is easy to observe that T is a stopping time and we can use the first Borel-Cantelli lemma (as in [40,Theorem 3.1]) to prove that T < 1 with probability one, since any finite sequence of pairs of agents that implements the appropriate changes in the steps 1) to 4) (if any) always exists with positive probability at any time t. Then, we can use the strong Markov property, and assume that the network formation process starts at T with the graph G(T). Since only new single edges may appear at future time-steps and these single edges cannot form directed cycles, we conclude that, with probability one, � Gðt þ 1Þ � � GðtÞ for any t � T, i.e., j � Gðt þ 1Þj � j � GðtÞj. Moreover, for any t � T, we conclude that G(t) has a pair of agents that can establish a single edge at time t + 1 if and only if j � Gðt þ 1Þj > j � GðtÞj with positive probability. Now, consider any invariant set S. It follow from these observations that if GðtÞ 2 S, then j � Gðt þ 1Þj ¼ j � GðtÞj with probability one. However, this only occurs if and only if no new single edges can be added, i.e., when jSj ¼ 1.
Then, we conclude that any invariant set is an invariant network and so that the network converges to an equilibrium network.