2.1. Behavioral-science-based intervention-design framework

In a study on illegal fishing, Battista and colleagues [4] observed that insights from behavioral science have the potential to support the design of interventions for modifying specific illicit behaviors. Their framework begins with stakeholder involvement to ensure that the process has a foundation of norms, beliefs, and community thinking about the behavior targeted for intervention. In this way, their process recognizes the power of primary and even primordial prevention [5,6] to disrupt illicit behavior and promote desirable alternative behaviors. Literature and artefactual experiments aid this design process before small pilots and deployment of interventions at scale. Our study [2] extended the original process with additional steps to allow for virtual examination of hypothetical interventions in a simulation environment and machine learning methods (RL specifically) to optimize candidate policies prior to tests and deployment in pilots or scaled-up deployments. Fig 1 illustrates the extended process. It is important to note that the framework is an iterative learning process that adds data and model refinements over time to produce more complete and optimal operating policies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Proposed behavioral and decision science framework that includes artifactual data collection, synthetic data generation, and reinforcement learning to optimize the impact of interventions.

https://doi.org/10.1371/journal.pcsy.0000079.g001

This paper illustrates a practical application of the framework above by examining optimal strategies in the domain of disrupting labor exploitation. This research continues previous work by the authors [2], which focused on designing and evaluating interventions to address labor exploitation using agent-based modeling (ABM). Moreover, our study aligns with the extensive literature advocating for a well-being economy [7–9], generally, and fair employment [10–12], specifically, emphasizing the relevance and importance of our research contribution.

Furthermore, we expand upon the proposed reinforcement learning framework to incorporate elements of dynamic adaptation, enabling the agent to adjust its strategies in response to evolving environmental conditions. By integrating adaptive mechanisms, our framework enhances its utility in dynamic and uncertain settings, presenting a robust approach for decision-making in complex socio-economic systems. This study presents an approach to optimally control complex systems that evolve in discrete time and through the occurrence of discrete events, which is well suited for reinforcement learning [13]. This approach applies to many common real-world problems where data are scarce and the agents in the system have conflicting objectives. When multiple agents apply reinforcement learning in a shared environment, the system’s dynamics move beyond the traditional

Markov Decision Process (MDP) model. In such multi-agent scenarios, the optimal policy for an individual agent is not solely determined by the environment’s dynamics but also depends on the policies adopted by other agents [14].

Given the specialized nature of our environment, which entails specific features and complexities like multi-agent scenarios and event spaces, we opted to construct a custom data generation environment in Python that was designed to mimic observed behaviors in a prior study of day laborers and employers. This approach allowed us to manipulate the environment’s realistic dynamics of a laborer making decisions in response to job offers, which may involve theft and reporting, prior to collecting additional data from the real world environment. The methodology utilizes Q-learning to derive efficient policies to maximize cumulative rewards over time.

2.2. Precarious labor and exploitation

We have previously examined day labor exploitation in [1,2]. Day labor is a form of precarious employment characterized by informality and, frequently, exploitation. Labor historians assert that precarious employment has been common for much of human history [15,16]. Two definitions of precarious work include Standing’s precariat social category (which includes seven forms of labor security) [17] and Cranford et al.’s continuum of security (along four criteria) [18]. We adopted Kalleberg’s definition [19], “employment that is uncertain, unpredictable, and risky from the point of view of the worker.” Kalleberg and Hewison conceptualize precarity as a process instead of a continuum or a category, pointing to the relationships between labor, capital, and the state that produces precarious work [20]. This definition of precarity is most appropriate for our studies because it prioritizes worker perception of precarity and invites inquiry into the formal or informal character of work according to government law and its enforcement.

For our purposes, exploitation refers to a situation whereby taking advantage of another entity, the actor doing the exploitation gains more than they deserve in the interaction, and the exploited entity gets less than they deserve [21]. The distinction between the definitions of precarity, (in)formality, and exploitation is essential for our research because the day laborers we interviewed did not always perceive precarity in their informal work, even if the wage theft they experienced was exploitative, because they viewed it in the context of their migration histories. Such adaptive and vicious cycles are typical in many corrupt and exploitative settings [22–27].

Germane to our case study, laws that define formal or informal labor and the likelihood that they would be enforced are of special concern for the immigrant laborers that we have interviewed. Informal labor describes jobs that do not offer standard terms, conditions, and benefits according to state law because the law does not pertain to these jobs or because the law is not enforced [15]. A review of federal and state laws to address wage theft in the US has shown that federal enforcement is weak and state laws are highly varied [28] even though the Fair Labor Standards Act has been in effect since 1938 [29]. Our interviews [1] revealed that day laborers rarely pursue formal remedies when wage theft is experienced because of perceptions of low success rates after excessive investments of time. It is in this context that we examined how worker centers might play a convening and coordinating role in precarious employment. In fact, worker centers have evolved from formal, community-based, and community-run organizations that provide services, education, and advocacy to laborers [30] to become a response to exploitation [31].

2.3. Application domain for RL: Exploitation in day-laborer community

This study is situated within our ongoing research on precarious labor and complements our previous work which empirically explored day labor dynamics and developed an agent-based model to assess interventions. Specifically, in [1], we reported on three empirical exploratory studies among day laborers in Texas that characterized the context of the day labor experience relative to rates and extent of exploitation, the alternatives available to laborers when they are making decisions to accept work and respond to exploitative outcomes during work, and the potential of worker centers to provide an informal channel for providing educational interventions to both employers and laborers that have the potential to reduce exploitation and provide a means to respond to it when exploitation occurs. In [2], we constructed an ABM informed by the reviewed theories and the artifactual evidence from our three studies in [1]. That simulation environment allowed us to examine select policy changes and assess the sensitivity of various assumptions on the efficacy of those proposals. We concluded [2] with the observation that education for laborers and employers about workers’ rights and the obligations of employers to workers is a critically needed intervention focused on principles of labor organization optimized for the informal setting of day labor. In addition, such training should also cover workplace safety, the second most common form of labor exploitation.

These works inform the current study, which applies the RL method to illustrate how such a model free machine learning approach can observe a partially observable, stochastic system and iterate toward an optimal policy. Once again, we focus on the most common form of exploitation experienced by day laborers: wage theft. Of interest in the current study is the decision to steal is made unilaterally by the employer, typically after the laborer has accepted and begun work. Further, wage theft occurs in a fraction of jobs, and the degree of theft is typically only a fraction of the agreed-upon wages. Fig 2 illustrates the decision process of a prospective day laborer. The black lines represent action-triggered state changes, and the black circled states represent the observed state space. The day laborer begins in an idle state and remains idle until he or she is offered a position by an employer. The laborer then chooses either to accept the position or decline the position. If they accept, they enter the working state. If theft does not occur, they return to idle and have been observed as working fairly. If a theft occurs before the job ends, they enter the observed working theft state and then decide whether to report it. Regardless of the reporting decision, they return to idle with a successful report, increasing their payoff.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Employment journey of a day laborer.

The employment journey of a day laborer as they accept work offered by an employer, move into a working state (fair or unfair) that is partially controlled unilaterally by that employer. Adapted from [2].

https://doi.org/10.1371/journal.pcsy.0000079.g002

The most crucial piece of this decision process that this study is concerned with is the decision of the day laborer to report. In other words, is the expected payoff of putting in a report great enough to constitute going through the reporting process? And ultimately, what can be done to dissuade employers from stealing wages from laborers they hire? We begin with a single-agent version of the problem that examines the decision-making by a laborer before expanding to a multi-agent formulation that encompasses decisions by both the laborer and the employer.

2.4. Stochastic models and reinforcement learning

In complex sociological systems, stochastic models play a pivotal role in facilitating decision-making processes, particularly in scenarios where outcomes exhibit a blend of randomness and controllability by decision-makers. Among the most prevalent stochastic models employed for such purposes is the Markov Decision Process (MDP) [32]. At its core, an MDP consists of a set of states, actions, transition probabilities, and rewards, encapsulating the dynamics of sequential decision-making under uncertainty modeling a single agent. Reinforcement learning (RL) algorithms seamlessly integrate with MDPs, leveraging their structure to learn optimal decision-making policies.

Definition 1: The Markov Decision Process in the single-agent notion is defined by the following set of tuples [33]: {Sn, A₁,..., A_m, r₁,...,r_m, S_n+1} where:

• S is the state of the environment
• A_i,s, i ∈ 1,...,m: denotes the i^th actions associated with state S.
• r_i,s, i ∈ 1,...,m: denotes the reward for the i^th action associated with state S.
• S_n+1 denotes the next state for the environment.

In this study, we wish to observe the driving factors for why a community member may choose to report wage theft or not based on decisions and interactions with other agents. It is reasonable to suppose that the goals of the employer and the day laborer would conflict with each other. We suspect that using Stackelberg games or extensive-form games in non-cooperative game theory may be an effective way to analyze this dynamic [34]. Additionally, efforts have been made to integrate game theory into reinforcement learning, as evidenced by these studies [14,33,35–39]. Moreover, the Markov decision process has also been widely introduced in the field of game theory with L.S. Shapley’s seminal work on Stochastic Games [40,41] and further integrated with more recent studies [42–46]. We develop an extension to this literature in the methods section using an extensive form game modeled as a Markov Decision Process. We provide the formal definition of an extensive-form game from [47] here.

Definition 2: An extensive form game is a list Γ = (N, V, E, x^o, (V_i)_i∈N, O, u) where

• N is the number of players.
• V is a finite set whose elements are vertices of the game tree.
• E ⊆ V × V is a finite set of pairs of vertices and denotes the edges between the vertices.
• x^o∈ V is the root of the game tree.
• (V_i)_i∈N is a partition of the vertices that denotes which agent makes decisions at each vertex.

In the context of this paper, the set of vertices will denote the state space in the MDP framework, the edges represent the respective actions in each state, and a reward is assigned to both players in each state transition. In other words, during the reinforcement learning process, the agent that gets to decide at a particular vertex V observes the current state and the set of all possible future rewards. That agent then chooses their action, which moves both players to the next state. The next player then observes the new state and makes their decision.

RL enhances traditional simulation methods in multi- and single-agent contexts, providing a powerful tool for problem-solving in dynamic or uncertain scenarios [32]. RL leverages simulation models as interactive learning environments, fostering collaboration between the model and the agent to facilitate adaptive decision-making in evolving contexts. At the heart of RL lies the concept of learning from feedback. Reinforcement learning agents navigate an environment, perform an action, and receive feedback through rewards or penalties. This collaborative process uses reward and punishment mechanisms as feedback to guide agents’ behavior toward developing an optimal future policy of actions. The key components of RL include the agent, environment, actions, states, and rewards, which collectively shape the learning process.

Q-learning [48] is a widely used reinforcement learning algorithm to train agents in decision-making within environments. Q-learning operates as a model-free algorithm. The agent interacts with the environment, deriving insights from the outcomes of its actions without constructing an internal model or a Markov Decision Process. Initially, the agent knows the potential states and actions within the environment. Subsequently, the agent uncovers the state transitions and associated rewards through exploration. The Q-value, a key metric in Q-learning, quantifies the expected cumulative return or utility of taking action’a’ in state’s’. This value encapsulates the agent’s understanding of the desirability of different actions in various states. The Q-learning update equation, represented as

(1)

encapsulates the iterative learning process, where α (0 <α ≤ 1) denotes the learning rate, r signifies the immediate reward, γ represents the discount factor, and max(Q(s^′,a^′)) reflects the maximum Q-value for the next state-action pair. In the single-agent case, the following action is chosen by the same agent; in the multi-agent problem, the following action is selected by another agent. The learning rate determines how much the Q-values are updated in each iteration and controls the weight given to new information versus old information when updating the Q-values. On the other hand, the discount factor determines the importance of future rewards and discounts the value of future rewards compared to immediate rewards. A value closer to 1 means future rewards are more equally valued as immediate rewards, while a value closer to 0 means more immediate rewards are valued. The agent refines its understanding of optimal actions in different states by iteratively updating Q-values based on observed rewards and future estimates. As the RL algorithm progresses and the number of samples increases, Q-values converge towards more accurate estimations, facilitating enhanced decision-making and policy optimization by the RL agent.

Reinforcement learning aims to train the agent’s behavior under varying circumstances, typically through trial-and-error exploration. Model-free algorithms, such as Q-learning, tackle the exploration-exploitation trade-off by balancing exploiting known actions for immediate rewards and exploring new actions for potential long-term benefit. In the epsilon-greedy method, the agent combines the exploitation of known actions with occasional exploration of new options to maintain a balance between exploiting prior knowledge and discovering new possibilities. This approach aims to maximize rewards while allowing room for experimentation. Another hyperparameter often controls this balance, typically denoted as ε in epsilon-greedy exploration strategies [49]. Reinforcement learning algorithms, particularly Q-learning, guide decision-making processes in various domains. At the core of RL lies the utility function, which quantifies the desirability of different outcomes and guides the agent towards optimal actions. Traditionally, utility functions in RL have primarily focused on monetary returns as the sole metric for evaluating outcomes. However, in many real-world scenarios, decision-making is influenced by many objectives that extend beyond mere financial gains, an area we return to in our discussion.

As a data-driven learning process, Q-learning explores the system as represented by observation. Within the decision-science extension of a behavioral-science-based intervention-design framework described above, an iterative process, training data can be sampled directly or generated from simulations built from artifactual or scaled-up empirical experiments. Fig 3 illustrates that the environment may encompass various data generation strategies, including the simplified simulation utilized in this study, a more realistic ABM (see [2]), or the real-world context (as studied in [1]). The learned policies can be tested within the simulation to evaluate their effectiveness before real-world deployment. The simulation environment serves as the training setting. The ABM can be adjusted to delegate specific decision points to the learning agent.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The batch Q-learning environment.

The batch QL environment may encompass various data generation strategies, including the simplified simulation utilized in this study as a proof of concept, a more realistic ABM (see [2] for a relevant example that we used in that study of precarious labor and wage theft), or the real-world context (as examined by us in [1]).

https://doi.org/10.1371/journal.pcsy.0000079.g003

3.1. Stochastic models and game theoretic formulation

We present two stochastic game-theoretic formulations of our day labor decision-making problem to illustrate the flexibility of our model-free RL approach to computationally developing policies.

3.1.1. Single-agent formulation.

With our application, we refer to the developments in decision theory under uncertainty. In our context, the prospective day laborer acts as a decision maker (DM) who starts in an ‘idle’ state and then, if offered a job, chooses whether to accept it. If accepted, the DM can decide to report the employer if they experienced wage theft. In this formulation, the theft and degree of theft are random outcomes experienced by the laborer, and the policy formulation goal is to manage those uncertainties through decisions to accept or reject jobs that are offered and to report or not when theft occurs. See Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. State/action diagram for the single agent formulation.

The state, action, and rewards for the single agent formulation, including events that precipitate state changes.

https://doi.org/10.1371/journal.pcsy.0000079.g004

We characterize our decision problem as outlined below.

• The set of possible observed future states Ω = {idle, working, working theft}.
• The set of all alternative actions A = {no action, decline, accept, report}
• The set of possible consequences X = {cost of not working, x(a;ω)}
• Preference relation ≿ over the set of probability distributions over X.

Here, x(a;ω) is a random variable denoting, in monetary terms, the payout to the DM once they work at the job and take action a. This would include the pay for their work accounting for any stolen wages and the potential pain and suffering they gain for a successful report of wage theft. As our consequences are expressed in monetary terms the outcomes are easily comparable assuming that the same day-laborer would prefer higher pay for the same work and would strictly prefer some payment to no payment at all. Therefore, our DM would prefer action a = ‘report’ over action b = ‘no action’ if the expected payout after reporting is at least as good as the expected payout of not reporting. Or in our notation, a ≿ b ⇔ E[x(a;ω)] ≥ E[x(b;ω)]. Whether to report or not is the highlight of what we are investigating.

3.1.2. Multi-agent formulation.

This study also models the same situation as a multi-agent process using an extensive-form game tree that is also traversed with our reinforcement learning algorithm; see Fig 5. In this context, the following extensive-form game is defined. Γ = (N, V, E, x^o, (V_i)_i∈N, O, u) where:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Game tree of the 2-player state space.

In the multiagent formulation, orange denotes the states where the employer makes the decisions and the actions that can be made, black lines represent the states where the day-laborer makes decisions and the actions they can take, and blue denotes the environment and the end results. The game is immediately replayed if the job does not begin and rewards are changed for both players in each state transition.

https://doi.org/10.1371/journal.pcsy.0000079.g005

• N = {0,1,2} where player 1 is the employer, player 2 is the day-laborer, and player 0 is the environment that does not maximize any utility.
• V = {Idle, Job Offered, Job Accepted, Stolen, Report Process, End} denoting the macrostates.
• E ⊆ V × V denotes the actions that the agents can make at each state that trigger the next state change.
• x^o is the idle state where the game begins.
• (V₁) = {Idle, Job Accepted}, (V₂) = {Job Offered, Stolen}, (V₀) = {Report Process} denotes which agent makes decisions at each vertex and “end” is the terminal node where the game ends.

3.2. Computational framework

We present the ABM-based data generation approach used and the model free RL Framework used to demonstrate policy development for the problem domain of precarious labor and interventions to address wage theft in that environment.

3.2.1. Data Generation via agent-based modeling environment.

We have implemented ABM-based environments to generate data for both the single-agent and multiagent formulations to illustrate how the RL methodology we are demonstrating can adapt to a deeper understanding of the problem domain through our iterative framework for intervention design [2].

As part of a demonstration of our iterative framework (Fig 1), the parameters used in our study were specified using summary statistics and other data collected by the authors during previous studies as reported on in [1,2] on day laborer communities in the Houston and Austin Texas metropolitan areas. Those studies resulted in, among other results, the following parameters and empirical estimates involved in the decision process of the day-laborer communities germane to the RL formulation of this study; see Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Empirically estimated system parameters.

https://doi.org/10.1371/journal.pcsy.0000079.t001

It is important to acknowledge that the data collection in these previous, artifactual studies involved interviews with a relatively small (n < 40) non-random sample of immigrant workers. We recognize that this sample size and non-random selection limit the generalizability of these specific parameter values. However, for the purpose of demonstrating the reinforcement learning methodology and its potential for policy development, these empirically derived parameters serve as a foundational starting point, reflecting observed real-world dynamics. Future work will focus on performing a more formal sensitivity analysis to rigorously assess how uncertainty in these estimated parameters propagates through the model and influences policy outcomes. Furthermore, per our iterative framework (Fig 1), subsequent research phases will involve collecting additional, larger-scale empirical data to validate the model’s predictions and refine the policies derived from the reinforcement learning approach.

Laborers, when they find work, typically earn between $80 and $150 per day and are assumed to experience costs of about $60 per day for bare necessities. They experience some degree of wage theft about 30% of the time with the degree of theft averaging about 25% of the wages agreed to at the beginning of the job. Currently, reporting options exist but are highly unlikely to produce an outcome that restores wages to a victimized laborer (we use a 1% success rate). Suppose a report of wage theft produces a positive outcome for the laborer. In that case, there is also the chance that the employer will need to pay punitive damages to the laborer (we assume a $500 value for such damages).

3.2.2. Computational RL framework.

The current study is focused on addressing complex decision-making problems in stochastic stationary environments using reinforcement learning as the primary policy development methodology. Specifically, the paper investigates the application of embedded discrete Markov chains to model decision processes in a real-world context, motivated by previous work by the authors.

Adopting a model-free approach, it is crucial to delineate the framework into two distinct components: the environment model (where data is collected from the real world or generated, as in the current study, from an agent-based simulation) and the reinforcement learning algorithm (where optimal policies are determined). The environment model encompasses the state space, action space, and the resultant rewards stemming from those actions. This study first analyzes the environment around a single laborer agent whose state evolves based on its actions. As discussed above, we then analyze the environment for the multi-agent scenario. Both scenarios involve a laborer receiving job offers with varying degrees of fairness, the possibility of theft, and potential reporting actions. The general process that is followed in this paper is expressed in Fig 6 below, which is elaborated on through the remainder of this section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Extended Q-learning environment.

A simulation environment generates states, actions, transitions, and rewards, which are learned by the reinforcement learning system. An optimal policy is found and used to direct the second round of the data generation process. The second round is then learned, and a new policy is established based on the previous one, indicating the optimal action at each state for a particular environment.

https://doi.org/10.1371/journal.pcsy.0000079.g006

The reinforcement learning algorithm employs batch Q-learning to derive the optimal policy. What sets our reinforcement learning algorithm apart is its generality and adaptability. It can ingest data from diverse sources, whether from complex real-world sociological systems, synthetic data generated from complex, evidence-based, agent-based models (ABM) designed to mimic the real world, or simplified custom environments designed to generate essential data to train the algorithm and produce policies that can be further tested in the real world or sophisticated simulation environments. This adaptability allows the algorithm to dynamically adjust to changes in environmental input data, continually refining its decision-making capabilities. The methodology proposed in this study offers a flexible model approach and learning from various data sources. We can initiate the process with data from simplified environments and construct an initial model. As more real-world data becomes available, the model can be further refined and optimized, illustrating the iterative and adaptive nature of the framework in tackling decision-making complexities within sociological systems.

1. Custom Environment: The custom environment encapsulates the states, actions, and dynamics of the decision-making process. In the environment, when the laborer and employer are in the “idle” state:
   • If the employer decides to offer a job and the laborer opts to “decline” the job offer, both the employer and the laborer face a penalty proportional to the cost of remaining idle, multiplied by the expected duration of remaining idle.
   • Accepting a job offer involves assessing the probability of theft.
   • If the employer decides to exploit, the laborer moves to the “workingTheft” state. In this case, the reward is computed as the daily pay minus a penalty related to the degree of theft, adjusted by the cost of living.
   • Otherwise, if the employer is fair, the laborer transitions to the “workingFair” state, earning the daily pay minus the cost of living.

During employment in the “workingFair” or “workingTheft” states:

• Choosing “noAction” signifies continuing the current job without additional action. At the end of the employment period, the laborer returns to the “idle” state, with no immediate reward but the possibility of future job offers.
• In the case of working under theft conditions, opting to “report” the exploitation may or may not succeed, resulting in different consequences. If successful, the laborer receives compensation for the daily pay lost due to theft, in addition to punitive damages for the pain and suffering. Otherwise, the laborer experiences no immediate reward if the report fails, but the job still ends.

1. Simulation and Data Generation: The script conducts two phases of data collection. In the first pass, feasible actions for each state are determined by sampling experiences with random actions. Subsequently, the main data collection phase uses an epsilon-greedy strategy to balance exploration and exploitation for a specified number of simulations of the abovementioned decision process. The resulting experiences are collected as a data frame, which includes state-action pairs, rewards, and next states. This data is then processed by a separate function that learns the state space and sets of actions before going into the reinforcement learning brain to keep the RL algorithm model free. Algorithm 1 below shows how the code learns the state space that is then read into the reinforcement learning algorithm 2.

Algorithm 1: Learning the State Space and Action Sets

Data (data frame of state, action, reward, next state, where each row is a new observation, could be from an agent-based model, environment function, or real data)

Procedure Learn State Space (data)

 Initialize state_action_dictionary = {}

 For each row in the data frame

  state = row[‘State’]

  action =   row[‘Action’]

  If state is not in the state$_$action$_$dictionary:

   Initialize state_action_dictionary[state] = []

 Append state_action_dictionary[state] to include (action)

Return state_action_dictionary

1. Q-Learning Implementation: The Q-learning algorithm derives optimal policies in the defined environment. The Q-values are stored in a dictionary (Q) representing the expected cumulative rewards for state-action pairs. The Q-learning process involves iteratively updating the Q-values based on the experiences collected during the simulation. In our case study, the Q-values represent the net present value of the maximum expected income for the day laborer if he or she takes the action at that state and continues to make the best actions moving forward. Best, in this case, means maximizing expected income. The Q-values associated with the firm have very similar interpretations.

The Q-learning update equation for the single-agent framework uses the traditional format of Equation 1 in Section 4.4. The multi-agent Q-learning update equation is a slight variation where instead of the agent choosing the next action to take, the other player chooses the next action that is taken.

This Q-learning update equation for player i with opponent j is represented in equation 2 below.

(2)

where the value agent i receives from the current state-action pair depends on the true belief of the action the opponent makes in the next round. The players then continue to take turns until the process is terminated.

The hyperparameters that are used in this study for the Q-Learning implementation include the initial learning rate (α) of 0.1 and the initial greedy action selection probability (∊) of 0.5, which both decay at a rate of 0.99 and both have minimum values of 0.01. We select out discount factor (γ) to be 0.9, a history window of 10 iterations. A sensitivity analysis measuring the effects of changing the initial values of α and ∊ as well as δ on the system is conducted.

The reinforcement learning functions (reinforcement_learning for single-agent and reinforcement_learning_combine for the multi-agent) update the Q-values using the Q-learning update rule above based on the collected experiences. The parameters, alpha and gamma, control the learning rate and discount factor, respectively. The function supports the option to use a pre-existing Q-model, facilitating iterative learning. A key distinction of this algorithm is that the Reinforcement/Q learning processes have no knowledge of any underlying model, decision process, or game tree. Instead, this process’s only inputs are the observed transition rewards after simulation or observing each state-action pair. The algorithm below outlines this process.

Algorithm 2: Reinforcement Learning

Data (learned state_action_dictionary, each observation lists the state, action, reward, and next state)

Procedure Reinforcement Learning (data, alpha, gamma, feasible actions, Q)

 If Q = NULL

  Initialize Q = {}

 For each experience observed in the data

  state =  experience[“State”]

  action =  experience[“Action”]

  reward =  experience[“Reward”]

  next_sate =  experience[“NextState”]

  NewQ[(state, action)] = currentQ(state, action) +

                 alpha * (reward + gamma * (max(Q(next_state, a)))

                 - currentQ(state, action)

                 where a is all feasible actions

Return NewQ

The Q-values obtained from the reinforcement learning process represent the learned policies for the decision-making process. In labor exploitation, the Q-values represent the net present value of expected income received when all future actions are chosen optimally. In the single-agent case, the same agent makes all the best actions. In the multi-agent setting, the competing agent determines the actions taken in the following state; then the process recursively alternates between the two agents. These Q-values provide insights into the optimal actions for each state, considering the stochastic nature of the environment. Higher Q-values suggest more favorable actions in corresponding states, indicating higher expected cumulative rewards. Lower or negative Q-values indicate less favorable actions for a state. A Q-value of zero typically implies that the action is neutral or does not contribute significantly to maximizing the expected cumulative reward.

By analyzing the Q-values, one can identify the optimal policy for the agent, which involves selecting actions that maximize the expected cumulative reward in each state. The RL algorithm updates these Q-values iteratively based on the observed rewards and transitions, aiming to converge toward an optimal policy over time. The results can be analyzed to understand the impact of fairness, theft, and reporting on the cumulative rewards of the laborer.

While batch Q-learning has well established convergence theoretically [32,50–52], in practice, tracking and establishing convergence is challenging and is the subject of ongoing research. Approaches generally focus on examining the observed learning rate of the algorithm, both visually and quantitatively, as well as hyperparameter tuning for α (for the learning rate) and ∊ (for the likelihood of exploring a different policy than the current best action.)

5.2.3. Batch Q-learning convergence diagnostics

Following established approaches to tracking learning rates and convergence discussed in the literature, e.g., [32,50–52], we define three empirical criteria to monitor convergence of Q-values in our batch reinforcement learning setting, each using a fixed-size look-back window.

Mean Change. The average of the absolute differences between consecutive Q-values within the window. Convergence is indicated when this value falls below a threshold τ_mean.

(3)

Variance of Q-Value Change. The variance of the absolute differences between consecutive Q-values within the window. Convergence is indicated when this value falls below a threshold τ_var.

(4)

Percentage of Stable State-Action Pairs. The percentage of Q-value changes within the window that are below a small stability threshold τ_stable. Convergence is indicated when this percentage exceeds a high value (e.g., 95%).

(5)

Where I(·) is the indicator function.

Learning Rate and Exploration Rate Decay The learning rate α and exploration rate ∊ decay over iterations (or runs) as follows:

Where α_min and ∊_min are the minimum allowed values for the respective rates, and decay_rate is the decay factor (e.g., ‘epsilon_decay‘) which is set at 0.99.

Our results, presented next, are based on implementations of the mean absolute change and variance of the Q-value change for the sake of parsimony. In the current study, we decay α and ∊ over the computational iterations such that the algorithm becomes less greedy and less likely to experiment with new policy candidates. We return to these considerations in our results and discussion sections.

4.1. Single-agent formulation

In the single-agent environment, after running the program with the associated parameters with a sample size of 100, we collected Q values from the data creation process. The following results include an initial sample size of 100 as no practical or significant effect that impacted the results was observed in increasing the initial sample step above 100 (1,000, 5,000, etc.). Then, we used those Q values as training data to go back into the environment, run again, and get the Q values after 1,000 iterations of batch Q-learning. To examine which parameters of our environment’s structural model are influencing this near tie between the two Q values, we ran a sensitivity analysis. In Fig 7 below, we test the counterfactual effect of increasing the probability of success beyond the current likelihood of 0.001. The output represents the laborer’s Q-values for reporting or not, expressed through the 1,000th iterations of batch-Q-learning performed at probabilities of 0.001, 0.01, and 0.02 for a successful report. The 95% confidence intervals provided were created using 1,000 bootstrap samples of 1,000 runs of the simulation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Single Agent Q-Values.

The variations in Q-values corresponding to two actions, “Report” and “NoAction,” within the “WorkingTheft” state across different probability values of report success. These results are generated by the Python code S1 File provided in the Supplement.

https://doi.org/10.1371/journal.pcsy.0000079.g007

We see above that as the probability of a successful report increases, so does the expected payoff to the day laborer, and thus, the Q-value of reporting increases approximately linearly. If the day laborer chooses not to report, the expected payoff also increases as the employer does not learn to stop stealing, and the laborer will be able to report successfully in the future, so the Q-value of No Action also increases. The overlapping of confidence intervals signifies a lack of statistical significance at those particular reporting success rates, indicating that the decision-maker does not have a clear optimal choice at most of 0.1% but does have a clear choice at 1% and 2%. As our algorithm chooses the highest Q-value at each state, it appears that holding all else fixed, a probability of reporting above 1 percent would remove this “indecision” for the decision-makers, and they would opt for reporting. This observation shows that the distance between the confidence intervals expands proportionally as the reporting success increases. This observation implies that systemic alterations influence behavioral change. It highlights the sensitivity of the policy to real-world systemic factors. Consequently, a non-substantial increase in reporting success is necessary to drive meaningful changes in decision-making and optimal policy formation in a simple environment where the laborer has no impact on the system.

4.2. Multi-agent formulation

We increased the realism in our analysis by including decision-making by both the laborer and the employer. In the multi-agent environment, after running the program with the empirically estimated parameters mentioned in the methods sections with 1,000 iterations of 100 samples, we collected Q-values from the data creation process. Then we used those Q-values as training data to go back into the environment, run again, and get the Q-values for both the employer and the day-laborer after batch Q-learning. Fig 8 below shows the Q-values of the employer’s decision on whether or not to steal on the left and the employee’s Q-values for their decision on whether or not to report a steal on the right. The top line in either graph indicates the optimal action to perform in that state. The width of the two lines represents a 95% confidence interval derived from 1000 bootstrap samples of 1000 runs of the simulation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Multi-Agent Q Values.

Illustration of the employee’s Q-values for their decision on whether or not to report a steal on the left and the employer’s decision on whether or not to steal on the right for various values of the probability of reporting success.

https://doi.org/10.1371/journal.pcsy.0000079.g008

Holding all else the same, performing a similar analysis as the single agent case by increasing the probability of success results in the lines swapping places for the employer, and the distance between the Q-values of the employee’s response increases. Fig 8 displays the Q-values for their decision whether or not to report a steal on the left and the employer’s decision on whether or not to steal on the right for the empirically estimated 0.1% chance of reporting success, a 1% chance, and a 2% chance. The y-axis represents the Q-values, and the x-axis represents the number of iterations. Again, the overlapping of confidence intervals signifies a lack of statistical significance at those particular reporting success rates, indicating that only the laborer does not have a clear optimal choice at 0.1% but does at all other probabilities. It is important to note that a range of values exists where it is optimal for the laborer to report a steal while it’s still optimal for the employer to steal (e.g., less than 2%). As the likelihood of reporting success increases (here as low as 2%), the policy converges to employers not stealing and laborers reporting wage theft that they experience.

4.3. Key findings

Following 1,000 runs of the simulation, each with 1,000 iterations, Table 2 below presents the average Q-values of interest after the 1,000 iterations for both the single-agent and multi-agent formulations. The table also includes the iteration at which the respective Q-value converged via mean absolute change, and variance of Q-value change, allowing for up to 10,000 iterations, both with a threshold of 0.01. The ordinality of the Q-values is the most important part of understanding the optimal action any agent should take in any scenario. In cases like these, where the Q-values have an interpretation of the net present value of wages, the cardinality can be used to calculate effect sizes. For example, in the multi-agent framework, an increase in the successful reporting rate from 0.1% to 1% results in a $15 increase to the worker for reporting on average and an $8 increase for not reporting. More significantly, the effect of raising the success rate from 1% to 2% results in an additional $49 increase to the worker for reporting on average. This larger increase is partly due to the employer’s optimal action shifting to not stealing at 2%, as shown earlier in Fig 8. This is also reflected in Table 2, where the change from the successful reporting from 0.1% to 1% brings the q values from the employer closer together and the Q-values swap order at 2%. This small incremental change, requiring only a 2 percentage point increase in reporting success, can likely be achieved through a nudge, such as advocacy groups or public service announcements. This continues in the discussion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Q-values and convergence statistics by agent action and reporting probability.

https://doi.org/10.1371/journal.pcsy.0000079.t002

The Python file that runs the simulations and provides the resulting graphics is provided as the Supplemental document S1 File, and written information of the Python file’s contents can be found in the following GitHub repository: https://github.com/EvanAldrich/RL-ProductiveEmployment. Sensitivity analyses were also conducted to examine the effects of the learning parameters’ values (alpha, gamma, epsilon) and environmental parameters (pTheft, propDegreeTheft) on both the single-agent and multi-agent systems, which are presented in the supplemental document S1 Appendix.