Modeling Loan Repayment Behavior and Cash Flow

Keeping Affirm’s credit network running sustainably depends on our ability to accurately estimate the performance of our loans. We measure loan performance in several ways including how much we expect to lose to charge offs (i.e. someone not repaying a part or all of a loan), how quickly a loan is repaid, and the monthly cash flow for a given set of loans. Accurately estimating these values is useful for a wide variety of analyses and applications. For more context, we will go through two motivating examples:

Underwriting Decisioning

Macroeconomic conditions and seasonality have a large effect on Affirm’s business. In response to these effects we often tune underwriting system levers in order to ensure that we are sustainably managing risk. Loss estimates play a crucial role in predicting the impact of underwriting system changes before we can actually observe the performance on newly originated loans.

Modeling Future Cash Flow for Budgeting

For the purposes of financial planning and budgeting, the finance team at Affirm needs to have an accurate view of the amount of money we expect to receive in any given month. This requires predicting the month-by-month cash flows over our entire portfolio of loans, looking a month, quarter or year in advance.

With these use-cases in mind, we set out to build a single, unified system that can answer the following question:

What do we expect the repayment and cash flow to look like for a finite set of slices of our loan portfolio?

Our initial solution to this problem used historical data to model the month-by-month pre-payment (loans being repaid early) and losses for each slice. These stratifications separated originations into different credit tiers, APR ranges and term lengths, treating all loans within a given group as homogenous.

Figure 1: Example of slices and loss curves. The curves show the expected proportion of principal that is lost month-over-month for the bucket of loans defined by each quadrant.

As the demand for more accurate predictions of repayment over our increasingly complex portfolio of loans grew, the model needed to handle an increasing number of dimensions on which to distinguish individual loans. Since the aforementioned problem statement limits us to a finite set of stratifications, it is not sufficient for more complex use-cases where the set of loans is arbitrary. Also, it limits us from modeling shared properties of repayment between the different slices. With this in mind, we decided to reframe the problem as:

What do we expect the repayment and cash flow to look like for an individual loan?

We built the Loan Transition Model (or LTM) which works in tandem with a Cash Flow Engine to answer this question for each and every single loan in our portfolio. This allows us to compute aggregate consumer repayment for any arbitrary set of loans. The LTM is responsible for modeling the month-by-month transition probabilities between different delinquency states, while the Cash Flow Engine applies loan amortization logic to consumer repayment behavior to compute aspects of repayment dynamics such as the split between principal and interest repayment. Below, we will dive deeper into both of these pieces.

Modeling Loan Transitions using Gradient Boosted Decision Trees

The LTM models the repayment behavior of an individual loan. But what do we mean by repayment behavior? We view this as the journey that a loan goes through from the day that it is originated, to the day that it is repaid in full or charged off. We formalize this as a walk through the following state machine:

Figure 2: State-machine for Affirm loans.

• current: the user has made all of the payments required thus far. This is also the initial state of a newly originated loan.
• dq(n-m): the user has missed a payment and is n to m days late on that payment.
• charged off: the user is more than 120 days late on a payment, at which point we assume a loss on that loan. This is a terminal state.
• paid off: the loan is fully paid off. This is a terminal state.

Over the last eight years, Affirm has observed a very large number of walks through this graph that we can use to estimate how future loans will behave. The LTM does exactly that.

Table 1: Example walk through the state-machine for a 6 month loan.

In order to limit the complexity of the problem, we made some simplifying assumptions. First, we needed to decide on the time-step size for our traversal of the state-machine. Given that the large majority of our loans follow a monthly payment schedule, the natural choice here was to model transitions that happened on a month-by-month basis. Secondly, we assumed that the Markov property holds for this problem. The Markov property states that:

The probability of transitioning to different states in the future only depends on the current state, and not on the sequence of states that preceded it.

This assumption allows our model to predict with each month’s state transition independently of all other transitions for the loan.

Putting it all together, the LTM predicts the state transition probabilities for a given loan at a given age (number of months since origination). The features that act as inputs into the model can be categorized as contextual and covariate. Contextual features, c, describe the loan itself, including signals such as the credit score assigned to the loan, the loan amount and term length. Covariate features vary from transition to transition and are meant to describe the state of the loan at each month. The main covariate features are the loan state, s, and the age, t. The model outputs the probability of the state, s’, at time t’ = t + 1, being each of the seven possible loan states. More formally, it tries to predict:

P(s’ = S | c ∩ s ∩ t) for S in {cur, dq1, dq31, dq61, dq91, co, po}

In fact, for each possible age a loan could have, the model can generate a transition matrix that looks something like the following:

Table 2: Example transition matrix produced by the LTM.

For example, if at time t, the loan is current, then the model predicts that the loan will also be current at time t + 1, with a probability of 89.5%. Since charged off and paid off are terminal states, the probability of staying in them is 100%.

The model itself is a gradient boosted decision tree that is trained to minimize the cross-entropy, or multi-class log loss, where each class represents one of the possible loan states. We chose this loss function because we interpret the output of the model as a probability distribution over the possible destination loan states, as seen in the transition matrix. The dataset used to train the model has one row per observed month for each loan in Affirm’s history. The rows for a single loan look like:

Table 3: Dataset rows for the loan state chain shown in Table 1.

One thing to notice here is that the next state for age t is the same as the current state for age t + 1.

Converting Transition Probabilities into Cash Flows

Once we have computed a set of transition matrices for a given loan, one for each possible time-step, we use the Cash Flow Engine to compute different monthly and overall estimates for the loan, such as the interest paid, principal paid, principal prepayments, and principal charged off. This is done by iteratively applying the transition matrices from the LTM to compute month-over-month probability-weighted principal balances. In order to better illustrate this process, we will follow a $100, 0% APR, 3-month loan through a couple iterations of the Cash Flow Engine.

We start with the initial probability-weighted principal balance vector (time-step t = 0), which has all of the principal, $100, in the current state, since this is the initial state for all loans.

Table 4: Probability-weighted principal balance vector for t = 0.

This vector is then multiplied row-wise by the transition matrix for the t = 1 transition, which looks very similar to the one seen in Table 2:

Table 5: Multiplying initial vector by the t = 1 transition matrix.

The resulting matrix specifies how much probability-weighted principal will be transitioning from each previous state to each next state. For example, we have $89.50 transitioning from current to current, representing someone making a payment, and $4.00 transitioning from current to dq 1–30, representing someone missing a payment.

The next step is to determine how much principal will remain after each of these state transitions occurs. For example, if we are transitioning from current to current, it means that the user has made a payment, in this case $33.33, so the amount of principal that will be left to repay after this transition is 0.6667 of the existing amount. For the transition of current to dq 1–30 no payment is made, so we will have 1.0 of the existing amount. These principal balance transitions are captured in what we call the amortization matrix. The matrix from the previous step is multiplied element-wise by the amortization matrix to produce a matrix of probability weighted amounts for each transition after payments are taken into account:

Table 6: Adjusting probability-weighted principal balances for payments using an amortization matrix.

Finally, we column-wise sum the resulting matrix to get the probability-weighted principal distribution vector for time-step t = 1. In this case, we would expect to have a probability-weighted balance of $59.70 in the current state and $4.00 in the dq 1–30 state at t = 1. Let’s follow for another iteration:

Table 7: Another iteration of the Cash Flow Engine to compute probability-weighted principal balances for t = 2.

We repeat this process until all of the principal has either been paid in payments, as captured by the amortization matrix, or it accumulates in the charged off and paid off states, since these are terminal. For interest-bearing loans, we apply additional logic to compute how much interest we would collect each month. At the end, we can compute the many values, such as the following:

• Total loss: the amount we expect to lose on the loan is the probability-weighted balance left in the charged off state.
• Total cash flow: the total incoming cash flow is the probability-weighted balance left in the paid off state plus the sum of all monthly principal and interest payments made over the lifetime of the loan.

Since this iterative process also computes these values for each intermediate time-step, we can also observe how these values change month-over-month.

Results

Figure 3: Comparing loss/charge off curves between the different methods.

From the validation we have done, it is clear that the LTM and Cash Flow Engine based approach (ML_LTM and LTMV1 curves) are much more accurate at predicting charge offs than the slice-based approach (LEGACY_VAL curve).

Computing Cash Flows at Scale

Given the way we’ve formulated the problem, we have run into scaling challenges, both with training the LTM and running the Cash Flow Engine. Assuming that we have n loans and the average term length for a loan is m, then the size of our training dataset would be n * m. To make matters worse, each transition matrix computation requires us to predict using the model five times, once for each previous loan state. Hence, when running the Cash Flow Engine, we actually require 5 * n * m model predictions. In some cases, the simulations we run using the Cash Flow Engine contain hundreds of millions of synthetic loans. With loans up to 48 months long, we need to run billions of model predictions for these simulations.

We addressed these scaling challenges by migrating both the LTM model training as well as the Cash Flow Engine to our Spark-based ML Platform. We leverage XGBoost4J-Spark distributed model training on top of Spark, which allows us to train on datasets that might not be able to fit on a single machine. Also, we implemented the logic for the Cash Flow Engine in PySpark to be able to easily distribute the computations described in the previous section. With the help of Kubernetes, we are able to declaratively specify how much resources (CPU cores or memory) we need for any of the aforementioned jobs. This allows us to tailor our Spark cluster size to the job at hand and gives us the confidence that this solution will horizontally scale into the future.

Acknowledgements

All of this would not have been possible without the following people:

• Anibhav Singla and Benson Lee, who originally formulated the problem and implemented the first versions of the LTM and Cash Flow Engine.
• Adam Johnston, Hossein Rahimi and Niloy Gupta, who built the infrastructure that enabled us to scale this solution to robustly handle massive datasets and simulations.

. . .

Interested in solving these types of problems? Join us!