Incredibly poor software design costs Citigroup $500M

>From the incomparable Bloomberg columnist Matt Levine
  (Relevant excerpts from paywalled item):

... The “easiest (or perhaps only)'' way to pay off some lenders but not
others was to instruct the software to pay off all the lenders! But tell it
only to *pretend* to pay them! Just send that money to a wash account! This
is all fine! Let's read another horrifying paragraph!

Because the vast majority of wire transactions processed by Citibank using
Flexcube involve the payment of funds to third parties, any payment entered
into the system is released as a wire payment unless the maker suppresses
the default option. Citibank's internal Fund Sighting Manual provides
instructions for suppressing Flexcube's default. When entering a payment,
the employee is presented with a menu with several *boxes* that can be
*checked* along with an associated field in which an account number can be
input. The Fund Sighting Manual explains that, in order to suppress payment
of a principal amount, “ALL of the below field[s] must be set to the wash
account: FRONT[;] FUND[; and] PRINCIPAL''—meaning that the employee had
to check all three of those boxes and input the wash account number into the
relevant fields.

This is just demented stuff. If you want to send out interest payments in
cash, but send the principal payment to the wash account, you have to check
the box next to PRINCIPAL and also the boxes next to FRONT and FUND.
PRINCIPAL sounds like principal: You are sending the principal to the wash
account, sure, right, yes, check that box.  FRONT and FUND sound like
nothing. So the Citi operations people messed it up:

Notwithstanding these instructions, Ravi, Raj, and Fratta all believed --
incorrectly—that the principal could be properly suppressed solely by
setting the PRINCIPAL field to the wash account. Accordingly, as Ravi built
out the transaction between 5:15 and 5:45 p.m. in his role as maker, he
checked off only the PRINCIPAL field, neglecting the FRONT and FUND
fields. Figure 1, below, “is an accurate image of the Flexcube screen after
[Ravi] input the data.''

At 5:45 p.m., Ravi emailed Raj for approval of the transaction, explaining
that “Princip[al] to Wash A[ccount] & Interest to DDA A[ccount].'' The
“DDA Account'' referenced the Demand Deposit Account, which is an
operational, external-facing account used by Citibank to collect payments
from customers and make transfers to lenders. After reviewing the
transaction, Raj believed—incorrectly—that the principal would be sent
to the wash account and only the interest payments would be sent out to the
Lenders.  Raj then emailed Fratta, seeking final approval under the six-eye
review process, explaining “NOTE: Principal set to Wash and Interest Notice
released to Investors.'' Fratta, also believing incorrectly that the default
instructions were being properly overridden and the principal payment would
be directed to the wash account, not to the Lenders, responded to Raj via
email, noting, “Looks good, please proceed. Principal is going to wash.''

The software gave him a warning, but not a very good one:

Raj then proceeded with the final steps to approve the transfers, which
prompted a warning on his computer screen—referred to as a “stop sign''
-- stating: “Account used is Wire Account and Funds will be sent out of the
bank. Do you want to continue?'' But “[t]he stop sign' did not indicate the
amount that would be sent out of the bank,' or whether it constituted an
amount equal to the intended interest payment, an amount equal to the
outstanding principal on the loan, or a total of both.'' Because Raj
intended to release “the interim interest payment to [the] [L]enders,'' he
therefore clicked “YES.''

Here's Figure 1; it does not particularly explain itself:

See, the “don't actually send the money'' box next to “PRINCIPAL'' is
checked, but that doesn't do anything, you have to check two other boxes to
make it not actually send the money.

When they discovered the error the next day, their first reaction was not
to email the lenders asking for the money back (that was their second
reaction); their *first *reaction was to email tech support to say the
software was broken:

At 10:26 a.m., Fratta emailed Citibank's technology support group:
“Yesterday we processed a payment with Principal to the wash and Interest
to be sent to lenders. All details in the front end screens yesterday le[d]
us to believe that the payment would be handled in that manner. . . .
Screenshots provided below indicating that the wash account . . . is present
and boxes checked appropriately for the principal components.''  Fratta then
forwarded the same email to members of his team, with the subject line
“Urgent Wash Account Does not Work.'' He stated: “Flexcube is not working
properly, and it will send your payments out the door to
lenders/borrowers. The wash account selection is not working. This lead
[sic] to ~1BN going out the door in error yesterday for an ABTF Deal,
Revlon.'' ...

Over the course of the day, Fratta learned that the principal payments --
which were made with Citibank's own money, as Revlon had provided funds only
for the interim interest payments to be made in connection with the roll up
transaction—were not caused by a technical error, but by human error: the
failure to select the FRONT and FUND fields when inputting the default
override instructions in Flexcube.

Nope, nope, he was right the first time, this whole setup is a “technical
error.'' Citi's software will only let you pay principal to some lenders if
you pretend to pay it to every lender, and it will only let you pretend to
pay principal to every lender if you check the “just pretend'' box next to
“PRINCIPAL'' (fine!) and “FUND'' (what?) and “FRONT'' (what even?). What a
terrifying thing......l