Incident Report: Failure to record an event prepayment

On July 23rd, a participant paid for Doorkeeper an event via our credit card processor, Stripe. However, due to a bug, Doorkeeper did not properly record the payment. Because payment was not reflected on Doorkeeper, the participant paid once more, and this time we recorded payment.

Upon being made aware of the duplicated payment, we fixed the issue that allowed the bug to happen. We also confirmed that this issue only occurred this one time, and added additional monitoring to help us catch similar issues in the future.

This issue gave us the chance to review our processes surrounding serious issues like this. As a policy, going forward we will disclose any issues with event registration, even those that are limited in scope like this one.

Incident Timeline

2018-07-23

The participant completed prepayment through Stripe, however the prepayment wasn’t properly recorded on Doorkeeper’s side. Although the participant saw an error, this wasn’t picked up by our existing monitoring.

Because the participant saw an error, and their registration wasn’t being shown as completed, they contacted the organizer, reporting the error and asking the organizer to confirm the payment status.

After this, the participant tried to pay for the event once more, however, the payment was blocked by their credit card company. Half an hour later, the participant tried to pay once more, and this time the payment went through. So at this point, the participant’s credit card had been charged twice.

2018-08-06

The organizer fully refunds one of the two duplicate payments to the participant.

2018-08-07

The participant’s credit card provider opens an investigation of the non-refunded payment.

2018-08-08

The organizer reports the investigation of the payment to Doorkeeper, informing us of the issue with the participant being charged twice.

2018-08-09

Doorkeeper fixes the issue that allowed duplicate prepayments, confirms that this is the only time this issue has ever occurred, and add an additional check to alert us when Stripe payments aren’t properly completed. We respond to the organizer, informing them of this.

Root cause

This issue resulted from how Doorkeeper created a charge with Stripe, and then recorded prepayment. A simplified version of our code is the following:

charge = Stripe::Charge.create(amount: @registration.amount, source: params['stripeToken'])
if charge.paid
  @registration.prepaid!
end

As long as the code is executed, it works as anticipated, and prepayment is recorded.

However, in the case of this issue, the charge was created with Stripe, but before we could record prepayment, the process was unexpectedly terminated.

A process terminating unexpectedly is a rare event, but something that can happen, and so our application should have handled this case. This is also why we received no alert, as because the process was abnormally terminated, no exception was raised within the code itself.

To resolve this issue, we now record that the registration is in a “processing payment” state before making the charge attempt with Stripe. When a registration is in this state, we don’t allow subsequent payments, and preventing double charges even if the process terminates unexpectedly.

Additional monitoring

To help avoid similar incidents in the future, we’ve added additional monitoring that will alert us when something unexpected occurs.

Every time a participant starts a prepayment attempt with Stripe or PayPal, we schedule a check to be performed five minutes later. This check will ensure that the prepayment resolves to a valid state, for example, successfully completing, or being denied by the credit card processor. In the case that the payment remains in an unresolved state, we’ll receive an alert. Such an alert would have allowed us to catch the prepayment that was made on Stripe’s side but not recorded on Doorkeeper’s side.

Additionally, we’re now implementing an additional check using the webhook that Stripe provides. This webhook sends a notification whenever a charge is made through an organizer’s Stripe account. For any charges created by Doorkeeper, we now ensuring that the charge is also recorded on Doorkeeper’s side. This also would have also allowed us to catch this issue.

Avoiding misunderstandings

The effect of this bug was magnified due to a number of misunderstandings that resulted because of it. These misunderstandings stemmed from the large number of parties involved: the participant, the organizer, the participant’s credit card company, Stripe, and Doorkeeper. As the application that performed the charge, we have the most complete view of the situation. Unfortunately, we were the last party to become aware of the issue. Had we been able to be more proactive on this, we could have avoided a lot of misunderstandings.

With our additional monitoring, we’ll be in a better position to catch future issues like this. But there is still a risk that other issues slip by unnoticed by us, and so we will work on ensuring we are notified should such an issue occurs. Specifically, we are considering

• Encouraging participants to contact us as opposed to the organizer when an issue with registration occurs
• Encouraging organizers to contact us if a participant reports an issue with registration
• Working with Stripe’s customer support to help ensure that should an organizer raise an issue with them, we’re also informed

We don’t have any concrete solutions for these yet, and we’re not sure to what degree we’ll be able to meet those goals, but we’ll work to minimize future misunderstandings.

Other similar past incidents

In addition to this issue, we’ve had other issues regarding prepayment in the past. As we were able to quickly work together with the individual organizer that was affected by each issue, we didn’t report them publicly. For the sake of completeness, we’re briefly summarizing them here.

2018-07-18

An organizer reported that Doorkeeper was showing that some participants had made “invalid” prepayment attempts via PayPal, but on PayPal’s side the prepayments were being shown as successful.

We determined that this was caused by PayPal returning a status of “Completed” (only the first letter capitalized) instead of “COMPLETED” (all capitals). As for all other organizers the status continued to be “COMPLETED”, and even for this organizer, only sometimes “Completed” was returned, we assume this was a bug with PayPal. That it was a PayPal issue is further supported in that it runs counter to PayPal’s documentation, which states that the value for status should be “COMPLETED”.

Because Doorkeeper was checking against known statuses, and otherwise marking a prepayment as invalid, the status of “Completed” caused us to consider the payment to be invalid. We worked around PayPal’s issue by making the check of the status case-insensitive.

In total, eight registrations for one specific organizer were affected by this issue. We provided the organizer with the details of each affected registration, and suggested course of action.

2018-06-05

Our automated monitoring detected an issue with an attempt to redirect a participant to PayPal’s payment page. When we investigated further, we discovered that PayPal has a bug where if the “memo” field to contains a special character “\b” (the backspace character code), PayPal’s API crashes, returning "Internal Error".

As a quick fix, we removed this character from the event’s title, so that it wouldn’t be included in the memo field. We then contacted the organizer about the issue, asking them to contact the single participant affected by the issue.

We then implemented a work around for PayPal’s bug so that we remove this character from any “memo” fields that we send them.

2017-07-21

An organizer reported that Doorkeeper wasn’t able to fetch detailed information about certain PayPal transactions. After further investigation, we discovered that Doorkeeper has marked payments as completed on our side, whereas they were “pending” on PayPal’s side.

We discovered that when a payment is marked as pending on PayPal’s side, their API returns an overall status of “COMPLETED”, and not “PENDING” as their documentation implies. Rather than looking at the overall payment status, we needed to check if any of the parallel payments had a “transactionStatus” of PENDING.

To resolve this issue, we changed our code to correctly mark the payment as pending. We also added alerting in the case that a payment was being marked as pending for a reason other than because it was an eCheck (a payment method PayPal supports for certain accounts outside of Japan), so that we could proactively contact the organizer about the pending payment.

The reason why the payment was being marked as pending in the first place was that the organizer changed their PayPal account email address on PayPal’s side, but not on Doorkeeper’s side. PayPal allows you to create a payment for any email address, not only those that correspond to a PayPal account. When you create a payment, it is marked as pending, and PayPal sends an email to that address informing the receiver that they can claim the payment. Because of this, we were creating payments that were being sent to an address the organizer no longer monitored. This behaviour highlights another edge case that implementers of PayPal need to be aware of.