When designing an improvement for a system, you

When designing an improvement for a system, you typically don’t want it to get worse. Good strategy:

• figure out an error rate for the old system
• compute the agreement between the new system + the old system
• if the agreement is significantly less than the error rate of the old approach you have a problem