I spied on the data in our production database. Saw the malformed records. Traced it back to our new development for the year. Even the comments in the code were wrong. Must have added to the confusion. I used the debugger to step through the code to get the records in a bad state. Then I made a code change, stepped through the code once more, and was convinced the fix was good.
Testers cannot run the code in debug mode. They use the release version. I told them it would be difficult to emulate the concurrency needed to see the problem. They gave it a try. A manager asked if there was anything we could do to help. His idea was to write a trigger on the table in question. We could then make a call to the Sleep procedure in the DBMS_LOCKS package. With things slowed down, he figured the testers could have a good chance to debug the problem. Somehow this did not feel right. But hey. It was worth a try.
