
Oracle could find no defect with the database software. Their analysis resulted them in saying that there must be a defect in our source code. The only value add they provided was some queries which were taking a long time on average.
This was most disappointing. Any monkey with a tool can identify the worst performing queries in the system. Now this is not to say that a query might not be poor. However these SQL statements were doing updates using only a primary key in the WHERE clause.
Sooner or later this problem will come around to our team. I told our project manager that I expected we would need a dedicated Oracle performance engineer. We used to have such individuals assigned to our project. They have since been replaced by consultants who are competent, but not performance experts.