Our software team often has difficult problems so solve. We maintain a system with a lot of users and a ton of data. Recently a certain functionality in our application has gotten terribly slow. The users don't even try to perform those functions any more. It is a difficult problem because we cannot make it happen in a development environment. We tried to add some logging but it did not shine any more light into the situation. The last resort was to allow a developer to run the application in a production environment. But it seems to take an act of congress to authorize this to happen.
Some consultants came up with a plan. They would fly out to the user's site. And they would see what the user was experiencing. Then at the same time an Oracle performance consultant would run some tools on the back end database to monitor what was going on. This would identify the problem, or at least the problem area, if in fact the problem exists on the database side. We make a couple calls to some stored procedures in a PL/SQL package. One of these calls could be hanging. I really hope this is not some weird hang up on the workstation. Because then this consultant test will give us no new information (other than they could not locate the problem).
