I ended up working with the business analyst to help figure out what was going wrong. The input data got sent to our ETL team who loaded the raw data in. Then we got some jobs that process that data and format it for reporting purposes. Finally, we have a reporting structure on top of the formatted data that the customers use.
So I explained all this to the analyst. Showed how a couple queries could be run at the staging and formatted levels to determine where the deficit was coming from. In the end, we determined the point at which the records were disappearing. At that time I excused myself. I had to get back to my day job. However I recommended that they have the customer officially submit this in a trouble ticket.
Later my team lead called me up. He wanted to review this discrepancy with me. Luckily I was very familiar with the scene. He had a guess as how he could solve the problem. And his justification was that the business analyst provided him the requirements. That seemed circular. The business analyst came to me and really knew very little. Now they were being used as the authority on this?
I told my team lead that we had to understand what the business customer actually wanted here, and to make sure their needs were met. Yes we needed to resolve loss of records. But you just can't hack away until some counts match. You got to know what you are doing. So I said let's identify the source of the problem, and also ensure we know how to test that business needs are being met after we make some changes.
