Rajya Sabha MP Saroj Pandey is accused of giving wrong affidavit