Indian Prime Minister Narendra Modi's Hindu nationalist party has lost a key state legislature election, a setback for the party as it faces massive anti-government protests against a contentious new citizenship law.