Al-'Allaamah, al-Imaam, Shaykh al-Albaani (rahimahullah) stated,
Indeed the matter of having an abundance of followers or a small amount is NOT the measuring tool to know if the caller is upon truth or falsehood. The Prophets ('alayhim as-salaatu was salaam), despite their call being the same and their religion being the same, differed in relation to the number of followers they had, some having many and others having few. There was even amongst them a Prophet who only one man believed in. Rather, there was amongst them (a Prophet) who didn't have anyone with him. In that there is a serious lesson for the caller and those who are being called in this (day and) time. It is upon the caller to remember this reality and put a foot forward for the sake of calling to Allah. He should not care about the amount of those who follow him being small. This is because nothing is upon him except clear conveyance (of the message). He has a beautiful example for himself in the Prophets of the past, those who didn't have anyone except a man or two with them.
Likewise, the one being called, it is upon him not to deem the small amount of followers of the caller as [an indication of] evil, and therefore take that as a matter of doubt regarding the truthful call and leaving off believing in it. Not to mention taking that as an evidence that his (i.e. the caller's) call is falsehood. Also, it is not for him to hold the position that if [the caller's] call was a truthful one, then the majority of the people would have followed him. Allah 'Azza wa Jall has stated:
وَمَا أَكْثَرُ النَّاسِ وَلَوْ حَرَصْتَ بِمُؤْمِنِينَ
Most of the people will not believe even if you gave great concern that they do. (Soorah Yusuf: 103).