Joe – who has earned the moniker ‘Britain’s most prolific dad’ – shared that he has faced cruel comments over the years, with some suggesting that he is only involved for ‘sex’