You surely know what a ban is. It simply means that we are preventing a user from performing certain actions on our website or disallowing him from even entering it. Banning is a common (though imperfect) technique of combating so-called trolls, spammers, aggresive users etc. "I will ban you by IP" is probably one of the most popular threat out there.
But, what is a hellban then? Sounds terrifying. I have met this concept a couple of years ago and found it really interesting. Suppose we have a really annoying, toxic user on a website. The idea is quite simple: instead of preventing him from leaving any comments on the website, we actually allow to do so. The gotcha is that no one (except for the admins) will actually see his comments! Why do trolls like to troll? To feed upon other users' negative emotions, to have fun looking at theirs aggressive reaction. The best tactic is to completely ignore trolls and not to feed them - in most cases they just lose interest and leave the site looking for other prey. If no one notices a troll, no one reacts to his provocations, he has no point in trolling.
That's the main idea behind hellban. From the troll's perspective, his comments are visible on the website, but for some reason no one responds to them. From other users' perspective, the toxic comments are not visible at all. Think about this: what would a troll do if you ban him? He'll create another account or use an open proxy to change his IP address and continue to misbehave. If, however, he doesn't see any reaction to his actions, he'll have no point in further attemps.
You may develop this concept even further and, for instance, allow hellbanned users to see all the comments (including ones posted by their hellbanned fellows)! That's a pretty interesting scenario: there will be a "shadow" community on your website which cannot contact ordinary users. Well, most likely they'll understand what's happening pretty soon :).
So, hellban does seem like a potential solution in fighting trolls, but no doubts it has some shortcomings. Still, you might want to apply it in one of your projects. And, once again: do not feed trolls.