I think part of the problem stems from that fact that 3K reputation is required to cast a vote and there are not that many users with sufficient reputation.
Most likely. In fact right now we only have 20 such users and at a glance (you could confirm this crudely by checking the "last seen" stamp on their profile tabs) I would guess less than half of them are very active, and of the ones that otherwise are I would guess not all of them bother with the queues.
Further, all three moderators are in that category. While I don't recall discussing this with Steve or Ghanima, I have in the past made my own feelings known about an issue with "the mod hammer" -- namely, it takes 5 user votes or one moderator vote, and I cannot switch back and forth. If I vote to close something it is closed. Because of this, the close queue is the one I tend to use least as I would prefer to leave it up to the slightly more democratic and community oriented "5 normal users". I'm already the hanging judge of Rpi.SE, having closed ~1500 questions (i.e., somewhere close to 10% of the total; note the total figure does not include deleted questions, and questions that are closed with no upvotes or answers get deleted automatically after a certain time).
It is a necessary tool but it is a hammer and one must be careful with hammers.
I'm not sure the last time I did access that queue, but it has to be at least a week ago (although not more than two, since I seem to have some closures via queue indicated this month). However, I'm still the sixth icon in the pile, meaning during that time only five other people have accessed it, and no one can vote on the same question twice.
So closing via the queue here is a slow process. In my opinion, this is not our fault, beyond the fact that the community is probably unusually stingy with upvotes, making it difficult for people to rise the ranks (and here's a tip: Go through that same list of top users and see how many "votes cast" is shown on everyone's profile tab; some of our most highly regarded users seem to have either very little regard for the system, or very little for other users...). As I have tried to indicate many times, the tooltip for the upvote icon on answers begins "This answer is useful...", not "This answer deserves a Nobel Prize" or "This answer helped me personally" or even "This answer is outstanding". A similar logic applies to questions. Being useful should be sufficient to receive recognition for your efforts. I forget this myself too often as well.
However, beyond that, Stack Exchange uses a one size fits all system with those numbers. It's 3000 rep, 5 votes on Stack Overflow, and 3000 rep, 5 votes everywhere else (except Beta sites, where the rep level is much lower). If this seems a little ill conceived, I agree, however, having done some digging into the issue on Meta.SE and gone through at least one brief discussion in the (moderator/staff only) "Teacher's Lounge" chat room, I can summarize a couple of things:
There are other moderators of small sites that are also concerned this is a dysfunctional mechanism that would be better dealt with using some kind of scaled/proportional standard (although the ideas around this come with their own problems and complications).
The overwhelming majority of very active, high rep Stack Exchange users are against us, end of story. As is often the case amongst significant majorities in democratic contexts, many of those in the majority don't need to bother to listening to arguments contra their own perspective and instead simply repeat warped, straw dog interpretations such as, "You just want to coddle people". If the numbers of those screaming the warped straw dog interpretations drown out their opposition, then that opposition never gets a chance to be heard. End of story again.
It might be worth observing that there is also a sizable percentage of aforementioned very active high rep users (likely with 100% overlap of the previous group) who have made it clear some smaller sites including this one should never have been allowed to exist in the first place, and if we take the whole realm a little too seriously and construe it as highly politicized (which perhaps it is), then it is in their political interests to see those sites handicapped by rules which pave the road to failure, even if they know the rules are not fair.1
Don't worry though, we are not currently on such a road, our user reputations are slowly but surely increasing, and we are doing a reasonable enough job of filling a niche not to have to worry about any elimination game.
1. Warped straw dog style retort to this one is I am a delusional conspiracy theorist. ;)