In this country, the majority of Christian prison chaplains of both Reformed and Roman Catholic traditions do not routinely wear the "dog collar" at work. I pretty much only wear mine if I'm to be involved in a memorial service or, as I did recently, I'm conducting a wedding.
I think the reason most of us opt for "civvies" is a good one. We are employed not by the churches but by the prison service and we are employed (1) to perform religious functions such as worship services but also (2) to offer our support and encouragement to those of all faiths and none. The latter forms the bulk of our working week actually as most prisoners are not practising Christians. It is important to be as approachable as possible and to make it as clear as we possibly can that although, yes, we are the God Squad, we are also a resource that is available to everyone. Probably we all instinctively feel that wearing the clerical collar might make it harder to get that important message across. We don't want unnecessary barriers getting in the way of what we do.
However, no matter how many times we explain in all sincerity to staff and prisoners alike that we are there to offer generic support to all, there is a sense in which the fact we are members of the clergy is still there. And maybe we try to squash it down when there's no need to do so. Even people who wouldn't describe themselves as practising Christians may have the remains or the beginnings of a faith. They may not really want us to be just like any other generic counsellor but to bring our sense of the numinous to the table, to have hope in God on their behalf until they can have it themselves.
As an experiment, I'm going to start wearing my clerical collar one random day per week.