Perhaps it's his intuition confirming what you have suspected about yourself all along?
Doesn't sound negative but I wasn't there so I don't know how he said it. My husband knew I was a witch way before I converted. He joked about it all the time. Finally I embraced what I always knew.
The best thing to do is just talk with him. Maybe he doesn't think its evil. You wont know until you ask. And on the off chance he does, maybe let him read what its all about online or a book.