Should I use direct references to classes as delegates?

So a lot of the time when I'm programming something (usually in Java or Objective-C) I use a delegate that implements an interface (or a protocol depending on which language I'm in), asserting that whatever class is passed to an object implements methods with predefined signatures.

I usually do this so that a thread can pass a message back to a controller when it is done executing, or a menu shown to the user can initiate some action when the user selects an option. I can see the use of generalising whatever is passed to my class to be any class that implements particular methods, as it allows me to reuse the code I've written.

However, the other day I was reading an article about "anti-patterns" and unnecessary abstraction. The writer was saying that programmers sometimes have a habit have making abstractions prematurely, such that they have no practical use and only serve to obfuscate the code.

I realised that sometimes, when I use delegates, only one class will ever implement the interface (or protocol) and create objects of the class that requires this delegate to be passed.

This made me think- should I instead be passing a reference to the single class that uses the code that requires a delegate, only generalising the code with an interface when it becomes necessary such that two classes can use the same code?

Thanks!