I have search the Internet but all resources I found seem not to work (I guess spambots are smarter now). am having a lot of spambots on my site, these spambots are able to register and login to the site then stay online(this is disturbing) I have used captcha on creating new topics(this stopped spam bots from posting to my forum) but what I want is something that will totally stop spambots from accessing my site(maybe redirect them to 404) please help. thanks in advance
but what I want is something that will totally stop spambots from accessing my site
This isn't possible. It's completely impossible. If this is what you want, stop, give up, move on.
If what you really want is to stop as many as you can:
Use some JavaScript to remove and element from the form. (This can be a hidden element.) If the element is present, you know that it's either a bot that didn't run your JavaScript, or a user that has JS disabled. If you need to support users that have disabled JS, don't do this. Note that some bots can and will run JavaScript, so this won't stop all.
Require e-mail verification. Plenty of bots get around this, but many don't.
You can keep your CAPTCHA if you want, but bots are pretty good at figuring this out, and when they aren't there is always Mechanical Turk and similar methods.
When a bot does get in and create thousands of accounts, figure out what happened by analyzing your server logs. Did you see 1,000 requests from the same IP address? Do something about that and throttle requests from one IP. Did they all have the same user-agent string and does it look bogus? Consider blocking it. This sort of work will go a long way, but requires your effort.