You can use event signups to let group members, clients, or other users invited to an event sign up.
If you also want to give invitees the option to decline an invitation, use event comments.
Enable event comments for the event.
Make sure the comments are visible for all users; this makes the comments visible to those with read-only permission and visible on the event page, as well.
Add a note in the event description to let people know they can use comments to decline the invitation.