Rails 4: internal messaging system

Rails 4: internal messaging system

I'm trying to write a simple messaging system for signed up users. I
realize there are gems for this but I'm trying to roll my own simpler one.
I've set up my models as follows:
class Conversation < ActiveRecord::Base
has_many :messages, dependent: :destroy
has_many :users
end
class Message < ActiveRecord::Base
belongs_to :conversation
end
class User < ActiveRecord::Base
has_many :conversations
end
I've set my routes and controller up so that I can make new conversations,
and add messages to those conversations. However, I'm trying to figure out
how I can make it so that only a logged in user can start a conversation
with ONE other user. No other users should be able to access that
conversation.
Is this a permissions (cancan) thing or should this be defined by some
controller logic?
Thanks!