Growing up in the Circle, the Inquisitor learned by necessity to avoid casually touching people lest the templars made assumptions about his affections. As much as he wants them, he can't help but tense when Dorian touches him unexpectedly in public. It shouldn't be such a hard subject to talk about but the Inquisitor finds he can't until he realizes that Dorian is pulling away emotionally (Dorian would probably assume that the Inquisitor was having second thoughts, that he didn't want to be with a 'Vint or perhaps the issue was that he was male... maybe Dorian was Trevelyan's first male lover and it took some persuading for their relationship to get started), and he fears it might be too late.