Something about the way the guy defers to Joel, practically bows, makes him incredibly uncomfortable. This is why he avoids avoxes in the Capitol, and why he avoids them here as well. Why he warned Tess so strenuously about them.

He doesn't want to see her like this.

"Jesus Christ, just... stay outta here, alright?" he snaps. "Go find your people, if you've got any left."

With that, he turns to go. He needs to get away from this guy, that's for damn sure.