xfce4 itself is a meta-package that will install a default configured xfce desktop environment.
In most cases the base files that comes with xfce4 are:
xfconf, xfce4-utils, xfwm4, xfce4-session, thunar, xfdesktop4, exo-utils
So you can do
sudo apt-get purge xfconf xfce4-utils xfwm4 xfce4-session thunar xfdesktop4 exo-utils xfce4-panel xfce4-terminal
Then most of the package that were associated with these package become autoremovable so you can run
sudo apt-get autoremove
Almost all xfce4 package depend upon libxfce4util-common. Just purge that one and you remove everything related to xfce