If you want to start Xvfb only on specific nodes specify its name or label. See the Restrict where this project can be run option for label expressions that you can use.