The chair is asked to solicit feedback from the candidate's advisees and summarize that feedback at the pretenure, tenure, and full professor evaluations. That document should be given to the candidate for inclusion in the file, as well as for reflection and comment in the self report.
There's no official requirement about form of the questionnaire; some departments prefer creating an anonymous Google Form while others stick to an email solicitation. While the Google Form is less work, either's just fine if you're not Google Form savvy.
Regarding protocol: have the candidate forward you the names/emails of all current advisees. I'd recommend contacting these students immediately following Advising Week, as a way of prompting well-informed reflection on that particular engagement. I'd avoid any contact in advance of the appointment to avoid any undue pressure during those appointments, either for the candidate or the advisee.
For your convenience, see the following two documents below. The first is a sample advisee email with possible questions for you to modify as your department sees fit, as long as the questions posed of this candidate's advisees are then the same questions posed to all department faculty advisees. While more challenging to track in your inbox, a bcc to all students is best to preserve anonymity (and by the way, it's helpful to flag them with a label or whatever in your inbox so you can find them quickly once you start to compile data).
As students reply to you, keep a Google Doc summation of the data in a form similar to the "Compiled Responses" document also attached here. It's a fairly efficient way to add their individual quantitative scores to the summary sheet, plus the narrative comments for the final two questions. (If you've created a Google Form, you're welcome to include the "Responses" page with pie chart and narrative comments as long as no identifying information is inadvertently revealed.)
Compliance is notoriously awful, so give it your best effort: plan to send out 2 follow-up reminders to those who haven't weighed in (I find that shifting to an individual email at that point with their name in the subject line helps... "John: follow-up request for advisee feedback"). After the 2nd pitiful plea, that's plenty. Tally up the averages, add your signature to the top of the memo to show it's the chair's official document, and send as a PDF for the candidate to add to their file. The goal is to be sure all this gets to them in time to reflect and comment on in their self-report.