Creating a Challenge Scratchpad
- Log in as a dev on Khan Academy (ask John to get permissions on the live site)
- Go to /cs/new to create a new program as usual. Enter a title for the program, and enter a comment or something in the code (it needs to be non-blank..)
- Click the "Settings" button.
- Set the category to "Challenge"
- Enter a difficulty level (this controls how many points the user will get for completing your challenge)
- Add a description
- Click save and close, click yes to the later modals.
- Great! Now you should see the "code" tab and the "tests" tab.
- Go to the Tests tab. Create tests as explained below
- If at any point you want to test out your challenge, add code to the "code" tab and see the challenge display update with your progress.
- When you save your challenge, make sure that the code tab is in the desired starting state for your challenge. (It is not allowed to be blank, so just put a comment if you want a blank slate)
- All done!
Tips for Challenge Design
- Newbie users often try to write code on the same line as a comment, so always have a new line after a comment, if that's the last thing in the program.
- Some people put variable initalization and assignment on different lines, which requires a different pattern.
- People will often play with variables that you don't expect them to. Either be lenient about it if you care, or put a check for it.
- People often put commands in different orders than you expect, making it harder to check. Try and catch that as early as possible to make future step pattern checking easier.
Responding to User Feedback
We have a Google spreadsheet form that collects user feedback. You can see the responses here
, and you can subscribe to notifications to find out when users submit a form.
Recommended way to process feedback:
- Do it one challenge at a time. Turn on "filter" for the spreadsheet, select the url column, and pick which challenge to debug. Prefer the high-volume ones or the ones that are earlier in the curriculum.
- Open the challenge up on Khan Academy, and press edit, so that you can see the tests tab.
- Go to khan-cs-staging.appspot.com and find the corresponding challenge. It may not exist or be up to date, so regardless, always copy the latest tests into that one.
- Now open the same url in an incognito window or separate browser, to mimic being a fresh user of the challenge.
- Paste the code from the spreadsheet row into the editor. You'll often immediately realize what edge case hasn't been accounted for, and needs a new hint or pattern.
- Edit the tests in the original dev browser, save, and reload the user browser until you like the results.
- Open quickdiff.com and paste the original code from khan-academy into the left, and the changed code from khan-cs-staging into the right. Review to make sure the changes are what you expected. Don't introduce regressions!
- Now paste the changed code into khan-academy and save.
- Look up the user using the DevAdmin panel. If they have an email address, email them. If they don't, check their profile and see if they have any program that you can leave feedback on. If not, you can email their coach or just let it be. Email something like "Thanks for the feedback, we checked out your code. Our system didn't accept it because bla bla, we've added a message to try and make that more clear. Please try again."
- Cross off the row in the spreadsheet.