This page aims to make the research community aware of the possibilities that the online survey company SoJump .com (wjx.cn; in short, SoJump) offers, and it aims to make it easier for scholars to use it. We also aim to make SoJump more accessible to researchers who do not speak Chinese by taking the reader through the steps necessary to field a study in SoJump and by providing screenshots translated into English. This project (published in the Journal of Behavioral and Experimental Finance) is joint with Lianjun Li (University of Science and Technology of China), Lina Ang, Noah Lim, and Wei Jie Seow (all at the National University of Singapore).
(Latest update: 2/2024)
SoJump.com is an online survey platform (wjx.cn, Chinese name being Wen Juan Xing, literally “Questionnaire Star”) where users can create and field online surveys (homepage screenshot displayed in Figure A1). SoJump also offers a broader array of products, including assessments, registration forms, and online polls, among others (Figure A2 shows a screenshot from SoJump illustrating the types of surveys that can be created using the platform). Thus, similar to MTurk, SoJump offers researchers a valuable alternative to college students as a traditional source of study participants. We provide a step-by-step guide on how to field a study in SoJump. Since researchers typically use SoJump to conduct survey research, here we will focus on the questionnaire function to demonstrate how to use SoJump.
To create a survey, researchers first have to create an account on the platform. Importantly, to create an account, a researcher must have access to a Chinese phone number, as specified in SoJump’s Personal Information Protection policy (available at the following web address: https://www.wjx.cn/wjx/license.aspx?type=1). For foreign researchers residing outside of China, this most likely requires partnering with a Chinese colleague who also resides outside of China but still has access to a Chinese phone number, or teaming up with a colleague who is currently in China.
After entering the Chinese phone number, researchers can choose between two registration options: a free account or an Enterprise subscription. With a free account, SoJump allows researchers to collect up to 20 survey responses at no cost, which can be a useful feature for first-time users who wish to explore how the platform works. The free account includes access to several key interfaces: the dashboard, editing a survey, notifications, and data analysis.
With a subscription, more functionalities become available. Specifically, SoJump will enable options including item or block randomization, which are essential if a researcher wants to implement a survey experiment. Other appealing and commonly used functions that are offered to subscription accounts include informed consent and jump logic among others. Informed consent will be useful particularly to scholars at institutions where research must be approved by an Institutional Review Board (IRB) for ethics compliance. In that context, researchers are typically required to ask participants to provide their consent to take part in the survey. Jump logic allows researchers to introduce if-then statements in the survey without using a programming language. For instance, with jump logic, participants will only see a question conditionally on answering a previous question in a certain way. An example is a question that asks participants to specify how often they experience certain symptoms of COVID-19. With jump logic, participants would see the question only if they previously answered in the affirmative to a question about whether they have any COVID-19 symptoms.
Regardless of whether the account is free or subscribed, customer service and the Help tab are available to all researchers. The SoJump website explains the details of all its functionalities. Moreover, customer support is also available through phone hotlines as well as the Chinese instant messaging apps Tencent QQ and WeChat. These resources, however, are available only to Chinese speakers. Researchers who do not speak Chinese will find the Help tab useful as the text can be automatically translated through their browser.
The dashboard of a registered account presents a list of all running, completed, and unpublished surveys (Figure A3). The user interface is in Chinese, but we provide screenshots with all the relevant pages translated into English by the browser. From the dashboard, a researcher can choose to publish a survey (which makes the survey accessible to prospective participants), temporarily pause the data collection, or close the survey (meaning that the survey becomes permanently unavailable to participants). The dashboard also indicates the status of the study (published or unpublished), the number of responses received, and the time of the last update for each study. On the dashboard, there are also three drop-down menus for each questionnaire where a researcher can pick an action: designing, sending, or analyzing as well as downloading a questionnaire. The dropdown menus further break down each action (Figure A3).
From the Design Questionnaire feature, researchers can access the Design Wizard, edit the questionnaire, modify the questionnaire settings, customize the questionnaire appearance, monitor the process approval, and send red packets and prizes to participants. The Design Questionnaire feature also allows researchers to export the questionnaire to a Word document. Figure A4 shows the Design Wizard screen, which researchers can use to create a new survey or to edit an existing one. Figure A5 zooms in on the Edit questionnaire feature, which includes a plethora of additional specifications that a researcher can set for the questionnaire. For instance, time controls set the date and time when a survey can be accessed by participants; they also allow researchers to set how a questionnaire ends (e.g., by jumping to a specific page or showing a thank you message; Figure A6). Also, the order of questions can be randomized.
Similar to Qualtrics, researchers can choose between many (49) types of question formats, such as multiple choice, matrix, and so on. Questions can be created from scratch or imported from a template or another survey. Surveys can be programmed in any language, including Chinese and English.
After entering the survey questions, researchers can also customize the user interface, including its look and feel, the header and footer, and the university or company name of the researcher fielding the survey. Additional available features include a progress bar, a survey reminder service if participants do not take the survey after a prespecified period of time, survey privacy settings, and password protection.
The Quality Control tab is also accessed from this menu. Quality Control allows researchers to filter responses, i.e., to set criteria that determine whether a response is valid (Figure A7) or to set quotas according to custom rules set by the researcher, such as participant age (Figure A8). The quota feature is particularly useful for researchers who wish to study specific target populations or achieve specific balances in their sample. Other helpful features for quality control include built-in timers that measure how much time a participant spent on each page; rules to automatically filter out invalid answers; screening pages; and duplicate response checks.
Researchers can also design longitudinal studies in SoJump by contacting customer service and agreeing on a schedule for survey release. Specifically, a researcher can conduct a maximum of three waves of a survey with the same participants. However, SoJump constrains waves to be spaced out up to one month between each other, thus limiting the possibilities for longer panel studies using this platform. At the same time, shorter panel studies have a greater chance of reducing participant attrition with each wave (Ribisl et al., 1996).
Finally, a helpful design feature of SoJump lies in the ability to embed JavaScript code within the survey, although integration with outside web applications in a similar way to Qualtrics (Carter & Del Ponte, 2022) is not available at this time.
Through the Send Questionnaire menu, researchers can create a link and QR code to the survey and send it via WeChat, email, or text to contacts that they know. These features will be useful to researchers who wish to employ a snowball sampling technique (Goodman, 1961; Parker et al., 2019). Alternatively, they can use the SoJump sampling service, which will give them access to the 3 million pool of registered SoJump users, according to the company itself. The recruitment of participants provided by SoJump through its sampling service will be of interest to researchers who wish to use SoJump as a survey platform similar to Qualtrics, but at a lower cost (similar to MTurk).
If they choose to have SoJump recruit participants for them, when they submit their sampling service request, researchers can specify the inclusion/exclusion criteria for their study, including location, age, gender, occupation, and more (Figure A10). Custom qualifications according to unlisted criteria (similar to MTurk) are possible but they must be approved by SoJump after review.
After they are done specifying their target sample characteristics, researchers can submit a request to SoJump. Momentarily, they will receive a confirmation that the request was received (Figure A12) and a quote will be provided. Then, researchers will need to correspond with customer service to confirm the sampling service (customer service has limited ability to communicate in English). Throughout this process, researchers will receive notifications about each step, such as when the data collection has begun or when the desired target sample size is reached (Figure A13, Figure A14, Figure A15).
Once the survey has been launched, there is no limit to how much time a single questionnaire can be active, and a single questionnaire can be filled out by a maximum of 100,000 participants. Also, there is no limit to how many questionnaires can be published at the same time. However, for survey invitations by email, the limit is 60 messages per hour.
Through the Analysis & Download menu, researchers can access basic features for downloading the data, performing statistical analyses, and checking the quality of the data.
Regarding the download functions, SoJump offers a rich suite of options, ranging from downloading a survey report into Microsoft Word, downloading the full answer sheet into Excel, a direct download into the statistical software SPSS, and a high-speed download option for particularly large datasets.
Data analysis capabilities include descriptive statistics, cross-tabulations, custom queries, and completion rate analysis. During this step, researchers can check each response and manually reject a response if it is not deemed valid. To reject a response, researchers can click on the trash bin icon next to each response (Figure A14). SoJump will stop collecting survey responses when the number of valid responses has reached the desired target and researchers confirm that they are satisfied with the quality of the data by responding to the notification that the study is complete (Figure A15). Importantly, researchers have three days to vet the quality of the data. If researchers take no action within three days, SoJump will automatically assume that researchers are satisfied with the data and permanently close the study.
In addition to manual validity checks conducted by the researcher, SoJump also independently monitors the process of survey completion, checks the validity of responses, and deletes invalid ones before they are visible to the researcher. The criteria for automatic deletion are not specified on the SoJump website, but. when the authors contacted customer service, SoJump provided a more detailed explanation. Specifically, SoJump explained that invalid responses are deleted according to the following criteria: (1) responses that do not meet the requirements specified by the researcher; (2) failed attention checks; (3) questionnaires that are filled out too quickly or where the participant always selected the same answer (SoJump’s staff conducts this type of quality control manually); (4) repeated IP addresses. Attention check items are designed by SoJump and are proprietary. However, customer service offered an example: “This question is to check whether you have read the question carefully and answered it. Please pick the item “Agree” from the following choices: a) Completely disagree; b) Disagree; c) Neither agree nor disagree; d) Agree; e). Completely agree).
The service fee for the sampling service depends on factors such as inclusion/exclusion criteria, sample size, survey length, and question type. SoJump automatically compensates participants in proportion to the service fee (typically, 20-30% of the unit price, according to customer service). Importantly, researchers do not have control over the flat compensation fee that respondents receive from SoJump for participating. However, researchers can decide to compensate participants with additional bonus money. Bonuses can be distributed to respondents using the Chinese instant messaging app WeChat (through the “red packets” function, which is equivalent to similar services such as Venmo or Zelle in the U.S. or PayNow in Singapore). Alternatively, researchers can design lotteries or instant draws to further incentivize participation (Figure A16).
SoJump’s pricing for survey responses in 2022 started at 6 yuan per individual response (approximately 85 cents in USD). Similar to other survey companies, the price increases as a result of survey length and question types. For instance, an open-ended item is more expensive than a multiple-choice item. Researchers should keep in mind that the per-unit cost is in addition to the cost of the SoJump Enterprise subscription, which is necessary to run fully-fledged studies. In 2022, the cost for the Enterprise subscription amounted to 128 yuan per week, 380 yuan per month, or 2,880 yuan per year (as of December 2022, one U.S. dollar was equivalent to 7 yuan; thus, these amounts translate to approximately $18/week, $54/month, and $412/year). If paper invoices are required by the researcher instead of electronic ones, an additional express delivery fee of 22 yuan (~$3) is due. Also, if researchers wish to use SMS invitations instead of the SoJump standard sampling service, there is an additional fee (in 2022, it amounted to 0.10 yuan per SMS, with a minimum purchase of 10 SMS).
In SoJump, researchers running incentivized studies such as economic experiments or studies of decision-making are limited in their opportunities to pay individual bonuses to participants in ways that minimize the amount of personal information collected from the participant. Collecting personal payment information from participants such as WeChat phone numbers or even bank account information substantially increases the need for enhanced data security and poses significant risks to participant confidentiality, making this option undesirable. As an alternative, researchers can incentivize participants’ decisions in their studies by paying participants a variable number of tickets for a lottery or an instant draw. This option does not directly involve cash incentives; Yet, ultimately, lottery or instant draw winners receive real money that is disbursed by SoJump without requiring participants to share personal information with researchers. Thus, lotteries or instant draws are generally to be preferred to WeChat compensation.
A key limitation of SoJump pertains to the survey topics that are allowed on the platform. At the bottom of the Design Wizard (Figure A4), a disclaimer warns researchers that in compliance with Chinese laws and regulations, fielding surveys on sensitive topics is forbidden. Forbidden topics include “politics, military affairs, religion, beliefs, ethnicity, human rights, democracy, national sovereignty, national unity, diplomatic events, etc.” This list excludes many of the topics of interest to political scientists. Even so, when we fielded our surveys in July 2020 and June 2021, we were able to ask a standard item about participants’ ideology.
Also, unlike MTurk but similar to Qualtrics, communicating with participants is not possible. Related, quality control and sampling mostly rest within SoJump’s responsibility and are not fully disclosed to researchers, who however have the option to build quality control features such as comprehension and manipulation checks within their surveys (Kane & Barabas, 2019).
Carter, B., & Del Ponte, A. (2022). Integrating Web Applications into Popular Survey Platforms for Online Experiments. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01792-w
Goodman, L. A. (1961). Snowball sampling. The Annals of Mathematical Statistics, 148–170.
Kane, J. V., & Barabas, J. (2019). No harm in checking: Using factual manipulation checks to assess attentiveness in experiments. American Journal of Political Science, 63(1), 234–249.
Parker, C., Scott, S., & Geddes, A. (2019). Snowball sampling. SAGE Research Methods Foundations.
Ribisl, K. M., Walton, M. A., Mowbray, C. T., Luke, D. A., Davidson II, W. S., & Bootsmiller, B. J. (1996). Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations. Evaluation and Program Planning, 19(1), 1–25.