Political party ratings in the parliamentary elections, December 2020: a comparison of telephone survey and face-to-face interview results

In the first of December 2020, the Kyiv International Institute of Sociology (KIIS) jointly with the Democratic Initiatives Foundation conducted an experiment by comparing the ratings of political parties that were gleaned from in-person interviews (a study that was commissioned by the Foundation) and telephone interviews (that KMIS did for its Omnibus). Both surveys had the same sample size of 2,000 respondents, asked the same phrased questions and had the same party lists. In both cases, the question was the first in the questionnaire so that the influence of other questions was eliminated. Also, the survey was conducted practically during the same time period: the face-to-face interviews were held on December 5-13 and the telephone interviews took place on December 4-8. The only difference is that the Democratic Initiatives’ face-to-face survey was based on a random sample at the first stages of selection with quota selection at the last stage. The KIIS survey was based on the main sample of randomly generated mobile numbers (that were not selected from a specific database) without additional quotas, with only people who are 18 years or older living on Ukrainian territory being polled.

In the Luhansk and Donetsk oblasts, the survey was conducted only on territory that the Ukrainian government controls.

The statistical sampling error (with a probability of 0.95 and design effect of 1.5) does not exceed: 3.3% for indicators close to 50%, 2.8% for indicators close to 25%, 2% for indicators close to 10%, and 1.4% for indicators close to 5%.

We received the following results (for each of the methods, first there is a percentage in relation to all, and then in relation to those who intend to vote and are decided on their choice – these data come the closest to possible outcome of the elections).

According to the survey, there are three parties that lead in the rankings: OPFL, Servant of the People and European Solidarity. Depending on the research method, OPFL and Servant of the People take either first or second place, whereas European Solidarity is slightly behind them (but the difference is insignificant). In Appendix I, we discuss in detail to what the divergence in results that were derived from the different methods could be attributable.

Columns 5 and 6 show the results of combined data from the two methods (rounded to the nearest whole number). The last column 6 shows what the election outcome could be if they had been held in early December. Four parties would certainly get elected to parliament – OPFL, Servant of the People, European Solidarity and Batkivshchyna, while Strength and Honor, as well as For the Future, have good chances.

            Appendix I. What is the difference between the two survey methods and which results are more reliable.

In 2020, Ukraine and the world were confronted with the COVID-19 pandemic. The pandemic made its mark, among others, in the realm of conducting sociological research.  This particularly has become apparent in the growth of telephone interviews (known as, computer-assisted telephone interviews or CATI, which are safe for interviewers / respondents compared to the more “familiar” surveys that employ in-person (so-called face-to-face) interviews.

The sociological community regularly runs into accusations over discrepancies in results regarding socially sensitive issues, especially political party ratings. However, any discrepancies among authoritative organizations are attributable to methodic factors (that is, differences in research methodology) and not to deliberate manipulation or falsification.
The advent of telephone interview results has contributed to the “diversity” of the results, which has intensified the controversy over the different results. Moreover, some journalists, experts, politicians and ordinary citizens do not critically analyze the information. When comparing ratings, they do not take into account the different research methodologies and as a result make erroneous conclusions.

There are a number of methodical factors that could affect the results of a survey. Among the important ones, it’s worth considering the following (and the list is not exhaustive):

• Research method: in-person interviews, telephone interviews, online interviews, etc. For example, in-person interviews theoretically cover the entire population, but actually people in institutional settings are excluded (hospitals, prisons, army bases, etc.) Telephone interviews can capture these categories of the population but do not include the 4% of respondents who do not have telephones. Online interviews can only cover Internet users. In addition, during in-person interviews respondents are usually shown an election ballot. During telephone interviews the party list is read aloud and the respondent chooses one party he/she hears. The demographic profiles of respondents also differ. Finally each method has other different features that potentially could affect the outcome of results.
• Phrasing of questions and alternatives: The list of political parties / candidates within the context of questions on political ratings is especially important. Until there is a final list of registered parties / candidates, every list is hypothetically possible and it’s possible to study the preferences of the population based on this list. Still, the presence or absence of certain candidates can fundamentally influence a respondent’s choice. Ancillary information also has an impact: for example, whether the name of the party  is listed together with  its leader or is the respondent only offered a list of political parties; whether the candidates are listed with their nominating parties, biographical elements, etc.
• The order of questions on the questionnaire: preceding questions create a certain context for the respondent and this could affect which party or candidate the respondent chooses. Moreover, what could also even have an impact is the order of what we are first asking – parliamentary elections or the presidential election. Regarding substantially “leading” parties, if first we ask about the presidential election, this could  “pull up” the rating of the parties in the next question.
• Rating calculation method: there are different ratings. For example, a percentage of all the polled respondents, a % of those who are decided, etc. Often, the rating is calculated based on the one question of which party or candidate the respondent is voting for. However, some pollsters could use several questions to calculate ratings: for example, first they ask respondents if they intend to vote and then ratings are calculated based only among those who intend to vote.
• Statistical information to determine sampling parameters: the last population census was conducted in 2001, thus sociologists for the most part have to work “in the dark.” Some polling firms use data from the State Statistics Service, others use data from the Central Election Commission. For example, the regional structure of these sources somewhat differ. Additionally, sociologists control the structure of the sample with such parameters as region, type of settlement, sex and age, but unfortunately, do not have reliable information regarding other important data like level of education, for example.

Before first comparing that actual ratings of parties, the demographic profile of the respondents in both samples is compared below. For the survey sample of the in-person interview method, the task for each oblast, urban / rural population, and sex and age was determined in advance. The sample of the telephone interview was entirely random, and after receiving the findings, the region / type of settlement / sax and age were adjusted to the statistics with statistical weighting. Thus, based on such paramaters as, region, type of settlement, and the age and sex of respondents, both samples are very similar. At the same time,   what is more important are differences in other characteristics like level of education and prosperity.

As is noticeable, respondents who take part in telephone interviews are more educated (44.5% have a higher education compared to 34% when respondents take part in face-to-face interviews) and are more prosperous. These factors have an effect on political choices, which is why differences in samples have an impact on ratings.

Table 1

A profile comparison of the respondents in the in-person and telephone surveys

Since there is no reliable statistical information regarding, for example, the level of education of respondents, it is not possible to determine which method data is more accurate. Obviously, telephone interviews exaggerate the share of the population with a higher education, but most likely, surveys using the in-person interview method, on the contrary, exaggerate the share of the population wihtout a higher education. Therefore we are not weighing this parameter.

In 2019, KIIS conducted a poll using both methods before the elections. Then, on the one hand, both methods quite reliable “predicted” the election outcome. On the other hand, the in-person interview method proved to be more accurate.

However, in 2020 because of the COVID-19 pandemic, the quality of the in-person interview method may have deteriorated – the percentage of those who decline an interview may depend on the situation in the locality, and the respondent's decision to allow the interviewer into their home may depend on their level of caution or, for example, trust in the authorities and information from the Ministry of Healthcare.

That is why it is not possible to assess which method works best.

Below in Table 2 is a comparison of political party ratings. If meaningfully analyzing it, then both methods demonstrate consistent results, nevertheless, they virtually show 3 leading parties with fairly close support.

However, a trend is noticed in the in-person interview method (if talking about the top parties) whereby there is more support for OPFL and Batkivshchyna and less for Servant of the People and European Solidarity. When taking into consideration that among voters of the OPFL and Batkivshchyna parties are people who are poorer and fewer people with a higher education, such differences in the results are expected.

Table 2

Political party ratings in elections to the Verkhovna Rada of Ukraine

*) the difference is significant at the level of 5%

Thus, when comparing ratings from different polling firms, the following factors that may affect the results should be taken into account:

1. Survey method (as our experiment proved)
2. The list of parties or candidates
3. Phrasing of the question and alternatives. (For example, is the party leader indicated near the party, are candidates listed along with the nominating party, etc.).
4. Order of questions on the questionnaire.

      If the rating is not in the beginning of the questionnaire, the preceding questions create a certain context for the respondent. Also what could have an impact is what is being asked first – elections to parliament or the presidential election.

5. Rating calculation method.

If both data is available, then it is most correct to compare the percentages for all because it is for them that errors are calculated (errors for those who are decided are much higher).

The press release (in Ukrainian) was prepared by Anton Hrushetskyi, Deputy Director of KIIS, and KIIS CEO Doctor of Sociology Volodymyr Paniotto.