Run a Coding Comparison query


In this topic


 


Understand Coding Comparison queries

A Coding Comparison query enables you to compare coding done by 2 users or 2 groups of users.

It provides two ways of measuring 'inter-rater reliability' or the degree of agreement between the users: through the calculation of the percentage agreement and 'Kappa coefficient'.  

When you run a Coding Comparison query the results are displayed in Detail View:

ui_coding_comparison_query_results.gif

1  The node that contains the coding that is being compared.

2   The source name and source folder location.

3   The source size, which is calculated as follows:

4   The Kappa coefficient—this column is only available if you select Display Kappa Coefficient.  If the users are in complete agreement then the Kappa coefficient (K) = 1. If there is no agreement among the raters (other than what would be expected by chance) then the Kappa coefficient (K) ≤ 0.

5   The green columns show percentage agreement:

6   The red columns show percentage disagreement:

ic_top_of_page.gifTop of Page

Run a Coding Comparison query

  1. On the Explore tab, in the Queries group, click New Query, and then click Coding Comparison.

rn_explore_queries.gif

The Coding Comparison Query dialog box opens.

  1. Select the users to include in user groups A and B in the Compare coding between box:

  1. In the At box—click Select to choose specific nodes or nodes in selected sets, classifications or Search Folders.

  1. In the Scope box, click Select to choose specific sources.

  1. Select what you want to include in the results:

  1. Click Run.

When the query has executed, the results are displayed in detail view.

NOTE

ic_top_of_page.gifTop of Page

How is the percentage agreement calculated?

NVivo calculates percentage agreement individually for each combination of node and source.

Percentage agreement is the percentage of the source’s content where the two users agree on whether the content may be coded at the node.

For example, if the source is a document with 1000 characters, where:

then the percentage agreement is calculated as (800 + 50) ÷ 1000 = 85%.

For other types of sources, different units are used instead of characters for this calculation. For example, for audios and videos the unit of measurement is seconds of duration, and for pictures the unit of measurement is image area in pixels.

ic_top_of_page.gifTop of Page

How is the Kappa coefficient calculated?

Cohen’s Kappa coefficient is a statistical measure of inter-rater reliability which many researchers regard as more useful than the percentage agreement figure, since it takes into account the amount of agreement that could be expected to occur through chance. For more information, refer to the Wikipedia article Cohen's kappa.

NVivo calculates the Kappa coefficient individually for each combination of node and source.

If the two users are in complete agreement about which content of the source should be coded at the node, then the Kappa coefficient is 1. If there is no agreement between the two users (other than what could be expected by chance), the Kappa coefficient is ≤ 0. A value between 0 and 1 indicates partial agreement.

The Kappa coefficient is calculated as follows. (Note that the units of measure used in this calculation depend on the source type. For example, for documents the units of measure are characters, while for audios and videos the units of measure are seconds of duration.)

For an example of how NVivo calculates Kappa coefficients, you can download the Coding Comparison Calculation Examples spreadsheet. The Kappa values in column F are calculated (using spreadsheet formulae) from the agreement/disagreement figures in columns H, I, K and L.

ic_top_of_page.gifTop of Page

How should the value of Kappa be interpreted?

One approximate set of guidelines for interpreting the value of Kappa is:

Kappa value Interpretation
Below 0.40 Poor agreement
0.40 – 0.75 Fair to good agreement
Over 0.75 Excellent agreement

ic_top_of_page.gifTop of Page

Why can the value of Kappa be low when the percentage agreement is high?

Because the Kappa coefficient calculation takes into account the likelihood of the agreement between users occurring by chance, the value of Kappa can be low even though the percentage agreement is high.

For example, if most of a source has not been coded at the node by either user, but each user has coded completely different small sections of the source at the node, then the percentage agreement between the users will be high. But since this situation would be highly likely to occur by chance (i.e. if the two users had each coded a small section at random), the Kappa coefficient is low.

Conversely, if most of a source has not been coded at the node by either user, but each user has coded almost the same sections of the source at the node, then the percentage agreement between the users will again be high. But this situation would be highly unlikely to occur by chance, so the Kappa coefficient is also high.

These examples indicate why many researchers regard the Kappa coefficient as a more useful measure of inter-rater reliability than the percentage agreement figure.

ic_top_of_page.gifTop of Page

What does a negative Kappa coefficient mean?

A Kappa coefficient less than or equal to zero indicates that there is no agreement between the two users (other than what could be expected by chance) on which content in the source may be coded at the node.

ic_top_of_page.gifTop of Page

All my Kappa coefficients are 0 or 1. Is something wrong?

This most often indicates that one of the two users being compared has not coded any of the selected sources at the selected nodes.

In your Coding Comparison query results:

If the user’s work has been imported from another NVivo project, this may indicate that this user’s coding has not been imported correctly and needs to be re-imported.

When using the Import Project feature in NVivo to import coding:

ic_top_of_page.gifTop of Page

How can I calculate an average Kappa coefficient or percentage agreement across multiple sources or nodes?

NVivo calculates the Kappa coefficient and percentage agreement individually for each combination of node and source.

However, the results of a Coding Comparison query can be exported from NVivo as a spreadsheet (using the Export List command) to allow you to perform further calculations.

If you wish to calculate an average Kappa coefficient or percentage agreement for a single node across multiple sources, or for multiple sources and nodes, you will need to consider how you want to weight the different sources in the calculation. For example, do you want to treat each source equally, or do you want to give more weight to large sources than small sources?

For some examples of how average Kappa coefficients and percentage agreements can be calculated from Coding Comparison query results exported from NVivo, download the Coding Comparison Calculation Examples spreadsheet. This spreadsheet includes four examples (with the average Kappa coefficients and percentage agreements calculated using spreadsheet formulas):

If your project has different types of sources (for example, documents and audios), you may need to give further consideration to how you want to weight these different sources (since document size is measured in characters, while audio size is measured in seconds of duration).

ic_top_of_page.gifTop of Page