Historical Data

This page includes several graphs providing insight into the historical development of rOpenSci’s software peer review process. The first three graphs includes results distinguishing general review from statistical software review starting from September 2021. Prior to this time, all reviews were “general”.

Package submissions

This chart shows the total number of packages submitted for review each quarter. A value of 12 per quarter equates to around one submission per week. The controls immediately below, and in all of the following plots, can be used to control the first year of data shown in the plot.

submissions = {
    return transpose(submissions_in).map(row => ({
        ...row,
        date: new Date(row.date),
        submissionYear: new Date(row.date).getFullYear()
    }));
}

minYear = Math.min(...submissions.map(d => d.submissionYear));
maxYear = Math.max(...submissions.map(d => d.submissionYear));
viewof submissionYear = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear // default = (min + max) / 2
    }
)

submissionsFiltered = {
    return submissions.filter(d => d.submissionYear >= submissionYear);
}

Plot.plot({
    style: `
        overflow: visible;
    `,
    marginLeft: 60,
    marginBottom: 50,
    x: {grid: true},
    y: {grid: true},
    marks: [
        Plot.ruleY([0]),
        Plot.lineY(
            submissionsFiltered,
            {
                x: "date",
                y: "submissions",
                stroke: "type",
                fontSize: 18
            }
        ),
        Plot.text(
            submissionsFiltered,
            Plot.selectLast({
                x: "date",
                y: "submissions",
                z: "type",
                text: "type",
                textAnchor: "start",
                dx: 3,
                fontSize: 18
            })
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisX(
            {
                fontSize: 20,
                label: "Year",
                labelAnchor: "center",
                labelOffset: 40,
                ticks: []
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 24,
                label: "Nr. submissions / quarter",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        ),
    ]
})

Packages under review

This chart shows numbers of packages simultaneously under review each quarter.

reviews = {
    return transpose(reviews_in).map(row => ({
        ...row,
        date: new Date(row.date),
        reviewYear: new Date(row.date).getFullYear()
    }));
}

// Use 'minYear' and 'maxYear' from above
viewof reviewYear = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear
    }
)

reviewsFiltered = {
    return reviews.filter(d => d.reviewYear >= reviewYear);
}

Plot.plot({
    style: `
        overflow: visible;
    `,
    marginLeft: 60,
    marginBottom: 50,
    x: {grid: true},
    y: {grid: true},
    marks: [
        Plot.ruleY(
            [0]
        ),
        Plot.lineY(
            reviewsFiltered,
            {
                x: "date",
                y: "reviews",
                stroke: "type",
                fontSize: 18
            }
        ),
        Plot.text(
            reviewsFiltered,
            Plot.selectLast({
                x: "date",
                y: "reviews",
                z: "type",
                text: "type",
                textAnchor: "start",
                dx: 3,
                fontSize: 18
            })
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisX(
            {
                fontSize: 20,
                label: "Year",
                labelAnchor: "center",
                labelOffset: 40,
                ticks: []
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 24,
                label: "Nr. active reviews / quarter",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        ),
    ]
})

Review durations

Review durations are displayed here in terms of two main measures:

1. Overall Review Durations, which are the durations of the entire review process, from the date packages are first submitted until they are accepted.
2. Individual Review Times, which are the times spent by individual reviewers on their package reviews.

The first two of the following charts are time series charts plotted against the months in which each review started, with data slightly smoothed to aid visual display. Data for “stats” reviews may not extend to current date, if data for the current quarter are insufficient for statistical analyses. In these cases, data for “general” reviews are also truncated to same length as those for “stats” reviews. Data for “all” reviews should nevertheless always extend to the current date.

Following the two time series charts are two histogram plots, again for the two distinct measures described above.

Overall Review durations

revDur = {
    return transpose(revDurIn).map(row => ({
        ...row,
        date: new Date(row.date),
        revDurYear: new Date(row.date).getFullYear(),
    }));
}

viewof revDurYear = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear,
    }
)

revDurFiltered = {
    return revDur.filter(d => d.revDurYear >= revDurYear);
}

maxRev = 1.05 * Math.max(...revDurFiltered.map(d => d.type === "all" && d.dur));

Plot.plot({
    style: `
        overflow: visible;
    `,
    marginLeft: 60,
    marginBottom: 50,
    x: {grid: true},
    y: {grid: true, domain: [0, maxRev]},
    marks: [
        Plot.ruleY(
            [0]
        ),
        Plot.lineY(
            revDurFiltered,
            {
                x: "date",
                y: "dur",
                stroke: "type",
                fontSize: 18
            }
        ),
        Plot.text(
            revDurFiltered,
            Plot.selectLast({
                x: "date",
                y: "dur",
                z: "type",
                text: "type",
                textAnchor: "start",
                dx: 4,
                fontSize: 18
            })
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisX(
            {
                fontSize: 20,
                label: "Year",
                labelAnchor: "center",
                labelOffset: 40,
                ticks: []
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 24,
                label: "Review duration (months)",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        ),
    ]
})

Individual Review Times

revHours = {
    return transpose(revHoursIn).map(row => ({
        ...row,
        date: new Date(row.date),
        revHoursYear: new Date(row.date).getFullYear(),
    }));
}
revHist = {
    return transpose(revHistIn)
}

viewof revHoursYear = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear,
    }
)

revHoursFiltered = {
    return revHours.filter(d => d.revHoursYear >= revHoursYear);
}

maxHours = 1.05 * Math.max(...revHoursFiltered.map(d => d.type === "all" && d.hours));

Plot.plot({
    style: `
        overflow: visible;
    `,
    marginLeft: 60,
    marginBottom: 50,
    x: {grid: true},
    y: {grid: true, domain: [0, maxHours]},
    marks: [
        Plot.ruleY(
            [0]
        ),
        Plot.lineY(
            revHoursFiltered,
            {
                x: "date",
                y: "hours",
                stroke: "type",
                fontSize: 18
            }
        ),
        Plot.text(
            revHoursFiltered,
            Plot.selectLast({
                x: "date",
                y: "hours",
                z: "type",
                text: "type",
                textAnchor: "start",
                dx: 4,
                fontSize: 18
            })
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisX(
            {
                fontSize: 20,
                label: "Year",
                labelAnchor: "center",
                labelOffset: 40,
                ticks: []
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 24,
                label: "Review time (hours)",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        ),
    ]
})

Histograms of Review Durations and Times

The following plots are histograms of overall durations of the entire review process (in days), followed by durations required for individual reviews (in hours). The “Start year” filter removes all data prior to the specified year.

viewof year = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear,
    }
)

revHistFiltered = {
    return revHist.filter(d => d.year >= year);
}

Plot.plot({
    height: 300,
    marginLeft: 60,
    marginTop: 50,
    marginBottom: 50,
    y: {grid: true},
    marks: [
        // Default for all preceding plots is the "Tabluea10" colour scheme:
        // https://www.tableau.com/blog/colors-upgrade-tableau-10-56782
        Plot.barY(
            revHistFiltered,
            Plot.binX(
                {y: "count"}, 
                {x: "days", stroke: "#4e79a7", strokeWidth: 2})
        ),
        Plot.text(
            ["Overall review durations in days"],
            {
                frameAnchor: "top", 
                fontSize: 24, 
                dy: -30}
        ),
        Plot.axisX(
            {
                fontSize: 14, 
                label: "", 
                labelArrow: false, 
                dy: 5}
        ),
        Plot.axisY(
            {
                fontSize: 14, 
                label: "", 
                labelArrow: false}
        ),
        Plot.axisY(
            {
                fontSize: 18, 
                label: "Frequency", 
                labelAnchor: "center", 
                labelOffset: 55, 
                ticks: []
            }
        )
    ]
})

Plot.plot({
    height: 300,
    marginLeft: 60,
    marginTop: 50,
    marginBottom: 50,
    y: {grid: true},
    marks: [
        Plot.barY(
            revHistFiltered,
            Plot.binX(
                {y: "count"}, 
                {x: "hours", stroke: "#4e79a7", strokeWidth: 2}
            )
        ),
        Plot.text(
            ["Individual review times in hours"],
            {
                frameAnchor: "top", 
                fontSize: 24, 
                dy: -30
            }
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false,
                dy: 5
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 18,
                label: "Frequency",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        )
    ]
})

Numbers of Editors

Warning in editor_vacation_status(airtable_base_id): Not all airtable editor
names match slack names. GitHub handles of mis-matches: ldecicco-usgs

rOpenSci relies on our team of volunteer editors. We can only provide review services if we have sufficient numbers of editors available to handle submissions. It is therefore important to monitor historical trends in overall editor availability. This graph shows total numbers of editors over time, and corresponding lines for numbers of those engaged during each time period in ongoing reviews (“Busy”), and numbers who are not handling any submissions during each period and are therefore free to accept new submissions (“Free”).

numEds = {
    return transpose(numEdsIn).map(row => ({
        ...row,
        month: new Date(row.month),
        numEdsYear: new Date(row.month).getFullYear(),
    }));
}

viewof numEdsYear = Inputs.range(
    [minYear, maxYear],
    {
        step: 1,
        label: "Start year",
        value: minYear,
    }
)

numEdsFiltered = {
    return numEds.filter(d => d.numEdsYear >= numEdsYear);
}

Plot.plot({
    style: `
        overflow: visible;
    `,
    marginLeft: 60,
    marginBottom: 50,
    x: {grid: true},
    y: {grid: true},
    marks: [
        Plot.ruleY(
            [0]),
        Plot.lineY(
            numEdsFiltered,
            {
                x: "month",
                y: "n",
                stroke: "status",
                fontSize: 18
            }
        ),
        Plot.text(
            numEdsFiltered,
            Plot.selectLast({
                x: "month",
                y: "n",
                z: "status",
                text: "status",
                textAnchor: "start",
                dx: 4,
                fontSize: 18
            })
        ),
        Plot.axisX(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisX(
            {
                fontSize: 20,
                label: "Year",
                labelAnchor: "center",
                labelOffset: 40,
                ticks: []
            }
        ),
        Plot.axisY(
            {
                fontSize: 14,
                label: "",
                labelArrow: false
            }
        ),
        Plot.axisY(
            {
                fontSize: 24,
                label: "Number of editors",
                labelAnchor: "center",
                labelOffset: 55,
                ticks: []
            }
        ),
    ]
})