The DfE pupil level checking exercise opened on 26 September 2019. The objective of this exercise is for schools to ensure that grades are correct for those pupils who will collectively form the school’s contribution to performance tables.
During the course of the first day, it became apparent that grades for several BTEC qualifications were missing. Pearson has acknowledged that they did not send all the grades to the DfE for this exercise. This omission will be rectified. It is possible that other qualifications have been omitted or incorrectly calculated. This is being investigated by DfE analysts.
The DfE calculate headline measures and national level statistics based on the complete set of data they receive from awarding organisations, together with other data they hold about pupils such as prior attainment data from the end of Key Stage 2. If that data is missing it means that measures, such as Progress 8, will not accurately reflect schools’ performance. It appears that this has been the case.
Mistakes with this checking exercise have happened before. For example, the DfE have in the past used an incorrect file which identified some pupils on school roles when they had been removed . Where errors occur the DfE rectifies them over the course of the year.
There are a number of serious issues arising from this situation:
1. Schools have always been invited to correct the information about their pupils in this exercise. However, where a system-wide error has been established it would be wrong of the DfE to insist that all errors are corrected at school level. In some cases, the resulting workload for schools would be considerable and unreasonable.
2. Because Progress 8 is a measure which is averaged across all schools’ performance, even those schools which did not enter pupils for a BTEC qualification are likely to see a change in their figures following a recalculation. All schools should therefore view overall Attainment 8 and Progress 8 figures with scepticism until they are recalculated. Schools with no BTEC entries may see a fall in their overall Progress 8 figure following a national recalculation.
3. Many schools have participated in collaboration exercises with one or more providers. These exercises have included the qualifications which have been omitted from the DfE figures and are therefore likely to be more accurate for some schools than the provisional DfE figures. The collaboration exercises are intended to give early insight for schools into their approximate performance, but in this case have also helped identify a system problem. Because the issue appears to be confined to non-GCSE qualifications, collaboration estimates for the mathematics, English and EBacc elements of Attainment 8 and Progress 8 are likely to be broadly accurate.
4. Ofsted helpfully indicated in their autumn update that these collaboration figures could be used by schools during inspections in the period before DfE figures were published at the end of September. Given that the DfE figures do not properly reflect performance, it would be helpful if Ofsted continue to acknowledge these figures until corrections are made by the DfE.
5. Although the data shared in September is only visible to schools, the DfE uses the same dataset to publish provisional school level tables intended to support parents as they choose secondary schools for their children. It would be wholly inappropriate for the DfE knowingly to use incorrect data for this purpose.
We have communicated with the DfE and Ofsted about this issue.
We believe the DfE needs to:
• update the tables checking website used by schools to acknowledge that some qualifications have been omitted or are
being incorrectly calculated in performance measures
• agree to update all schools’ data and let schools know when this has happened
• extend the window for checking data if necessary
• make clear what schools should do during the checking exercise with regard to qualifications which are missing at a
• ensure that data which is published on public sites uses correctly processed data from a complete dataset
We believe Ofsted needs to:
• extend the period where inspectors would accept collaboration data, until DfE data is recalculated
• ensure inspectors do not draw any conclusions from the incorrect data
We advise schools to:
• wait until the DfE have reissued figures before discussing them with staff or governors
• understand that all schools will be affected by this recalculation
• wait until the DfE clarifies its approach before making wholesale changes to the checking exercise
ASCL has made a data toolkit available for schools which uses the data checking file. The tookit has three main uses:
• A utility to assist data managers to check the data, reducing the workload involved in the process.
• Interactive, visual tools and graphs for Attainment 8 and Progress 8.
• A collaboration process which allows schools to upload prior attainment and grade information to construct subject level
This is now open at www.ascl.smidreport.com
. Schools can use this toolkit immediately to assist with the checking process. If new files are produced by the DfE they can be uploaded and will automatically remove any old files. For the second year we are making the ASCL Data toolkit available free online at ascl.smidreport.com
The toolkit has three main uses:
- A utility for data managers to assist the data checking process.
- Visual and analysis tools for Progress 8.
- Data collaboration to allow schools to share subject level results and to crowdsource the national transition matrices several months before the DfE release them.
Whilst the first function can be used immediately, we recommend schools wait until the DfE refresh each school’s pupil data file before using the analysis sections. This is because the open bucket of Progress 8 has been affected by the omission of BTEC results. The toolkit will be updated with correct national figures as soon as the DfE release them.
ASCL will continue with our discussions with the DfE on this matter, and will update members about this process accordingly.
Update (1pm, 27 September 2019)
The DfE has advised that the omitted Pearson BTEC results will be uploaded to the late results page of the CE website this afternoon, emails will be sent to schools informing them of this.
We plan to provide updated csv files and update the summary page of the CE website for all schools, ahead of publication, but as I hope you can appreciate, I can’t give you the exact timing of that right now, just that it will be as soon as possible, and we’ll keep the checking exercise homepage updated with messages about this.
With regards to GCSE statistics, this is being investigated, however, the individual results are included in the data, the DfE is only looking into their inclusion in the measures.
Update (3.30pm 27 September 2019)
Two further updates have now been received. One from Ofsted, and a second message from the DfE.
Ofsted has confirmed that inspectors are being notified that they should continue to follow the recent guidance about collaboration data from third parties until the DfE has resolved the issue with the checking data.
The DfE has advised that the following message will appear on the homepage of the checking exercise website:
"We have received and uploaded the missing Pearson BTEC results. Schools with missing results can now view these on the ‘View late results’ link towards the bottom of this page. Schools do not need to request the addition of the previously missing BTEC results.
The inclusion of these late results will not automatically update the headline measures included on the summary page of this website, this is an interim step, to enable schools to see a complete set of pupil level data.
Schools should continue to complete the key stage 4 checking exercise, based on the pupil level data provided, by 5:00pm on Thursday 10 October.
A further update to the website will be provided as soon as possible before publication. We are treating this as a priority, and will share the timing of this as soon as we have it. This will refresh the csv files and summary page.
These refreshed statistics will include the missing Pearson BTEC data (but no other late results) and will update the headline measures, including Progress 8 and Attainment 8 at both school level and national level.
This updated information will then be published in the provisional performance tables in October.
Please continue to access this ‘Home’ page, where further updates will be provided."
Update (11.30am 30 September 2019)
The DfE has confirmed they are aware that GCSE Statistics results are not counting in the headline measures as expected. They also advise that this issue will be corrected in the update to the csv files and summary page (please see the BTEC results update). This correction will also be included in the DfE’s provisional performance tables publication.
Individual pupil level GCSE statistics results are included in the checking exercise data, and schools should continue to check their data as usual.
Update: BTEC results (8 October 2019)
The DfE has uploaded refreshed provisional data onto the summary page and updated the csv files on the tables checking website. These updates reflect the Pearson BTEC results loaded to the website on 27 and 30 September 2019 and some Pearson Certificate in Digital Applications results, which should have been included in the provisional data – these individual results are available to view on the ‘View late results’ link.
The refreshed headline measures on the summary page also incorporate the updates the DfE has made to the way GCSE statistics is being reported.
No other late results will be included in the recalculated headline measures or provisional publication, but they will continue to be uploaded as they are received and will be included in the revised publication in January.
These refreshed headline measures will be published in the provisional performance tables in October.
Schools should continue to complete the Key Stage 4 checking exercise, based on the pupil level data provided, by 5pm on Thursday 10 October.
Please continue to access the ‘Home’ page, where further updates will be provided.
If ASCL members notice any further systematic omissions, please call the Hotline on 0116 299 1122 and let us know.