FCO Records: Policy & Practice
gov.uk/fco
Information Management Department
FCO Records: Policy and Practice
Contents
Introduction
2
FCO Records: Myth & Reality
3
What we keep and why
8
Opening historical records to the public
11
FCO legacy records
19
Records in the future
21
Summary of discussion
26
Further information
30
1
Introduction
On Friday 17 May 2013 the Foreign and Commonwealth Office (FCO) welcomed academics and research students to an open forum to discuss and discover more about FCO Records. The aim of the day was to inform the audience about FCO record keeping and the process for releasing files to the public, and to receive feedback and questions from the academic community about their experiences in accessing FCO records. The forum was organised by the Information Management Department (IMD) in the FCO which is home to the Archive Management Team, the Knowledge and Information Management Team, the Freedom of Information (FOI) and Data Protection Act (DPA) team and the FCO Historians. This publication reproduces the presentations given in the morning session and provides a summary of the discussion from the question Christine Ferguson, Departmental Records Officer, addressing the audience and answer session in the afternoon. The Programme was as follows: 11.00
Welcome: Christine Ferguson, Departmental Records Officer Presentations: FCO Records: Myth & Reality What we keep and why Opening historical records to the public FCO special collections Records in the future
12.30 Lunch 13:30 Audience Q&A with panel: Professor Patrick Salmon, FCO Chief Historian (Chair) Martin Tucker, Head of the Archive Management Team John Thompson, Senior Sensitivity Reviewer Rosie Dyas, Head of the Information Rights Team Jim Daly, Sensitivity and Publications Reviewer Carryl Allardice, Head of the Knowledge and Information Management Team 15:00 Close
2
FCO Records: Myth and Reality Christine Ferguson, Departmental Records Officer
Welcome to the FCO—it is the first time we have held an event of this kind, specifically focussing on management of our public records. Why? The clue lies in the title of this presentation: we have seen a fair amount of criticism of the FCO in parts of the media, suggesting deliberate attempts to conceal information, particularly in relation to the colonial period and the Mau Mau veterans High Court case. By way of illustration, see this sample of media reporting and compare it to the extract from the High Court judgement in the Mau Mau case:
“Sins of colonialists lay concealed for decades in secret archive What better place to bury thousands of documents from former colonies than one of the government's most secure facilities?” “Secret government files from the final years of the British empire are still being concealed despite a pledge by William Hague, the foreign secretary, that they would be declassified and opened to the public.” “Many historians remain suspicious of the FCO and believe it may seek to retain some of its secret files……....................... …………….........warns that the FCO's history of concealment and denial is such that the public should also continue to be sceptical” “Perhaps even recent foreign secretaries have not known what the building holds. This country's culture of secrecy, aided by bland, official amnesia and mendacious cover-ups, has contrived to hide — or "lose" — some 2,000 boxes of records relating to 37 former British administrations.”
“I am anxious to emphasise (given the nature of some of the publicity about this case) that my clear impression is that, in the context of the present court proceedings and whatever may have happened in earlier years before claims were intimated, the defendant has sought scrupulously to comply with its disclosure obligations………….The defendant continues to take its disclosure obligations seriously and has reported the discovery of some further documents of potential relevance in a letter to the court of 30 July 2012 written after the conclusion of the present hearing.” Extract from the court judgement by The Hon. Mr Justice McCombe in Mutua and Others v FCO, 5 October 2012
3
Obviously, the media story is not always the most accurate. It seemed to us that there was scope to be more proactive in communicating what we are doing on the public records front. Over the past couple of years, we have put a lot of information onto the FCO website, now on gov.uk since we moved to a single Government website towards the end of last year. Thought it would be good to have the opportunity as well to meet directly with key stakeholders in the academic community, so that we can explain things in a bit more detail, hopefully demystify some of the processes involved in selecting and preparing files for public release and also, of course answer any questions you might have, which is why we have dedicated the afternoon session to a Q and A panel to make sure we cover whatever issues may be on your minds. Also very much welcome your views—are we meeting your needs? Can you access the information you want readily? Are there other things you would like us to be doing? To keep to our schedule, I propose to do presentations from our side this morning to give you the overview—ask you to make a note of any Qs and save them until the session after lunch, when we can drill down into any areas of particular interest to you. That is, broadly speaking, our motivation and plan for today’s event. We are delighted that so many of you have made the effort to join us. So, returning to my theme of Myth and Reality, what is the Reality? i) Our archive management in the past has not been as good as it should have been
Although we were able to deliver fully on our legal disclosure obligations in the Mau Mau case, as confirmed by the judge, given the age of the files we should obviously not have been holding them in the first place or at least not without a clearer idea of why we were holding them, what they contained and what we were intending to do with them.
Background to that situation is set out comprehensively in the report by Anthony Cary, former High Commissioner in Ottawa, following a detailed internal investigation he conducted. For anyone who is interested but has not seen the report, it is available on the gov.uk website and we have a couple of hard copies here also. Do not intend to go into the whys and wherefores in detail now but would highlight that Mr Cary was quite clear that there was no intent to conceal anything and that the staff handling the disclosure and related FOI requests had acted in good faith. He highlighted a number of weaknesses in our archive management practices which we’ve been working hard to address since then.
4
Important to distinguish here between the regular FCO departmental files—created by FCO in London or at Embassies and High Commissions overseas where we have good track record of transferring files to The National Archives (TNA) at the due time and other material held in the archive.
Key weakness identified by Mr Cary was previous practice of accepting odd collections of material into the archive without clarity over their status or ultimate destination. Key improvement is that we have now drawn up a high-level inventory of what is held in the archive—that was published on our website last year. Everything on that inventory is now held under a relevant provision of the Public Records Act (PRA).
Work is currently underway on a more detailed audit of the material, some of which dates back to 19th century, to help us determine whether it should be selected for transfer to TNA (if they are interested in taking it), offered to another place of deposit or possibly a library or academic institution, or, if of no real interest to anyone, no longer be kept at all. There will be an opportunity for you to express a view on this when we come to the item on the special collections (a term we are using to differentiate from regular FCO file holdings).
ii) We face significant challenges
To deal with this legacy material at the same time as a doubling of our normal transfer workload following the Govt’s decision to release public records after 20 years rather than 30 years as hitherto. As no doubt all are aware, this change is being implemented over a 10 year transition period starting in 2013. So this year and the next 9 years we will be transferring two years worth of files each year—itself a big ask, particularly in the current economic climate with downward pressure on civil service staffing and budgets.
We committed ourselves to an ambitious timetable for releasing the entire collection of colonial administration files, the so-called “migrated archive” within 2 years. We are on track to complete this but it has caused a backlog in transfer of FCO departmental files which we’re working hard to rectify.
iii) We are absolutely committed to meeting our public records obligations in full
True from Foreign Secretary (FS) downwards (number of public statements to this effect); both FS and Mr Lidington take a keen interest and we are keeping them upto-date on progress; in addition, we have a very dedicated archive team working hard to increase output of files.
5
Despite austerity, have secured additional resources for the public records work: doubled Sensitivity Reviewers from 14 to 28, have additional admin support staff and currently strengthening management layer in archive team, so well placed to meet demands ahead.
Nearly doubled the number of files transferred to TNA in 2012 (see below)
Files 12000
10000
8000
6000
Files
4000
2000
0 2009
2010
2011
2012
Number of files transferred to The National Archives by the FCO in 2009 - 2012
This includes more than half of the colonial administration files (remaining 3 tranches available in July, September and November). It is worth noting that normally, only around 40% of any given series of FCO files would be selected and the rest destroyed. In the case of the colonial files, given the prevalence of conspiracy theories, we agreed with TNA and with our Independent Reviewer, Professor Badger, that none of the files would be destroyed. There has been a minimal amount of material—less than 1%—redacted or not open at TNA. Any closed material has followed PRA or FOIA requirements e.g. for personal data. NB there is no FOI exemption for embarrassing the FCO/HMG so that has not been a factor. The whole process has been overseen by Independent Reviewer.
Despite significant reinforcement of our efforts, we cannot do everything as quickly as we—or you—might like. We want to be sure we are prioritising material of widest/greatest interest. To do this, we draw on information kindly supplied by TNA about most frequently read classes of files. We will be explaining in a bit more detail in the next talk, but broad priority has been:
6
1. Completion of colonial files release. 2. Regular FCO departmental file transfers, including clearing the backlog of 1981/82 files. 3. Special Collections (to be prioritised with input from Historians and others). 
We are committed to transparency about how we are tackling the challenges I have outlined and when records will be available to read—as mentioned, lot of info on our website, including a timetable for transfer of the colonial administration files and indicative dates for release of backlogged 1981 and 1982 files; we will keep that updated as records transfer to TNA and as plans develop for future years.

If there is more information you would find it useful for us to share in that way, please let us know.
So that is an overview of the reality, as seen from the FCO perspective. In the following presentations, we will look in a little more detail at how we decide what records to keep in the first place, how decisions are taken about which records make it to TNA and what material needs to remain closed, at how we are planning to tackle the special collections and at the forthcoming challenge of electronic records. Hope you find the day of interest. We are very keen to get your feedback so please complete a feedback sheet before you leave. That will help us to decide whether we should run a similar event at some point in the future and, if so, will help to shape the content. Thank you for your attention.
7
What Should We Keep and Why Carryl Allardice, Head of Knowledge and Information Management Team
Why are we talking to you about what the FCO should keep and why? Currently when you go to The National Archives to consult FCO papers, you and other researchers are only looking at paper records, because all the material we have transferred to The National Archives has been in paper format. But by 2016 we will start to prepare to transfer both paper and electronic records to TNA, and we will be the first Government Department to transfer electronic records in bulk from 2017 onwards. We will gradually transfer more and more electronic records to TNA, and fewer and fewer paper files. We will not print our any of our electronic records—once transferred to TNA in electronic format it will be TNA’s responsibility to maintain the FCO’s electronic records in digital format for future generations to read. Therefore we need to make sure that when we transfer electronic information to TNA in the future, we transfer FCO information that you want to look at, that it is easy for you to do research and to find the information that historians and researchers are looking for. As you know, the whole information lifecycle starts at the point of creation and knowing what we need to keep is therefore really important for the FCO.
Keeping the right information In the paper world it was a lot easier for FCO staff to look at a paper file and to work out what needed to be kept and what didn’t need to kept. But in an electronic world it is a lot harder to work out what we should keep, and it’s also a lot easier to delete information, and not to keep information that really needs to be kept. One of the greatest risks which we and other Government Departments are managing is the need to ensure that the right information is kept. In the FCO we regard information and knowledge as core assets, and although we provide guidance to staff about what we need them to keep to provide evidence of decisions, so we can account for those decisions, we also need to staff to know:
why we need to keep the information, what information we need to keep, how long we need keep information for in a digital format (e.g. delete it after 7 or 10 years?) and what to get rid of when it is no longer needed.
8
Today’s public records are also not just traditional records in a digital form. There are new types of digital objects such as wikis, blogs, emails, instant messaging which did not exist in the paper world. If information has value to the FCO, it must be retained. If it does not have value it should be disposed of—unless it is required to meet our legal obligations e.g. tax and financial records.
What is different between keeping paper and keeping electronic information? Over the last 100 years a paper file when it has been opened in an FCO Department, business unit or overseas post, is held for 3 years. In the 4th year, staff in that Department or overseas Posts are expected to review the contents of the paper file and to weed it. In the process, staff dispose of any ephemeral material in the file that has no long-term business value. The file is then sent to the Archives in Hanslope Park where it will traditionally remain for the next 27 years before it is sensitivity-reviewed and transferred to The National Archives at the 30 year mark. When I started working at the FCO in 2005 there were 3 Paper File Collectors employed to collect all the FCO’s paper files in London, by 2007 there were 2 Paper File Collectors and since 2010 there has been just one Paper File Collector. So over the last 8 years there has been a considerable decline in the number of paper files that the FCO is keeping, as increasingly, staff keep only electronic information in electronic files. Our electronic record keeping is the oldest in Government and dates back to 1992 when records were kept on a system called Aramis. At the time the FCO was still keeping large quantities of paper records, and often it was keeping both the hard copy and electronic copy. It was not until 2000 when the FCO moved to a new electronic record system called Registry, that the electronic record started to have primacy. Up until that point many staff regarded the paper copy as the primary record. From 2000 onwards we start to see a steady decline in the number of paper files being kept by staff in London. So over the last 12 years we are now down to steady trickle of paper files finding their way to the FCO’s Archives. However in 2008 we rolled out a new electronic record management system called iRecords. We also migrated all the legacy content from the UK Registry system and all the overseas records which were kept by our staff in our electronic Registries at posts overseas—and the legacy records went into iRecords—which now contains about 11- 12 million records; and last year we migrated the 6 million earlier legacy records from Aramis into iRecords. So the good news is that all our electronic records wherever they were created in the world, are sitting in our electronic records management system – all 17 to 18 million of them!
9
Why does it matter to you? The not-so-good news is that unlike in the paper world we have not weeded anything in the 4th year. It is all there. Some of you may think that is actually very good news that we still have all our electronic records since 1992 in one place, and that we have not deleted anything. But there is a vast quantity of duplication. If we exclude what we estimate to be duplicate records then the figures drop dramatically—we estimate we have approximately 4.5 million unique electronic records. There is still a lot of the ephemeral stuff and it is not in a series of neatly organised files, as it would be in the paper world—it is split into lots of different electronic folders with different folder names, and there are lots of duplicate records. Another issue is the fact that many staff only register the final version of a document while leaving all the drafts in the shared area, rather than keeping the story intact and putting all the drafts and the final version together in the records system. It is a problem which we know is replicated across Government.
What should we be keeping? Digital or electronic public records are not unique, fixed and static objects like records are in the paper world. We have developed ‘What to Keep’ (WTK) guidance for staff drawing on best practice from The National Archives, and other government departments like Cabinet Office, HMT, DfID and Home Office. Last year my team introduced a new programme of Information Management Health Checks—which are not quite information audits—but the Health Checks are designed to identify areas of improvement. We then work with Departments and Directorates to develop an action plan and as part of the follow-up process, we customise our ‘WTK guidance’, so we have for example WTK for Private Office, WTK for our Crisis Teams, for HR, Corporate Procurement and WTK for Legal Advisers in addition to our generic WTK. As we start to consider de-duplication of our digital records, we want to make sure that what we are keeping is what future historians and researchers want us to keep. We do not want to delete electronic records which should be kept. So as we work out what we should be keeping we would like to hear from you about what you think should be kept, and we would like you to let us know.
10
Opening Records to the Public Martin Tucker, Head of Archives
Introduction I would like to give a brief outline of the FCO's archival records management and transfer programme and show how the transition to 20-year release fits into our overall records transfer programme. In terms of its records the FCO’s records are very wide–ranging in scope and although many cover political and commercial diplomacy the FCO also creates a very large number of case files, for instance on consular cases. Predominantly the FCO is now creating electronic records but as I’ll show later we are still managing very large amounts of archive paper records. The great majority of records which we transfer to The National Archives over the next 10 years up until 20-year release will be paper records. I joined the FCO in 2008 as an information specialist and I've been responsible for the FCO's archive records since 2010. I will be covering:
The work of the Archive Management Team. Our current Archive holdings. How the FCO manages transfer process. The FCO records transfer programme. Impact of the 20-year rule. Resources. Volumes of records. Format. Electronic records transfer.
The Archive Management Team We are based in a purpose-built records facility at Hanslope Park near Milton Keynes (which was built in the early 1990s) and unlike some other government departments we haven't contracted out our records storage, we have our own in-house facility. So if we need to carry out a search of our records we don't have to order anything from a remote store, all our records are immediately available to us.
11
Most of our staff are involved with the transfer of records to The National Archives, although we have a team dedicated to retrieving files for FCO needs and we also have a small team which answers historical FOI enquiries. We have approximately 50 staff, and around half of those are sensitivity reviewers who are responsible for reviewing records for any residual sensitivity before those records are released to The National Archives. The rest of our team are either records managers or archive support staff.
Number of files How many files do we hold in our archive? As an approximation we believe we have around 1 million files in the archive and those are overwhelmingly paper records. About three quarters of our paper records are standard departmental files, so they are files which originate from departments which are clearly recognisable as current FCO departments or their predecessors, like Middle East Department; around 250,000 of our paper files we call “Special Collections�, these are records which sit outside the FCO departmental file plan and some of these are quite old. We also manage some archive electronic records, the equivalent of around 40,000 files. Everything that counts as a legacy record, i.e. those over 30-years old, is currently held in compliance with the Public Records Act under a legal instrument granted by the Lord Chancellor. This legal instrument has been granted to allow the FCO one year to develop a prioritised plan for our legacy records which we will be presenting to the Advisory Council in November 2013. Post-meeting note: revised figures for departmental files and special collections are being published on the FCO archive records page on gov.uk.
The records transfer process We start by selecting our files for permanent preservation using a document called Operational Selection Policy 13, which is a selection policy for diplomatic records and which is available on The National Archives website. Broadly speaking, the material that is selected is foreign policy rather than FCO case files. That selection is then quality assured by TNA. We then sensitivity review the files and that is by far the most time-consuming part of the process because all of the files have to be read through; we catalogue those files so they are searchable on The National Archives catalogue; physically prepare the files and then uplift those files to TNA. Only around 1% of the files are redacted, that's to say the content is blanked out because of continuing sensitivity. For standard FCO departmental files the whole transfer process takes about 1 year from the start of the transfer process in the FCO to public release at TNA. 12
Prioritising files for release We have a system of prioritising files for release so that we review and prepare the high priority files first. Our aim wherever possible is to make sure that the high priority files are available at TNA’s main annual opening. The “A+” files are the high priority FCO geographic department files which we know are very popular at The National Archives, such as Middle East Department; the other priority A classes are the remaining geographic departments like such as West Africa Department or Mexico and Caribbean; the priority B classes are the FCO's trans-border and thematic departments like Energy Department; and the C classes are the corporate support functions like finance and also consular case files. Our current thinking is that we should continue to use this system of prioritisation because it reflects the demand for FCO files at TNA. The prioritisation of our transfer also is also related to the rate at which we destroy our records because there is more ephemeral material amongst the C classes so we destroy a higher proportion of these lower priority classes. Whereas for, say, Middle East department, we’d probably be preserving about 80% of the material we create.
The FCO records transfer programme Information about the FCO’s records transfer programmes is available at gov.uk. The simplest way to access this page is to enter “FCO archive records” into the gov.uk search engine and then select the “detailed guidance” tab. Our current archive management and records transfer projects are:
Colonial administration files. Transfer of 1981 and 1982 files. Special Collections. Transition to 20-year release.
Colonial administration files These are files returned by the UK’s former colonies to the UK at independence e.g. Kenya, Malaya and Cyprus. A Colonial Office telegram of 3rd May 1961 required the return of files deemed sensitive to the UK. On 5th May 2011 the Foreign Secretary stated in a written ministerial statement: “I believe that it is the right thing to do for the information in these files now to be properly examined and recorded and made available to the public through the National Archives. This will be taken forward rapidly.” 13
The migrated archive files are being released in 8 tranches with c.18,000 files being released by end Nov 2013. A timetable and FAQs are available on gov.uk. The last release was on 26th April 2013 (tranche 5) and consisted of files from the following territories: Ceylon, Kenya, Malta, Mauritius, Nigeria, New Hebrides, Northern Rhodesia, Nyasaland, Palestine, Sierra Leone and Singapore. The next release, tranche 6, containing further Cyprus files date will be announced on gov.uk.
Backlog of 1981 and 1982 files The colonial administration files project involves the review and preparation of a substantial number of records so it has affected annual transfer work. It means that the FCO does have a backlog of around 20,000 1981 and 1982 files awaiting transfer. We have agreed with the Lord Chancellor's Advisory Council a timetable for the preparation of these files and again if you go to the relevant pages on the gov.uk website you can the order in which we will be preparing those files for release. We are now gradually transferring resources away from the colonial files project back onto the review and release of our backlog of 1981 and 1982 files and we are very focussed on trying to clear that backlog.
Special collections We are currently carrying out an inventory of all of our special collections (file series which sit outside the normal departmental file series). An example would be material we hold about the Allied Control Commission in Germany after the 2nd World War. We have put a high-level inventory of our holdings onto the gov.uk website and we are currently putting together a much more detailed inventory of all of our legacy files. We will be making recommendations to the Lord Chancellor’s Advisory Council for the review and release of our special collections towards the end of this year. One of the challenges for the FCO is that we prioritise the review and release of this older material as against the more recent material which is due for release in line with The National Archives’ plan for Government Departments to transition to 20-year release.
The transition to 20-year release The National Archives’ transitional period plan requires us to transfer 2 years in one year as shown in this timetable.
14
Year of transfer
Record year 1
Record year 2
2013
1983
1984
2014
1985
1986
2015
1987
1988
2016
1989
1990
2017
1991
1992 (start of electronic record transfer)
2018
1993
1994
2019
1995
1996
2020
1997
1998
2021
1999
2000
2022
2001
2002
Files from 1992 onwards will be both paper and electronic records. The significance of that particular year is that the FCO was a very early adopter of electronic records. The FCO introduced a system called ARAMIS in 1992 whereby if the user created correspondence which was being sent outside the FCO, then the ARAMIS system forced you to register that document as a record. So we will begin working on the transfer of digital records in 2016. How many files does the transition to 20-year release require us to transfer? This table shows us that the numbers are very large indeed. The greatest number of files over this period that we expect to transfer (including both paper and electronic files) is in 2018 for 1993-1994 files when we expect to select around 41,000. Over the whole transitional period we think we will need to transfer around 300,000 files and to review around 0.5 million files (or 50 million documents)
15
Record year
Expected number of files transferring to the TNA
1983-4
27868
1985-6
31268
1987-8
37424
1989-90
39788
1991-2
40219
1993-4
41113
1995-6
39457
1997-8
35785
1999-2000
28781
2001-2
21653
This graph shows the volumes we are need to transfer and the green line at the bottom shows the electronic records. This highlights really that the electronic records represent a very small proportion of the total over this period. The sharp drop you see at the end of this graph probably does not indicate the final picture for these later records. There are likely to be more of these recent records which still need to come into the archive so our total count of all the records which need to transfer over the transitional period probably isn’t complete yet.
45000 40000 35000 30000 25000
Paper
20000
Electronic Total
15000 10000 5000 0 83/84 85/86 87/88 89/90 91/92 93/94 95/96 97/98 99/00 01/02
16
What is the impact of the 20-year rule on the FCO’s archive management? Most obviously, the transition to 20 year release requires an uplift in resources. The overall cost to the FCO is about £6.5m and most of that cost will be on staffing. Already we have doubled the number of sensitivity reviewers we have and made a smaller uplift in resources amongst records managers as well. Another significant change for those of us in the records management community is that the work is much more project–based than it used to be and the FCO has been working very closely with TNA on a project management methodology for records transfer. Finally the most important point is that the proportion of files we select for permanent preservation will not change simply because of the larger volume of records. We will use the same criteria for selection which we have used in the past and which I have already outlined.
Electronic file transfer I mentioned earlier that the FCO implemented an electronic records management system in 1992 called ARAMIS and in 1998 the FCO built another system called MINERVA to ensure the records from ARAMIS were preserved when ARAMIS was decommissioned. So there are around 4 million records on the ARAMIS system, equivalent to 40,000 files. We are in the process of migrating these records to current systems. One of the particular challenges of the MINERVA data set is that it originates from a “hybrid” system meaning the data set contains full text electronic records and references to paper documents. At present, we don’t know how much duplication there is between the MINERVA data set and our paper holdings because users may have printed out documents and put them on hard copy files. Then there is the additional problem of version control, whether the electronic documents are really the same as the paper documents or are there significant differences. So in this respect managing transfer from a data set such as the MINERVA one is going to be more difficult than if we were dealing with purely electronic records. The next issue for us is the extent to which we can use technology to help us review files for transfer, for instance one possibility is that we could we use search technology to help us adopt a more thematic approach to the selection of files rather than selection which is based around the structure of our organisation. And finally a more relevant point for historians is what will it be like to carry out historical research using electronic records created by government rather than paper records? One of the things which strikes me about paper records, particularly the older ones like the colonial administration files, is the very high standard of registry work and the fact that a paper file is 17
very much a finished product. The narrative has to some extent already been created and it is quite easy to follow the thread of this narrative from the back of the file (the oldest papers) to the front (the most recent papers). I think that the files which the FCO will release over the transitional period like a halfway house between paper records and the modern electronic record, because the records are organised into a traditional file structure and the style of communication is still quite formal. With e-mail and other more recent forms of electronic communication the same level of structure is not present, and in order to make the most of the records it researchers will need good search skills whereas using a paper file is much more of a browsing activity where the structure has already been provided for the researcher.
18
Special Collections Martin Tucker, Head of Archives We use the term “Special Collections” for those files which site outside the FCO corporate file plan. That’s to say, these files are not organised into the FCO’s standard filing structure. These are not standard files produced by FCO Departments or by FCO posts, they tend to be more specialist in nature and come in a wider range of formats Why does the FCO still have such large of holdings of legacy records? The principal reason is that the FCO has been very concerned in the past to preserve records of value, but has not always had the resources to review and release its records in line with the timescales required by the legislation. Generally the FCO in the past has prioritised the review and release of files at the 30-year mark over the review of legacy records. Over the last two years this has changed as we have been reviewing and releasing colonial administration files but the result has been a backlog of annual transfer files. So with limited resources it can be quite difficult to achieve the right balance between annual transfer and the release of legacy records.
What’s in the special collections? It is not easy to summarise the contents of the special collections and until recently current records managers at Hanslope didn’t have a very good understanding of the nature and extent of these holdings. The first step we have taken towards better management of these legacy records is to commission a specialist information management company to compile a detailed inventory of the special collections. That inventory is currently under way and we expect it to be complete in about a month’s time. We will in due course publish a more detailed inventory on the gov.uk website. There are some substantial record series amongst the special collections. Here are a few examples:
Records of the Allied Control Commission, i.e. records relating to the Allied Occupation Zones in Germany after the end of World War II in Europe.
Records of the Foreign Compensation Commission which relate to British claims for compensation under British and international law for losses suffered overseas.
Legacy records of the FCO’s predecessors including the Foreign Office and the Colonial Office.
19
There are also many smaller specialist collections and quite a wide variety of formats including conventional paper files, microfilm, index cards, computer disks and tapes and audio and video tapes. Currently all of the special collection records are legally retained under a Lord Chancellor’s Instrument. This legal instrument has been granted to the FCO in order to provide some time for us to we carry out an inventory of our holdings and put together a prioritised plan for the release of the special collections. Obviously the review of release of special collection records is going to depend on resources and it’s too early to say right now what resources we will have to review these legacy records. We need to understand the value and importance of our legacy records in order to produce our prioritised plan. We would very much appreciate your input today into this process. We’ve put two copies of the inventory on flip charts and we’ve going to provide you with each with ten stickers to put up on the inventory over lunch so you can flag up records of particular interest to you. Please bear in mind that we’re providing a snapshot of the inventory to date and you’ll see that it is a work in progress. Please put your sticker in the middle column of the inventory. In addition, if there are records if particular interest please do talk to us during the break or flag up your interest in the Q&A session this afternoon.
20
Records in the Future Carryl Allardice, Head of Knowledge and Information Management Team
In looking at records in the future I plan to:
touch on the origins of FCO’s record keeping, paint a picture of what current records management looks like in the FCO, look at what we are working towards in terms of future records management and examine the challenges historians and researchers may face when they access FCO’s electronic records at The National Archives in the future.
Why we value record keeping in the Foreign Office The Foreign Office record keeping goes back to 1800 when the decision was taken to poach Richard Ancell from the State Papers Office to be its first Librarian and Keeper of Printed Papers. He started work on 1 January 1801. The State Papers Office was primarily responsible for registering and taking care of government records. This was the time of the enlightenment, the age of reason, when governments throughout Europe sought to subject administration to rational principles. So the establishment in 1782 of a separate government department, the Foreign Office, to handle foreign affairs was an obvious example of how such principles were put into practice. Another was the decision of Lord Grenville, who was Foreign Secretary from 1791-1801, “to create within the Foreign Office a department solely responsible for organising and retaining its records”1. It was a move of remarkable bureaucratic innovation, as the Foreign Office decided it would keep its own records instead of sending them to the State Papers Office, and it was Richard Ancell’s responsibility to organise these papers. Along with the King’s messengers, who fell within the ambit of his control, Ancell ensured that Foreign Office information was not only well organised, but it was also distributed globally. As Keith Hamilton one of our Historians has said: “Ancell’s appointment marked the beginning of a knowledge management revolution”2 in the Foreign Office. In order to facilitate access the Foreign Office’s papers, Ancell compiled a two volume compendium of diplomatic records entitled A Collection of Printed Treaties, Conventions and other State Papers between Great Britain and Foreign Powers. The volumes organised the documents in order of Countries and Dates, with a Table of Contents arranged in geographical and chronological order, were completed in 1802. This two volume work which we still have in 1
2
Keith Hamilton, “Richard Ancell, 1755-1844”. Talk given in the FCO, October 2005. Ibid.
21
the Office today, demonstrates that being able to search and retrieve information in the early 1800’s was as important as it is for Foreign Office staff today to find what the information they need.
Current FCO records management But the world has changed since Richard Ancell’s day. The challenge for FCO staff today is that they do not work on paper or even in one place on our IT systems. Staff have information stored in:
Outlook—with Inboxes of 100 MB storage space, personal drives, shared drives where team, departments and Directorates share their information in folders, team or collaboration sites (using Microsoft SharePoint 2007), iRecords—the FCO’s records management system (iRecords) and databases designed for specific purposes such as parliamentary questions and consular cases.
On top of this, staff are increasingly using pan-government collaboration tools where they store information (e.g. Huddle and Collaborate, HMRC’s eRooms, etc). This means that information is scattered across a plethora of information storage areas. The need to keep information in context is one of the primary concerns of FCO staff. Many staff think of the paper file as the golden age of record-keeping, because they were able to call for a paper file and read the contents in date order and have a complete picture of the subject matter when they had finished reading the file. Currently we cannot do that in the FCO’s digital world. But we are not alone. This is a common problem across Whitehall. An unintentional by product of the move to Electronic Document and Records Management System (EDRMS) systems in the early 2000s was that registering documents became much more onerous for staff (our own Registry system which was implemented in 2000 required 27 clicks to register a single document), so in many ways the implementation of these EDRMS contributed to an unfortunate decline in the traditional record-keeping culture amongst civil servants across Government. Other things have changed too. In the old days there were registry clerks, paper keepers, or filing clerks to uphold standards and to maintain order in the paper files. Today there are very few staff doing that kind of work in Government, because the investment in EDRMS was intended to deliver staff savings and reduce overall Government expenditure. The 22
current premise is that everyone who creates information is also responsible for registering their own work—doing their own record-keeping. But that is obviously much more difficult to monitor. Over and above this is the fact that digital records must exist within a container or a folder (as if they were a physical item) and most electronic records are still allocated a location within a hierarchical classification scheme that organises content as if physically arranging it—and this is at the heart of the problem. Civil servants are also being asked to add metadata to records to enable multiple search routes—in much the same way as registry clerks used index cards for paper records for nearly two centuries. Indeed the way we define electronic retention and destruction periods is also according to when the file was opened or closed, or when the digital record was created, which is a way of working which would have been entirely familiar to Foreign Office registry clerks as far back as 1801. In order to address what was happening to our own record keeping, we implemented a new system in 2008 called iRecords. We tried to address the deficiencies of our previous registry system (with its 27 clicks to register a document), by developing iRecords which is an electronic records system which only needs 2 clicks to register a document. Even though iRecords is a significant improvement on its predecessor, and in fact is probably state-ofthe-art compared to some of the records systems in use elsewhere in Whitehall, it still perpetuates the way we worked with paper and paper files. Steve Bailey, who writes voraciously on the subject of records management, said in 2009 “we try to manage electronic records as we have always managed paper records and this is why we are failing . . . The real challenge is not the fact that records are now electronic—it is the sheer volume with which they are now produced. And that requires a different approach to their management.” Bailey suggested that we need to start moving—towards “automated records management”.3 Well we did! In 2011 we successfully implemented the automated record keeping of all our diplomatic telegrams into our iRecords system. The benefit of our DipTels Project was that we have made sure we are automatically capturing all our key knowledge assets—our diplomatic reporting—our crown jewels.
Future records management Following the success of our DipTels Project we have agreed that automated record-keeping is the future for records management in the FCO. Last year the FCO set up a Knowledge Excellence Programme which is in the process of delivering a single information and records management environment, based upon Microsoft SharePoint 2013. This will, with the 3
Steve Bailey (2009) ‘Forget electronic records management, it’s automated records management we desperately need’ in Records Management Journal v.19 no.2: 91 –97.
23
addition of some software, allow us to automate our record keeping processes when we will be able to replace shared drives with team sites, so we can “hoover up” records from the team sites into our records store automatically—when a predetermined threshold is reached. It will also allow staff to drag and drop emails directly from Outlook into records folders. We are working towards implementing the new records system in 2014/15. We anticipate that our future record-keeping approach will lead to:
Improved records management practices. Improved user experience in terms of searching for information which is both relevant and accurate. A reduced risk of key information loss. A more complete corporate record, which we can interrogate both for records and FOI purposes, and which will be available in due course for permanent preservation at TNA.
What are the potential challenges for you accessing FCO’s electronic records in the future?
Duplication: we hope to start de-duplicating records when we move to the new records system in 2014, as we estimate at least 60% of the electronic records we currently hold are duplicates. We need therefore to do some mass de-duplication, so that when you look at records at The National Archives in future you will not be faced with multiple copies of the same document. Naming documents in the digital world is also important: the title of a document called ‘Visit’ doesn’t tell you very much, so a big priority for us is to get staff to name their emails and documents with meaningful titles both for the benefit of their own colleagues today, but also for historians and researchers in the future. Fragmentation: there is the fact that records will not be organised in neat folders which you can open and browse—the contents may be scattered over a number of folders. I came to a view about two years ago that in order to support the sensitivity review process and the move from 30 to 20 year transfer of records, documents should not be moved from their original information management environment into a separate records system, and that we needed a single information management environment to support digital ways of working. So at present there is a risk that the records from the last 12 years are going to be more fragmented than we would like when it comes to transferring them to The National Archives. Search—We are going to try to help not only our own staff but also historians and researchers in the future, by adding metadata to the records we have kept. For example we currently have more than a million Diplomatic Telegrams with no metadata other than title, author and DipTel number. There are also challenges with
24
emails and although email-threading software exists we do not currently have it in the FCO. Digital skills: At the heart of the challenge I would imagine the Historians and Researchers in the future will need excellent digital search skills to be able to do the kind of data mining required to extract information in a coherent way, and to find the nuggets or gems which have been so much more easily presented in the paper world.
I hope I have convinced you that record keeping for the FCO is important, and that we are doing our best to make it as accessible as possible for historians and researchers in future. As the Foreign Secretary, William Hague has said in a message to all FCO staff: “It is incumbent on every member of the Foreign Office to take individual responsibility for properly recording and registering the work we do and the decisions we take . . . we have an obligation to future generations…... Good record keeping allows us to account to the British public and parliament, deliver on foreign policy and learn from the past while providing insights for posterity. I ask everyone to play their part.” William Hague, Foreign Secretary, 15 June 2011.
25
Summary of discussion
This is a summary of the main points raised during the question and answer session, and the points made in response. Some additional notes and references are included. What is the definition of ‘sensitivity’ when it comes to reviewing files?
‘Sensitivity’ is a general term used to reflect material which needs to be protected and is covered by an exemption in the Freedom of Information Act, e.g. Section 27 of the FOI Act exempts from disclosure information which could damage international relations. Material may be withheld from release under legal exemptions. Details of the different types of closure and retention are described in The National Archives publication Access to public records. If a record is described as “closed” this means the material is exempted from release under a Freedom of Information Act exemption. This material will be held in the closed room at The National Archives. The FCO, like all government departments, must apply to the Lord Chancellor’s Advisory Council on National Records and Archives (LCAC) to close any material. All closed material must be re-reviewed after the specified period of closure and such material will either be released or an application must be made to the LCAC for a further period of closure. Material may also be ‘retained’ by the FCO under section 3.4 of the Public Records Act. In this case, the material will be physically held by the FCO for a specified period in our retained archive. There are different categories of retention, all described in Access to public records. For example, the FCO may hold records under “administrative retention” (with the Lord Chancellor’s approval for the specific records concerned) because we have a backlog of files awaiting review. An example is the colonial administration files which are held under administrative retention in order to provide the FCO with time to review and release these records by the end of 2013. In some instances, the Lord Chancellor has given approval for the blanket retention of certain types of record of a similar character. This includes a legal instrument known as “the Lord Chancellor’s Intelligence and Security Instrument” which relates to records which need to be retained for the purposes of national security. The FCO may retain records under the Lord Chancellor’s Intelligence and Security Instrument on the understanding that such records are re-reviewed for release after 10 years Chaired by the Master of the Rolls, The Lord Chancellor’s Advisory Council is an independent body which meets quarterly. It may, and frequently does, challenge requests from Government departments to withhold material. FCO sensitivity reviewers are all former senior diplomats with extensive experience in dealing with international affairs. FCO records which have been closed or retained may be requested under the Freedom of Information Act. For material held closed at TNA, an FOI request should be made to 26
TNA itself (details are on their website). For retained records, FOI requests should be made to the FCO (details are on the FCO site on gov.uk) Why retain a document when just one word might be sensitive?
The Waldegrave Initiative in 1993 introduced the major change of redacting rather than retaining a document or closing a whole file. Redaction is now done on a word by word basis. However, when the context would make a missing name too obvious, surrounding text may need to be redacted.
Specific records—No files in TNA relating to black Political Activists in London from the 1920s
It would be difficult and time-consuming for us to try to establish whether any documentation still exists on “keep and destroy” decisions for this period and on this subject. In terms of current practice, a record of all keep and destroy decisions is kept electronically while in the past keep and destroy decisions were recorded by “stamping off” registers to indicate that files were destroyed. We are sometimes able to work back through past registers to establish what has been destroyed, but unless we know the file reference and/or the subject matter is very specific this can be difficult especially for older material.
How can historians help the FCO better prioritise material for release?
IMD will better prioritise the release of files by understanding the audience’s needs. Currently for the annual release of files, priority is currently given to those records in file series which TNA has identified as being most frequently requested.
What form will electronic archives take on transfer?
The FCO will transfer material in digital format and it will be accessed in that format at TNA.
Will migrated archive material be returned to its countries of origin?
No—they have been classed as UK Public Records. Digitising or copying the migrated archives for individual countries would be a huge resource issue for the FCO.
27
What checks are there on FCO staff to ensure that they are registering material on iRecords?
Individual civil servants have a statutory obligation to keep records. What they should keep is set down in guidance issued by the Information Management Department (IMD) who also work to raise awareness of the importance of good records management amongst staff in the FCO. IMD conducts Information Management Health Checks for FCO departments and advises on areas for improvement. An improved and more automated electronic record-keeping system is expected to be available in the FCO in 2014/15.
How do Historians know if a file has been retained?
The National Archives catalogue indicates whether a file is closed or retained. Determining whether a particular document within a file is retained requires consulting the file.
Is there a less formal approach to accessing unreleased material than FOI?
No—for the good reason that the formal FOI process provides a level playing field for all and requires a response from the FCO within 20 working days. When there is an FOI request the file is reviewed and a fresh decision is made—even if the file was only reviewed recently.
What FOI assistance can historians expect?
The FOI cost limit is £600 which roughly equates to 3.5 days in time. The cost per request varies according to the time needed to find the information, e.g. if the information cuts across many files it will take longer. Hence too wide an FOI request may be rejected as too costly. Increasingly enquirers give specific file numbers, if they have them, which saves time. The FCO FOI team help enquirers to narrow down requests so that they fall within the cost limit.
Why did I receive a negative response to my FOI request when it now appears the FCO does hold material on the subject in the special collections?
The special collections were not routinely searched as part of an FOI request. This was due to the fact that no detailed inventory of the material existed. The High Level Inventory published in late 2012 was the first step to get a grip of this situation. The more detailed inventory being prepared will help archive staff search more effectively 28
for relevant records. The most in-depth understanding that the archive staff gain of the content of files will come when the files are catalogued. Clarity will be needed in the future on how electronic records are structured.
The file plan of digitally registered information provides context and includes all registered documents for a given area. File plans will accompany transferred records. But undeniably digitised records lose the feel of paper and omit additional handwritten comments or underlining unless a scanned version is registered into the electronic records system.
What is the position with embassy files?
Files from overseas posts are classed as departmental records and are transferred back to the UK in line with the FCO’s retention and destruction policies. More delegation has been granted to posts to weed files prior to their transfer back to the UK, e.g. to de-duplicate copies of documents already sent to London that will already exist in FCO departmental files. There is currently a back-log of post files awaiting review and release.
29
Further information Freedom of Information To make a Freedom of Information request either: Email: foi-dpa.imd@fco.gov.uk Or write to: FOI and DPA Team Information Management Department Foreign and Commonwealth Office Room K4.10 – K4.13 King Charles Street London SW1A 2AH For FOI/DPA enquiries please ring: 020 7008 0123.
Archive management Further information on FCO Archive Records can be found at: https://www.gov.uk/archiverecords.
FCO Historians FCO Historians have recently digitised their publications produced over the last 25 years and made them available online at www.issuu.com/fcohistorians. These documents include the History Notes series examining subjects from Foreign Office history; Occasional Papers reproducing lectures given in the FCO or research papers written by the FCO Historians; Witness Seminars recording oral testimony given by FCO alumni; Documents from the Archives reproducing original documents on specific topics from the Foreign Office archives; and other transcripts of seminars and events. FCO Historians also publish collections of key documents on the formulation of British foreign policy since 1945 in the series: Documents on British Policy Overseas.
30