What is Corpus Creation?

Corpus creation is the process of building a dataset. For a digital humanities project, this often entails either finding a collection of texts or images online or digitizing physical holdings.

Tools for cleaning and processing texts

You have a digitized corpus, now what? The answer to this depends on what your data looks like and what your research questions are. For a few possibilities:

• For most typescript material, you could run it through optical character recognition (OCR) software. OCRing texts turns them from images into searchable, machine-actionable texts. One of the leading tools in this area is ABBYY FineReader. While not free software, ABBYY is available on two of the computers in the Digital Humanities Lab. Depending on the quality of the scan and the type of document you’re working with, some hand cleaning post OCRing may be necessary (for example, ‘l’ may be misidentified as ‘1’).
• For handwritten material, humans are still the best transcribers. Depending on the size and content of your corpus, crowdsourcing could be an option.
• For messy tabular data, you could use OpenRefine, a free tool for cleaning and transforming textual data.

How can the DHLab help?

Digital Humanities Lab staff can advise on strategies for building and cleaning your corpus during our weekly Office Hours. We also regularly offer workshops that are relevant to corpus creation. Visit our Workshops page to learn about what’s coming up and our GitHub page for tutorials from past sessions. For information on the use of our scanners for corpus creation, as well as information on databases that are already available for text and data mining, please visit our Data Resources page.