I am trying to process text for quantitative text analysis. I need to read in pdfs of transcripts from WHO plenary meetings and process the text into a speaker turn dataframe, identifying the speaker and everything that they say afterward. I am attempting this in R, since this is the software with which I am most familiar. My current code is the following:
pacman::p_load(tidyverse,
stringr,
fs,
tibble,
pdftools,
stringr,
dplyr,
purrr)
speaker_list_unique <- structure(list(country = c("Afghanistan", "Afghanistan", "Albania",
"Albania", "Albania", "Albania", "Argentina", "Argentina", "Argentina",
"Australia", "Australia", "Australia", "Australia", "Australia",
"Australia", "Australia", "Austria", "Austria", "Austria", "Austria",
"Belgium", "Belgium", "Belgium", "Belgium", "Belgium"), speaker = c("Dr. G. FAROUK, Deputy Minister for Public Health (Chief Delegate)",
"Dr. A. ZAHIR, Director-General of the Kabul Municipal Hospitals",
"Mr. B. SHTYLLA, Minister Plenipotentiary, Ministry of Foreign Affairs (Chief Delegate)",
"Dr. S. KLosi, Ministry of Public Health", "Mr. V. NATHANAIL, Ministry of Foreign Affairs",
"Mr. F. KOTA, Assistant Chief, Department for International Organizations, Ministry of Foreign Affairs",
"Dr. A. ZWANCK, Professor of Hygiene, University of Buenos Aires (Chief Delegate)",
"Dr. G. GALVEZ BUNGE, Director-General, Department of Sanitary Legislation, Ministry of Public Health",
"Dr. A. A. Pozzo, Director of Technical Education and Scientific Research, Ministry of Public Health",
"Dr. G. M. REDSHAW, Chief Medical Officer, Australia House, London (Chief Delegate)",
"Mr. B. C. BALLARD, Counsellor, Australian Embassy, Paris", "Mr. W. G. A. LANDALE, Second Secretary, Australian Legation, The Hague",
"Dr. H. E. DOWNES, Assistant Director-General of Health (Chief Delegate)",
"Dr. D. A. DOWLING, Chief Medical Officer, Australia House, London",
"Mr. J. PLIMSOLL, Department of External Affairs", "Mr. J. R. ROWLAND, Department of External Affairs",
"Dr. F. REUTER, Professor, University of Vienna ; Chief, Bureau of Public Health, Ministry of Social Welfare (Chief Delegate)",
"Dr. F. PUNTIGAM, Counsellor, Ministry of Social Welfare", "Mr. K. STROBL, Counsellor, Ministry of Social Welfare",
"Dr. A. KHAUM, Director of Public Health (Chief Delegate)", "M. A. VERBIST, Ministre de la Santé publique et de la Famille (Chief Delegate)",
"M. L. A. D. GEERAERTS, Directeur de Chancellerie de première claise au Ministère des Affaires étrangères et du Commerce extérieur",
"Professor M. DE LAËT, Secrétaire général du Ministère de la Santé publique et de la Famille",
"Dr. A. N. DUREN, Conseiller medical au Ministère des Colonies",
"Baron C. VAN DER BRUGGEN, Attaché de Cabinet au Ministère de la Santé publique et de la Famille"
), speaker_condensed = c("FAROUK", "ZAHIR", "SHTYLLA", "KLOSI",
"NATHANAIL", "KOTA", "ZWANCK", "GÁLVEZ BUNGE", "POZZO", "REDSHAW",
"BALLARD", "LANDALE", "DOWNES", "DOWLING", "PLIMSOLL", "ROWLAND",
"REUTER", "PUNTIGAM", "STROBL", "KHAUM", "VERBIST", "GEERAERTS",
"DE LAËT", "DUREN", "VAN DER BRUGGEN"), organization = c(NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_, NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_, NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_, NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_, NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_)), row.names = c(NA,
-25L), class = c("tbl_df", "tbl", "data.frame"))
pdf_path <- "webscrape/who/plenary/manual/WHA_1948.pdf"
text_boundaries = tribble(~year,~start_page,~end_page,
1948,23,106,
1949,79,147,
1950,97,187)
# read target pages
start_page <- text_boundaries$start_page[1]
end_page <- min(text_boundaries$end_page[1], length(pdf_text(pdf_path)))
pdf_pages <- pdf_text(pdf_path)[start_page:end_page]
# two columns function
read_two_columns <- function(page_text) {
# Split lines
lines <- str_split(page_text, "\n")[[1]]
# split lines in half based on character width
max_width <- max(nchar(lines))
mid <- ceiling(max_width / 2)
left_lines <- str_sub(lines, 1, mid)
right_lines <- str_sub(lines, mid+1, nchar(lines))
# Collapse left first then right
paste(c(left_lines, right_lines), collapse = " ")
}
# Apply to all pages
full_text <- map_chr(pdf_pages, read_two_columns) %>%
paste(collapse = " ")
# Clean spacing
full_text <- str_squish(full_text)
# 4. locate speaker turns
speaker_patterns <- speaker_list_unique$speaker_condensed
# Prepare regex
pattern <- paste0("\\b(", paste(speaker_patterns, collapse="|"), ")\\b")
# Find all matches
matches <- str_locate_all(full_text, regex(pattern, ignore_case = TRUE))[[1]]
# build speaker turn df
speaker_turns <- map_dfr(seq_len(nrow(matches)), function(i) {
start_pos <- matches[i, "start"]
end_pos <- if(i < nrow(matches)) matches[i + 1, "start"] - 1 else nchar(full_text)
speaker_match <- str_sub(full_text, start_pos, matches[i, "end"])
tibble(
speaker_condensed = toupper(str_squish(speaker_match)),
text = str_squish(substr(full_text, start_pos, end_pos))
)
})
# join back
speaker_turns <- speaker_turns %>%
left_join(
speaker_list_unique %>% select(speaker_condensed, country, speaker),
by = "speaker_condensed"
)
speaker_turns
Where I use manually identified speakers names in the speaker_list_unique dataframe to search the text. My issue is that the pdf documents are in column form, but the OCR-ing is such that it reads texts across the columns. For example, as shown below, when I highlight text going down the left column, if goes across the divider and reads in sentences out of order, rather than doing the left column and then the right column. The same occurs when reading the pdf in r: even if I set it up to read in 2-column form, the reading runs across the page rather than getting the text in order.
My question is how to read the pdfs so I can create 1 comprehensive text string that runs across all the target pdf pages. My thinking would be to split each page down the divider so that the text from the right side of the page can't be read as it works through the left side and vice versa. However, there are hundreds of pages and pdfs from years 1948-2009, so I need a way to automate this. Is there a way in r to physically separate the pages and reassemble them in order so that they can be read and converted to text strings to assemble the speaker turn dataframe?