When I ran each R code chunk in the

title: "Step 2: Data Import and Schema Sanity Check"
author: "Dr. James Daniel"
date: today
format:
  html:
    toc: true
    code-fold: true
execute:
  echo: true
  warning: false
  message: true
---

# Objective
This document performs the **data import and schema inspection** stage of the project *Why Nigerians Left Nigeria – Sentiment Analysis of Public Tweets*.  
It ensures the raw dataset is properly read, inspected, documented, and summarized before any cleaning or modeling.

---

## Step 1 – Import the Excel file

```{r}
# ---- Load required libraries ----
library(readxl)     # for reading Excel files
library(dplyr)      # for data manipulation
library(janitor)    # for cleaning column names
library(skimr)      # for quick data overview
library(writexl)    # for saving data dictionary
library(snakecase) # for checking POSIXt class
library(lubridate) # for checking Date class

# ---- Define file path (relative to project root) ----
excel_path <- "C:/Users/USER/Documents/why_nigerians_left_sentiment/data/raw/Why did you decide to leave Nigeria.csv"

# ---- Import the first sheet ----
tweets_raw <- read.csv(excel_path)

# ---- Peek at structure ----
glimpse(tweets_raw)

Reads the Excel file into R as a tibble and shows all column names, types, and sample values.

Step 2 – Clean column names (optional but smart)

tweets <- tweets_raw %>%
  janitor::clean_names()  # converts to snake_case (tweet_id, created_at, like_count, etc.)

head(tweets)

Step 3 – Create a simple data dictionary

# ---- 1. Basic structure summary ----
skim(tweets)

# ---- 2. Create metadata table ----
data_dict <- data.frame(
  column_name = names(tweets),
  # Fix: Paste multiple class names together (e.g., "POSIXct, POSIXt")
  data_type   = sapply(tweets, function(x) paste(class(x), collapse = ", ")),
  # Check for NA or empty strings
  missing_pct = sapply(tweets, function(x) {
    if(is.POSIXt(x) || is.Date(x)) {
      mean(is.na(x)) * 100
    } else {
      mean(is.na(x) | x == "") * 100
    }
  })
) %>%
  arrange(desc(missing_pct))

# ---- 3. Save dictionary ----
dir.create("reports", showWarnings = FALSE)
write_xlsx(data_dict, "reports/data_dictionary.xlsx")

# Display first few rows
head(data_dict)

Creates reports/data_dictionary.xlsx listing column names, types, and missingness.

Step 4 – Quality checks (sanity summary)

# ---- Dataset size ----
nrow(tweets)
ncol(tweets)

# ---- Duplicates ----
unique_ids <- dplyr::n_distinct(tweets$tweet_id)
duplicates  <- sum(duplicated(tweets$tweet_id))

# ---- Missing text values ----
missing_text <- sum(is.na(tweets$text) | tweets$text == "")

# ---- Date range (if available) ----
if ("created_at" %in% names(tweets)) summary(tweets$created_at)

# ---- Compact summary table ----
import_summary <- tibble(
  total_rows       = nrow(tweets),
  unique_tweetID   = unique_ids,
  duplicate_rows   = duplicates,
  missing_text     = missing_text,
  missing_text_pct = round(mean(is.na(tweets$text) | tweets$text == "") * 100, 2)
)
import_summary

Outputs overall size, duplicate count, and missing text percentage.

Step 5 – Save a 100-row “preview” subset

# ---- Save small sample for fast testing ----
dir.create("data/clean", recursive = TRUE, showWarnings = FALSE)

preview <- tweets %>% slice_head(n = 100)
write.csv(preview, "data/clean/preview.csv", row.names = FALSE)

# Confirm file saved
file.exists("data/clean/preview.csv")

Creates data/clean/preview.csv for lightweight prototyping.

Step 6 – Export tidy import summary

# Save compact import summary for reporting
write.csv(import_summary, "reports/import_summary.csv", row.names = FALSE)

Produces reports/import_summary.csv — ready to cite in your methodology.

Expected Folder Outcomes

After successful execution:

data/
 ├── raw/
 │   └── Why did you decide to leave Nigeria.csv
 └── clean/
     └── preview.csv
reports/
 ├── data_dictionary.xlsx
 └── import_summary.csv
scripts/
 └── import_and_schema_check.R

Next Step

Proceed to Step 3: Cleaning & De-duplication, where you will:

• Remove retweets and empty texts,
• Normalize tokens,
• Prepare the corpus for sentiment analysis.

file, they all ran successfully, but when I ran the knite the same quarto file I got the following error messege:  ==> quarto preview 01_data_import_schema_check.qmd --to html --no-watch-inputs --no-browse



processing file: 01_data_import_schema_check.qmd
  |........                                            |  15% [unnamed-chunk-1]

Error:
! package or namespace load failed for 'janitor' in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]):
 there is no package called 'snakecase'
Backtrace:
    ▆
 1. └─base::library(janitor)
 2.   └─base::tryCatch(...)
 3.     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 4.       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 5.         └─value[[3L]](cond)

Quitting from 01_data_import_schema_check.qmd:23-41 [unnamed-chunk-1]
                                                                                                            
Execution halted