importing 4 million rows of data from a python script gathering data from server shares
09:44 01 Jun 2026

I have a few shares that need to be audited which will generate 10 or so fields of data per file. I need to have it walk through all the sub directories. expected 3-4 million rows of data after it walks through the shares. I was needing to know if it was best to batch an array of 20 rows at a time or if it would be best to have each row get entered as it was generated. the script has to generate a md5 for each file as part of the data it gathers so it may not be instantly finished with data as it hits each file. if there's a good document that covers this i would be grateful.

python-3.x database filesystems