I’m deploying MongoDB using the Dockerfile below and experiencing critical corruption issues
I’m deploying MongoDB using the Dockerfile below and experiencing critical corruption issues.
Environment:
Inserting approximately 1 million documents per day
Server crashes every 3 days with segmentation fault
Error occurs on collections actively being queried
Running validation command shows corruption:
The Docker environment is as follows:
FROM rockylinux/rockylinux:8.10
RUN dnf update -y && \
dnf install -y wget tar gzip gcc gcc-c++ make cmake openssl-devel libcurl-devel compat-openssl10 && \
dnf clean all
RUN cd /tmp && \
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel8-8.0.14.tgz && \
tar -xzf mongodb-linux-x86_64-rhel8-8.0.14.tgz && \
mv mongodb-linux-x86_64-rhel88-8.0.14 /opt/mongodb && \
ln -s /opt/mongodb/bin/* /usr/local/bin/ && \
rm mongodb-linux-x86_64-rhel8-8.0.14.tgz
RUN cd /tmp && \
wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-rhel88-x86_64-100.14.0.tgz && \
tar -xzf mongodb-database-tools-rhel88-x86_64-100.14.0.tgz && \
mv mongodb-database-tools-rhel88-x86_64-100.14.0 /opt/mongodb-tools && \
ln -s /opt/mongodb-tools/bin/* /usr/local/bin/ && \
rm -rf mongodb-database-tools-rhel88-x86_64-100.14.0.tgz
RUN cd /tmp && \
wget https://downloads.mongodb.com/compass/mongosh-2.5.10-linux-x64.tgz && \
tar -xzf mongosh-2.5.10-linux-x64.tgz && \
mv mongosh-2.5.10-linux-x64 /opt/mongosh && \
ln -s /opt/mongosh/bin/* /usr/local/bin/ && \
rm mongosh-2.5.10-linux-x64.tgz
RUN useradd -r -s /bin/bash mongod
RUN mkdir -p /data/db && \
chown -R mongod:mongod /data/db && \
chmod 755 /data/db
RUN mkdir -p /etc/mongodb && \
echo "systemLog:" > /etc/mongodb/mongod.conf && \
echo " destination: file" >> /etc/mongodb/mongod.conf && \
echo " path: /data/db/mongod.log" >> /etc/mongodb/mongod.conf && \
echo " logAppend: false" >> /etc/mongodb/mongod.conf && \
echo "storage:" >> /etc/mongodb/mongod.conf && \
echo " dbPath: /data/db" >> /etc/mongodb/mongod.conf && \
echo " wiredTiger:" >> /etc/mongodb/mongod.conf && \
echo " engineConfig:" >> /etc/mongodb/mongod.conf && \
echo " cacheSizeGB: 2" >> /etc/mongodb/mongod.conf && \
echo "net:" >> /etc/mongodb/mongod.conf && \
echo " bindIp: 0.0.0.0" >> /etc/mongodb/mongod.conf && \
echo " port: 27017" >> /etc/mongodb/mongod.conf && \
echo "security:" >> /etc/mongodb/mongod.conf && \
echo " authorization: enabled" >> /etc/mongodb/mongod.conf && \
chown mongod:mongod /etc/mongodb/mongod.conf
RUN echo '#!/bin/bash' > /entrypoint.sh && \
echo 'if [ "$(id -u)" = "0" ]; then' >> /entrypoint.sh && \
echo ' chown -R mongod:mongod /data/db' >> /entrypoint.sh && \
echo ' chmod 755 /data/db' >> /entrypoint.sh && \
echo ' touch /data/db/mongod.log' >> /entrypoint.sh && \
echo ' chown mongod:mongod /data/db/mongod.log' >> /entrypoint.sh && \
echo ' chmod 644 /data/db/mongod.log' >> /entrypoint.sh && \
echo ' exec su -p mongod -c "mongod --config /etc/mongodb/mongod.conf"' >> /entrypoint.sh && \
echo 'else' >> /entrypoint.sh && \
echo ' exec mongod --config /etc/mongodb/mongod.conf' >> /entrypoint.sh && \
echo 'fi' >> /entrypoint.sh && \
chmod +x /entrypoint.sh
RUN echo '#!/bin/bash' > /init-mongo.sh && \
echo 'sleep 10' >> /init-mongo.sh && \
echo 'mongo admin --eval "db.createUser({user: \"~\", pwd: \"~\", roles: [\"root\"]})"' >> /init-mongo.sh && \
chmod +x /init-mongo.sh
EXPOSE 27017
CMD ["/entrypoint.sh"]
db.runCommand({ validate: "collection_~", full: true })
{
ns: 'db.collection_~',
uuid: UUID('d6d750af-68f0-4e67-8de4-55d13c2071a3'),
nIndexes: 2,
keysPerIndex: { _id_: 0, hash_1: 0 },
indexDetails: {
_id_: {
valid: true,
warnings: [
'Could not complete validation of table:index-1-12115536060148844241. This is a transient issue as the collection was actively in use by other operations.'
]
},
hash_1: {
valid: true,
warnings: [
'Could not complete validation of table:index-122-5622870854223694318. This is a transient issue as the collection was actively in use by other operations.'
]
}
},
valid: false,
repaired: false,
warnings: [
'Could not complete validation of table:index-1-12115536060148844241. This is a transient issue as the collection was actively in use by other operations.',
'Could not complete validation of table:index-122-5622870854223694318. This is a transient issue as the collection was actively in use by other operations.'
],
errors: [
`{"ts_sec":1769490757,"ts_usec":960889,"thread":"13:0x7ff850c78700","session_dhandle_name":"file:collection-0-12115536060148844241.wt","session_name":"WT_SESSION.verify","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_block_read_off:273:collection-0-12115536060148844241.wt: potential hardware corruption, read checksum error for 24576B block at offset 923348992: calculated block checksum of 0xf9e5746f doesn't match expected checksum of 0xde51de59"}`,
'{"ts_sec":1769490757,"ts_usec":961464,"thread":"13:0x7ff850c78700","session_dhandle_name":"file:collection-0-12115536060148844241.wt","session_name":"WT_SESSION.verify","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 923348992, 24576, 0xde51de59}: (chunk 1 of 24): 00 00 00 00 00 00 00 00 7e 47 30 00 00 00 00 00 3e c9 01 00 6c 00 00 00 07 05 00 01 00 60 00 00 00 00 00 00 01 00 00 00 11 e3 1f 0d 25 80 c7 ba 3a 08 00 00 07 5f 69 64 00 69 74 79 8f 04 f5 19 8a 5b 00 00 00 00 00 00 fe 91 07 80 bc 7c da 50 85 07 63 68 61 6e 6e 65 6c 47 72 6f 75 70 49 64 00 69 01 ce 21 a5 6e d0 e9 e1 d5 e7 0a 11 1c 01 17 88 11 48 a5 31 fc b2 96 32 5d ed fe 02 63 6f 6c 6c 65 63 74 54 79 70 65 00 05 00 00 00 53 46 54 50 00 09 63 09 16 44 41 74 00 19 d2 fa ee 9b 01 00 00 02 73 6f 75 72 63 65 05 28 00 0a 0d 28 10 5f 46 49 4c 45 11 1a 14 46 6f 72 6d 61 74 05 44 80 4a 53 4f 4e 00 02 66 69 6c 65 50 61 74 68 00 3a 00 00 00 2f 68 6e 6e 2f 32 30 32 36 2f 31 2f 32 34 05 0a 4c 30 31 32 34 31 36 34 39 33 35 5f 68 6e 6e 5f 6e 69 64 73 2d 01 22 01 18 3c 2d 31 36 31 35 2e 6a 73 6f 6e 00 09 65 76 65 6e 01 8a 08 6c fe d5 09 8a f0 5b 68 61 73 68 00 41 00 00 00 65 39 39 37 33 35 35 38 37 64 66 32 35 64 30 33 33 32 32 64 64 31 35 36 65 61 32 64 65 62 37 37 36 33 64 66 30 31 64 62 61 62 39 34 36 39 31 31 63 34 62 30 36 63 63 36 37 65 65 65 66 64 61 36 00 08 68 61 73 68 43 6f 6d 70 6c 65 74 65 64 00 01 03 73 05 e5 44 4d 65 74 61 00 16 00 00 00 07 5f 69 64 00 69 11 48 00 2d 2f 78 f6 00 03 72 61 77 44 61 74 61 00 b8 06 00 00 02 40 74 69 6d 65 73 74 61 6d 70 00 19 00 00 00 01 be 98 2d 30 31 2d 32 34 54 30 37 3a 30 39 3a 32 30 2e 33 36 34 5a 00 03 65 63 73 00 17 00 00 00 02 76 65 72 73 69 6f 6e 00 21 6b 24 38 2e 31 30 00 00 03 6b 63 73 05 6d 15 1c 14 04 00 00 00 31 2e 01 1b 29 74 28 00 1a 01 00 00 03 67 65 6f 00 b0 01 43 90 63 6f 75 6e 74 72 79 5f 6e 61 6d 65 00 09 00 00 00 50 6f 72 74 75 67 61 6c 00 02 63 6f 6e 74 69 6e 65 6e 74 5f 05 1d 24 07 00 00 00 45 75 72 6f 70 65 2e 1b 00 14 63 6f 64 65 00 03 01 1b 08 55 00 02 01 c1 14 7a 6f 6e 65 00 0e 01 11 05 2c 18 2f 4c 69 73 62 6f 6e 01 33 09 69 08 69 73 6f 19 35 20 50 54 00 03 6c 6f 63 61 74 01 c6 c0 1f 00 00 00 01 6c 6f 6e 00 84 0d 4f af 94 45 22 c0 01 6c 61 74 00 c3 64 aa 60 54 5a 43 40 00 00 03 61 73 00 3b 00 00 00 03 6f 72 67 61 6e 69 7a 61 05 36 00 1c 01 c6 05 a1 f0 4f 0d 00 00 00 5a 65 6e 6c 61 79 65 72 20 49 6e 63 00 00 10 6e 75 6d 62 65 72 00 63 55 00 00 00 10 70 6f 72 74 00 34 a6 00 00 02 69 70 00 0f 00 00 00 31 38 35 2e 31 38 30 2e 31 34 31 2e 33 37 00 00 03 63 6c 6f 75 64 00 0e 01 00 00 02 72 65 67 01 94 01 f5 44 eb b6 84 eb 8b b9 00 03 69 6e 73 74 61 6e 63 65 00 58 01 73 b0 69 64 00 25 00 00 00 39 61 66 62 32 65 31 61 2d 30 32 33 37 2d 34 65 62 34 2d 62 35 36 39 2d 34 37 36 32 39 38 61 62 66 30 65 35 00 02 05 a0 01 aa a8 73 65 72 76 65 72 2d 63 76 65 2d 73 70 72 69 6e 67 63 6f 72 65 2d 73 72 76 31 32 00 00 02 61 76 61 69 6c 61 62 69 6c 69 74 79 5f 25 49 21 be 34 54 4e 4e 00 03 70 72 6f 6a 65 63 74 00 5e 0d 7c 10 21 00 00 00 62 41 23 68 63 36 32 39 39 65 36 34 39 64 32 39 30 39 31 33 66 38 65 30 37 39 33 66 33 35 39 0d 78 d0 26 00 00 00 ec b5 9c ea b7 bc 20 ed 95 b4 ed 82 b9 20 ea b3 b5 ea b2 a9 20 ec 95 85 ec 9a a9 20 ec b7 a8 ec 95 bd ec a0 90 00 00 02 70 72 6f 76 69 64 65 72 00 01 79 1c 41 50 54 00 00 03 6f 62 09 b5 25 a0 29 6a 78 10 00 00 00 74 72 61 70 77 61 6c 6c "}',
~
'{"ts_sec":1769490757,"ts_usec":976277,"thread":"13:0x7ff850c78700","session_dhandle_name":"file:collection-0-12115536060148844241.wt","session_name":"WT_SESSION.verify","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 923348992, 24576, 0xde51de59}: (chunk 24 of 24): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "}',
'{"ts_sec":1769490758,"ts_usec":9643,"thread":"13:0x7ff850c78700","session_dhandle_name":"file:collection-0-12115536060148844241.wt","session_name":"WT_SESSION.verify","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__verify_ckptfrag_chk:520:checkpoint ranges never verified: 141"}',
'{"ts_sec":1769490758,"ts_usec":12813,"thread":"13:0x7ff850c78700","session_dhandle_name":"file:collection-0-12115536060148844241.wt","session_name":"WT_SESSION.verify","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__verify_filefrag_chk:434:file ranges never verified: 140"}',
'verify() returned WT_ERROR: non-specific WiredTiger error. This indicates structural damage. Not examining individual documents.'
],
extraIndexEntries: [],
missingIndexEntries: [],
corruptRecords: [],
advice: 'A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.',
ok: 1
}
Key errors:
potential hardware corruption, read checksum error for 24576B block at offset 923348992: calculated block checksum of 0xf9e5746f doesn't match expected checksum of 0xde51de59checkpoint ranges never verified: 141file ranges never verified: 140verify() returned WT_ERROR: non-specific WiredTiger error
Root Cause Hypothesis:
The corruption appears related to my Spring Boot application frequently restarting without graceful shutdown. The application connects to MongoDB for testing, and ungraceful shutdowns may be causing WiredTiger data file corruption.
What I’ve Tested:
I manually forced Spring Boot termination during testing but could NOT reproduce the corruption, which makes me question my configuration.
Questions:
Are there configuration issues in my Dockerfile that could contribute to this corruption?
Should I adjust WiredTiger cache settings, journaling intervals, or fsync parameters for high-write workloads?
Are there Docker volume mount or storage driver settings that could cause checksum errors?
What MongoDB configuration changes would make it more resilient to ungraceful client disconnections?
An interesting point is that after rebooting the server, MongoDB can start again without issues.
For now, our workaround is to reboot the server, delete the affected collection, and then recreate it to resolve the problem."