Speed inserts in database with a table with +47 million data per each year, containing more than 10 years?
04:04 16 Nov 2011

I'm moving my database to MySQL but there is one table that is pretty big (more than 47 million rows for each year) and it takes years to insert all the data.

I already tried everything I found in the forum, partitioning, index, fulltext... and is true it works better but its far away from efficient.

For giving more background: The data base is about hydrological data recorded during the past ten years. In my data base there are three tables: one keeping the objects to be saved, another to save the sensors of each object and the last one and the one which is causing the big problem is where I keep hourly all the information collected for each sensor (this table will have 47 million data for each year int the database)

Is there any way I can performed the inserts in less than 10 hours?

What I'm trying to do (because it's simpler) is to transfer the data form our custom database (which is filesystem-based, with custom binary files) to a standard SQL database. The real source of the info comes from our propietary software and it is the data alocated in the memory: the program has a large set of objects in which we store vectors of date/value which can be huge depending on the processing (because we can choose any startDate and endDate).We want to store hourly data.This would be the ultimate goal of the SQL database.

mysql