【MySQL】批量插入大量數據方法

原創

LU_ZHAO

2020-03-30 17:06

參考了網上的方法。

有說換引擎爲myisam，那myisam比innodb快的原因是什麼呢。

再複習一下區別：

事務處理：

MyISAM是非事務安全型的，而InnoDB是事務安全型的（支持事務處理等高級處理）
鎖機制不同：

MyISAM是表級鎖，而InnoDB是行級鎖
select ,update ,insert ,delete 操作：
1. MyISAM：如果執行大量的SELECT，MyISAM是更好的選擇
2. InnoDB：如果你的數據執行大量的INSERT或UPDATE，出於性能方面的考慮，應該使用InnoDB表
查詢表的行數不同：
1. MyISAM：select count() from table,MyISAM只要簡單的讀出保存好的行數，注意的是，當count()語句包含 where條件時，兩種表的操作是一樣的
2. InnoDB ： InnoDB 中不保存表的具體行數，也就是說，執行select count(*) from table時，InnoDB要掃描一遍整個表來計算有多少行
外鍵支持：

myisam表不支持外鍵，而InnoDB支持

爲什麼MyISAM會比Innodb 的查詢速度快？ INNODB在做SELECT的時候，要維護的東西比MYISAM引擎多很多：

數據塊，INNODB要緩存，MYISAM只緩存索引塊，這中間還有換進換出的減少；
innodb尋址要映射到塊，再到行，MYISAM 記錄的直接是文件的OFFSET，定位比INNODB要快
INNODB還需要維護MVCC一致；雖然你的場景沒有，但他還是需要去檢查和維護

MVCC ( Multi-Version Concurrency Control )多版本併發控制

MyISAM適合：做很多count 的計算；插入不頻繁，查詢非常頻繁；沒有事務。

InnoDB適合：可靠性要求比較高，或者要求事務；表更新和查詢都相當的頻繁，並且行鎖定的機會比較大的情況。

爲什麼是如果你的數據執行大量的INSERT或UPDATE，出於性能方面的考慮，應該使用InnoDB表，但是實際可能卻相反呢？？？

I have a table with 17 million rows. I need to grab 1 column of that table and insert it all into another table. Here’s what I did:
INSERT IGNORE INTO table1(name) SELECT name FROM main WHERE ID < 500001
InnoDB executes in around 3 minutes and 45 seconds
However, MyISAM executes in just below 4 seconds. Why the difference?
I see everyone praising InnoDB but honestly I don’t see how it’s better for me. It’s so much slower. I understand that it’s great for integrity and whatnot, but many of my tables will not be updated (just read). Should I even bother with InnoDB?

1

The difference is most likely due to configuration of innoDB, which takes a bit more tweaking than myISAM. The idea of innoDB is to keep most of your data in memory, and flushing/reading to disk only when you have a few spare cpu cycles.

should you even bother with InnoDB is a really good question. If you’re going to keep using MySQL, it’s highly recommended you get some experience with InnoDB. But if you’re doing a quick-and-dirty job for a database that won’t see a lot of traffic and not worried about scale, then the ease of MyISAM may just be a win for you. InnoDB can be overkill in many instances where someone just wants a simple database.

2

InnoDB will be slightly slower because it is ACID compliant, has MVCC and does useful things like actually check foreign keys etc.

3

The reason is very simple. When you insert a row into MyISAM, it just puts it into the server’s memory and hopes that the server will flush it to disk at some point in the future. Good luck if the server crashes.

When you insert a row into InnoDB it syncs the transaction durably to disk, and that requires it to wait for the disk to spin. Do the math on your system and see how long that takes.

You can improve this by relaxing innodb_flush_log_at_trx_commit or by batching rows within a transaction instead of doing one transaction per row.

I highly recommend reading High Performance MySQL 3rd Edition (I am the author).

總結就是 innodb有一系列機制來保證數據安全，所以插入可能會慢但是更安全。