Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tidb-server oom-killer #8270

Closed
xiquan-wu opened this issue Nov 12, 2018 · 7 comments
Closed

tidb-server oom-killer #8270

xiquan-wu opened this issue Nov 12, 2018 · 7 comments
Labels
type/bug The issue is confirmed as a bug.

Comments

@xiquan-wu
Copy link

xiquan-wu commented Nov 12, 2018 •
edited
Loading

Install :tidb-ansible
tidb-server memory：32GB
OS：centos7

问题说明：
在测试tidb过程中使用sysbench插入450000000数据，每次必出现tidb-server oom，大约能成功写入的数据量为7千万~9千万条数据左右，服务器内存为32G。请问如何能够解决OOM的问题，或者如何控制tidb-server使用内存的大小？

The test sample:
sysbench --report-interval=1 --db-driver=mysql --mysql-port=4000 --time=300
--mysql-user=root --mysql-db=sysbench --mysql-host=192.168.1.36
--threads=64 --tables=1 --mysql_storage_engine=innodb --skip_trx=off --db-ps-mode=disable
--table-size=450000000 --rand-type=uniform --create_secondary=on
oltp_insert prepare

Error log：

Nov 8 19:47:26 192-168-1-36-TIDB kernel: tidb-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: tidb-server cpuset=/ mems_allowed=0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: CPU: 1 PID: 11455 Comm: tidb-server Kdump: loaded Not tainted 3.10.0-862.14.4.el7.x86_64 #1
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Call Trace:
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] dump_stack+0x19/0x1b
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] dump_header+0x90/0x229
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] ? cred_has_capability+0x6b/0x120
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] oom_kill_process+0x254/0x3d0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] ? selinux_capable+0x1c/0x40
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] out_of_memory+0x4b6/0x4f0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] __alloc_pages_slowpath+0x5d6/0x724
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] __alloc_pages_nodemask+0x405/0x420
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] alloc_pages_current+0x98/0x110
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] __page_cache_alloc+0x97/0xb0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] filemap_fault+0x298/0x490
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] xfs_filemap_fault+0x5f/0xe0 [xfs]
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] __do_fault.isra.58+0x8a/0x100
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] do_read_fault.isra.60+0x4c/0x1b0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] handle_pte_fault+0x2f4/0xd10
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] handle_mm_fault+0x39d/0x9b0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] __do_page_fault+0x197/0x4f0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] do_page_fault+0x35/0x90
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] ? error_swapgs+0xa7/0xbd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [] page_fault+0x28/0x30
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Mem-Info:
Nov 8 19:47:26 192-168-1-36-TIDB kernel: active_anon:8037913 inactive_anon:2167 isolated_anon:0#012 active_file:2298 inactive_file:2781 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:4884 slab_unreclaimable:9498#012 mapped:1706 shmem:2254 pagetables:16780 bounce:0#012 free:49887 free_pcp:1139 free_cma:0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 DMA free:15844kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 8 19:47:26 192-168-1-36-TIDB kernel: lowmem_reserve[]: 0 2829 31993 31993
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 DMA32 free:122444kB min:5972kB low:7464kB high:8956kB active_anon:2751496kB inactive_anon:416kB active_file:260kB inactive_file:588kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129280kB managed:2897824kB mlocked:0kB dirty:0kB writeback:0kB mapped:380kB shmem:424kB slab_reclaimable:1552kB slab_unreclaimable:3032kB kernel_stack:304kB pagetables:5880kB unstable:0kB bounce:0kB free_pcp:1880kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4653 all_unreclaimable? yes
Nov 8 19:47:26 192-168-1-36-TIDB kernel: lowmem_reserve[]: 0 0 29163 29163
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 Normal free:61260kB min:61576kB low:76968kB high:92364kB active_anon:29400156kB inactive_anon:8252kB active_file:8932kB inactive_file:10536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866976kB mlocked:0kB dirty:0kB writeback:4kB mapped:6444kB shmem:8592kB slab_reclaimable:17984kB slab_unreclaimable:34928kB kernel_stack:4064kB pagetables:61240kB unstable:0kB bounce:0kB free_pcp:2676kB local_pcp:696kB free_cma:0kB writeback_tmp:0kB pages_scanned:31295 all_unreclaimable? yes
Nov 8 19:47:26 192-168-1-36-TIDB kernel: lowmem_reserve[]: 0 0 0 0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 DMA: 14kB (U) 08kB 016kB 132kB (U) 164kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15844kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 DMA32: 6544kB (UE) 2658kB (UEM) 11816kB (UEM) 9432kB (UEM) 2564kB (UEM) 10128kB (UEM) 8256kB (UEM) 3512kB (M) 21024kB (EM) 12048kB (U) 254096kB (UM) = 122592kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 Normal: 18744kB (UE) 7808kB (UEM) 30116kB (UEM) 17932kB (UE) 9964kB (UE) 49128kB (UE) 52256kB (UEM) 21512kB (UM) 01024kB 02048kB 0*4096kB = 60952kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: 7621 total pagecache pages
Nov 8 19:47:26 192-168-1-36-TIDB kernel: 0 pages in swap cache
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Swap cache stats: add 0, delete 0, find 0/0
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Free swap = 0kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Total swap = 0kB
Nov 8 19:47:26 192-168-1-36-TIDB kernel: 8388494 pages RAM
Nov 8 19:47:26 192-168-1-36-TIDB kernel: 0 pages HighMem/MovableOnly
Nov 8 19:47:26 192-168-1-36-TIDB kernel: 193317 pages reserved
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 529] 0 529 9769 1220 23 0 0 systemd-journal
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 555] 0 555 50259 387 32 0 0 lvmetad
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 560] 0 560 12069 755 27 0 -1000 systemd-udevd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 665] 0 665 13877 101 26 0 -1000 auditd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 693] 32 693 17305 134 37 0 0 rpcbind
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 696] 0 696 5418 89 15 0 0 irqbalance
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 697] 0 697 6594 85 19 0 0 systemd-logind
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 698] 999 698 134608 2370 61 0 0 polkitd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 700] 81 700 16606 177 34 0 -900 dbus-daemon
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 702] 998 702 29438 114 27 0 0 chronyd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 706] 0 706 48770 114 37 0 0 gssproxy
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 718] 0 718 119676 566 86 0 0 NetworkManager
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 943] 0 943 28203 254 57 0 -1000 sshd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 947] 0 947 55679 1092 44 0 0 rsyslogd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 950] 0 950 143463 2745 99 0 0 tuned
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 959] 0 959 31570 165 20 0 0 crond
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 967] 0 967 6476 51 18 0 0 atd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 987] 0 987 27522 36 9 0 0 agetty
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 1729] 0 1729 22408 253 43 0 0 master
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [ 1788] 89 1788 22451 251 44 0 0 qmgr
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [10704] 0 10704 39704 339 82 0 0 sshd
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [10708] 0 10708 28859 99 16 0 0 bash
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [10881] 1000 10881 161517 3318 40 0 0 node_exporter
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [10882] 1000 10882 28293 46 11 0 0 run_node_export
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [10883] 1000 10883 27041 26 13 0 0 tee
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11053] 1000 11053 5546 1563 15 0 0 blackbox_export
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11055] 1000 11055 28293 46 11 0 0 run_blackbox_ex
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11056] 1000 11056 27041 25 11 0 0 tee
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11408] 1000 11408 8035345 8007312 15665 0 0 tidb-server
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11522] 0 11522 40506 186 36 0 0 top
Nov 8 19:47:26 192-168-1-36-TIDB kernel: [11642] 89 11642 22434 253 45 0 0 pickup
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Out of memory: Kill process 11408 (tidb-server) score 978 or sacrifice child
Nov 8 19:47:26 192-168-1-36-TIDB kernel: Killed process 11408 (tidb-server) total-vm:32141380kB, anon-rss:32028992kB, file-rss:256kB, shmem-rss:0kB
Nov 8 19:47:26 192-168-1-36-TIDB systemd: tidb-4000.service: main process exited, code=killed, status=9/KILL
Nov 8 19:47:26 192-168-1-36-TIDB systemd: Unit tidb-4000.service entered failed state.
Nov 8 19:47:26 192-168-1-36-TIDB systemd: tidb-4000.service failed.
Nov 8 19:47:41 192-168-1-36-TIDB systemd: tidb-4000.service holdoff time over, scheduling restart.
@alivxxx alivxxx added the type/bug The issue is confirmed as a bug. label Nov 12, 2018
@alivxxx
Copy link
Contributor

alivxxx commented Nov 12, 2018

Thanks for your feedback.
Seems like it is a similar question with #8195.
PTAL @tiancaiamao

@tiancaiamao
Copy link
Contributor

What's the TiDB version? @xiquan-wu

See tidb-ansible/inventory.ini

If it's

tidb_version = latest

Please try again, this problem should have been fixed.

@xiquan-wu
Copy link
Author

xiquan-wu commented Nov 12, 2018 •
edited
Loading

grep tidb_version inventory.ini
tidb_version = latest

latest version

@xiquan-wu
Copy link
Author

./tidb-server -V
Release Version: None
Git Commit Hash: 59c4db7
Git Branch: master
UTC Build Time: 2018-10-31 04:59:39
GoVersion: go version go1.11 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

@tiancaiamao
Copy link
Contributor

Oh, it's a buggy version, UTC Build Time: 2018-10-31 04:59:39
The bug is fixed after 2018-11-7
@xiquan-wu

@xiquan-wu
Copy link
Author

OK，I reinstall

@zimulala
Copy link
Contributor

If you have any problems, you can open it again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tiancaiamao @zimulala @xiquan-wu @alivxxx