什么用于限制docker容器

输入:Windows Host,带有内部Ubuntu和Java / Spring应用程序的Docker(限制为2 Gb和2 Gb交换)。应用程序并不简单,我无法创建简单的原型。内存泄漏的问题。但是,所有参数都低于限制,我无法理解为什么在一定负载下一段时间后应用程序被杀死。我发现有许多工具可以测量已用内存,但我无法将它们相互链接。

我已经限制了我知道的所有JVM标志,并添加了诸如Javamelody之类的监视工具。我使用了顶级,免费的sar,docker stat,vmstat工具,它们都显示内存使用没有增加。我不考虑哪个工具或哪个部分?

码头工人:

FROM adoptopenjdk/openjdk11:x86_64-ubuntu-jdk-11.0.3_7-slim
VOLUME /tmp
VOLUME /javamelody
RUN apt-get update && apt-get install procps wget -y
RUN apt-get install fontconfig ttf-dejavu -y
RUN apt-get install libjemalloc1 libjemalloc-dev -y
RUN apt-get install unzip -y
RUN mkdir /usr/local/async-profiler/ &&\
      wget -O /usr/local/async-profiler/async-profiler.tar.gz https://github.com/jvm-profiling-tools/async-profiler/releases/download/v1.6/async-profiler-1.6-linux-x64.tar.gz &&\
      cd /usr/local/async-profiler/ &&\
      tar -xvzf async-profiler.tar.gz &&\
      rm -f /usr/local/async-profiler/async-profiler.tar.gz
RUN mkdir /usr/local/flame-graph/ &&\
    wget -O /usr/local/flame-graph/flame-graph.zip https://github.com/brendangregg/FlameGraph/archive/v1.0.zip &&\
    cd /usr/local/flame-graph/ && unzip flame-graph.zip &&\
    cd /usr/local/flame-graph/
RUN apt-get install sudo -y
RUN apt-get install sysstat -y
RUN apt-get install vim -y
ARG JAR_FILE
COPY target/${JAR_FILE} app.jar
COPY ./docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD java \
            -Xmx896m \
            -XX:CompressedClassSpaceSize=256m \
            -Xss512k \
            -XX:MaxMetaspaceSize=640m \
            -XX:MaxDirectMemorySize=512m \
            -XX:+UseStringDeduplication \
            -XX:+PrintFlagsFinal \
            -XX:+UnlockDiagnosticVMOptions \
            -XX:+PrintNMTStatistics \
            -XX:NativeMemoryTracking=summary \
            -XX:+CrashOnOutOfMemoryError \
            -XX:ErrorFile=/tmp/hs_err_pid%p.log \
            -XX:+HeapDumpOnOutOfMemoryError \
            -XX:HeapDumpPath=tmp/ \
            -jar /app.jar

码头工人组成:

version: '3'

services:
  risky-app:
    image: risky:latest
    security_opt:
      - seccomp:unconfined
    cap_add:
      - ALL
    ports:
      - "9998:9998"
    expose:
      - "9998"
    deploy:
      resources:
        limits:
          memory: 600m
        reservations:
          memory: 600m
    environment:

sar -r 1 10

07:08:00 AM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:08:01 AM     96140    326208   1940876     95.28     58436    351536   6696124    161.97    869360    921972      3512
07:08:02 AM     96148    326356   1940868     95.28     58452    351572   6696124    161.97    869376    922096      3548
07:08:03 AM     96140    326272   1940876     95.28     58452    351592   6696124    161.97    869248    922104      3640
07:08:04 AM     95384    325852   1941632     95.32     58476    351852   6700112    162.07    869280    922460      3908
07:08:05 AM     95384    325864   1941632     95.32     58500    351844   6700112    162.07    869288    922460      3916
07:08:06 AM     95392    326076   1941624     95.32     58516    352040   6700112    162.07    869384    922644      4088
07:08:07 AM     95148    325916   1941868     95.33     58532    352272   6700112    162.07    869384    922728      4172
07:08:08 AM     75788    292372   1961228     96.28     56976    337580   6784708    164.11    897532    913128      4524
07:08:09 AM    113916    328664   1923100     94.41     56992    337708   6699800    162.06    861984    911204      4644
07:08:10 AM    113680    328548   1923336     94.42     57016    337840   6699800    162.06    861992    911288      4760
Average:        97312    323213   1939704     95.22     58035    347584   6707313    162.24    870683    919208      4071

码头工人统计

CONTAINER ID        NAME                     CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
4e8eb37052ef        engine-app_risky-app_1   11.65%              588.8MiB / 600MiB   98.14%              88.5MB / 86.5MB     198MB / 98.3MB      86

最佳

top - 07:09:09 up  1:28,  0 users,  load average: 0.55, 0.50, 0.44
Tasks:   5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.8 us,  6.9 sy,  0.0 ni, 80.8 id,  0.5 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem :  2037016 total,   117588 free,  1492880 used,   426548 buff/cache
KiB Swap:  2097148 total,  1040636 free,  1056512 used.   331312 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                             
    6 root      20   0 2903772 598740   8624 S   9.3 29.4  12:49.02 java
    1 root      20   0    4628      0      0 S   0.0  0.0   0:00.01 sh
   52 root      20   0   20180    956    956 S   0.0  0.0   0:00.00 bash
   62 root      20   0   38256   1036    796 R   0.0  0.1   0:00.87 top                                                                                                                                                                                                 
   93 root      20   0   20308   2016   1540 S   0.0  0.1   0:00.03 bash

免费-tk

root@4e8eb37052ef:/sys/fs/cgroup/memory# free -tk
              total        used        free      shared  buff/cache   available
Mem:        2037016     1490672      104868       68972      441476      332968
Swap:       2097148     1056768     1040380
Total:      4134164     2547440     1145248

jcmd 6 VM.native_memory摘要

Total: reserved=1690873KB, committed=567805KB
-                 Java Heap (reserved=917504KB, committed=237308KB)
                            (mmap: reserved=917504KB, committed=237308KB)

-                     Class (reserved=422788KB, committed=183004KB)
                            (classes #32044)
                            (  instance classes #30284, array classes #1760)
                            (malloc=7044KB #108298)
                            (mmap: reserved=415744KB, committed=175960KB)
                            (  Metadata:   )
                            (    reserved=153600KB, committed=153088KB)
                            (    used=148830KB)
                            (    free=4258KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=262144KB, committed=22872KB)
                            (    used=20326KB)
                            (    free=2546KB)
                            (    waste=0KB =0.00%)

-                    Thread (reserved=44744KB, committed=8536KB)
                            (thread #82)
                            (stack: reserved=44352KB, committed=8144KB)
                            (malloc=295KB #414)
                            (arena=97KB #163)

-                      Code (reserved=255774KB, committed=91102KB)
                            (malloc=8086KB #26448)
                            (mmap: reserved=247688KB, committed=83016KB)

-                        GC (reserved=4023KB, committed=1815KB)
                            (malloc=1031KB #4988)
                            (mmap: reserved=2992KB, committed=784KB)

-                  Compiler (reserved=720KB, committed=720KB)
                            (malloc=589KB #2416)
                            (arena=131KB #5)

-                  Internal (reserved=1752KB, committed=1752KB)
                            (malloc=1712KB #7582)
                            (mmap: reserved=40KB, committed=40KB)

-                     Other (reserved=186KB, committed=186KB)
                            (malloc=186KB #20)

-                    Symbol (reserved=33305KB, committed=33305KB)
                            (malloc=30228KB #396733)
                            (arena=3077KB #1)

-    Native Memory Tracking (reserved=8690KB, committed=8690KB)
                            (malloc=26KB #332)
                            (tracking overhead=8664KB)

-               Arena Chunk (reserved=178KB, committed=178KB)
                            (malloc=178KB)

-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #186)

-                 Arguments (reserved=18KB, committed=18KB)
                            (malloc=18KB #481)

-                    Module (reserved=1187KB, committed=1187KB)
                            (malloc=1187KB #6351)

vmstat -s

root@4e8eb37052ef:/sys/fs/cgroup/memory# vmstat -s
      2037016 K total memory
      1490396 K used memory
       858356 K active memory
       930076 K inactive memory
        99064 K free memory
        58984 K buffer memory
       388572 K swap cache
      2097148 K total swap
      1057280 K used swap
      1039868 K free swap
       117815 non-nice user cpu ticks
            0 nice user cpu ticks
        40635 system cpu ticks
       893337 idle cpu ticks
         7225 IO-wait cpu ticks
            0 IRQ cpu ticks
        10712 softirq cpu ticks
            0 stolen cpu ticks
      1518777 pages paged in
      5141480 pages paged out
        20256 pages swapped in
       293129 pages swapped out
      6238589 interrupts
     32714396 CPU context switches
   1581054022 boot time
         9558 forks

内存状态

root@4e8eb37052ef:/sys/fs/cgroup/memory# cat memory.stat
cache 9875456
rss 612818944
rss_huge 0
shmem 0
mapped_file 1892352
dirty 405504
writeback 0
swap 69070848
pgpgin 320232
pgpgout 168267
pgfault 258357
pgmajfault 2475
inactive_anon 306356224
active_anon 306417664
inactive_file 4861952
active_file 4804608
unevictable 0
hierarchical_memory_limit 629145600
hierarchical_memsw_limit 1258291200
total_cache 9875456
total_rss 612818944
total_rss_huge 0
total_shmem 0
total_mapped_file 1892352
total_dirty 405504
total_writeback 0
total_swap 69070848
total_pgpgin 320232
total_pgpgout 168267
total_pgfault 258357
total_pgmajfault 2475
total_inactive_anon 306356224
total_active_anon 306417664
total_inactive_file 4861952
total_active_file 4804608
total_unevictable 0

问题:

  1. 哪个指标用作docker限制和kubernetes限制的限制?这样,越过该行之后,应用程序将被OOMKilled
  2. 这些是我们可以用来跟踪的所有可能的(有用的)工具吗?
  3. 码头集装箱中的内存?
  4. 是否有其他未在此处显示的隐藏内存消耗器
  5. 为什么sar free命令显示奇怪的内存使用量度?

观察结果:

  1. 应用程序被终止,但是在6000之后没有达到限制
  2. 请求(用作负载测试)
  3. 如果我完全关闭日志记录40000,应用程序不会被杀死
  4. 请求,这很好
  5. 如果我关闭控制台输出,应用程序将被杀死,仅保留
  6. 用rotaiion记录文件10MB
  7. 如果我增加的话,应用程序运行得更好(杀死前有8000个请求)
  8. DB最大连接大小从默认(?)到200。用于postgres和
  9. HikaryCP作为连接池
评论