ab-mrubyを使って自分のサイトを継続的にテストしてみた

Posted by on 2013年6月11日 Leave a comment (0) Go to comments

最近良く聞く継続的なんとかを僕も試してみたくて、早速最近作ったabコマンドベースのベンチマークで、自由にRubyでベンチマークパターンやテストケースを書けるab-mrubyを使ってWebサーバの継続的なテストをしてみました。

ab-mrubyだとどのように書けるか書いてみたので紹介します。

まずはベンチマークパターン

自分のblogとmoblogとregisterunderflowに対するベンチマークパターンを以下のように記述しました。

$ cat ab-mruby.conf.rb
print <<EOS
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                          CONFIG PHASE

======================================================================
  Target Information  URL: #{get_config('TargetURL').to_s}
  Target Information HOST: #{get_config('TargetHost').to_s}
  Target Information PORT: #{get_config('TargetPort').to_s}
  Target Information PATH: #{get_config('TargetPath').to_s}
  Target Information  SSL: #{get_config('TargetisSSL').to_s}
EOS

# defined ab config pattern
if get_config("TargetHost").to_s == "blog.matsumoto-r.jp"

  add_config(
    "TotalRequests"         => 100,                       # int
    "Concurrency"           => 10,                        # int max 20000
    "KeepAlive"             => true,                      # true or false or nil
    "VerboseLevel"          => 1,                         # int 1 ~ 5
    "ShowProgress"          => false,                     # true, false or nil
    "ShowPercentile"        => false,                     # true, false or nil
    "ShowConfidence"        => false,                     # true, false or nil
    "WaitSocketError"       => true,                      # true, false or nil
    "RequestTimeOut"        => 30,                        # int sec
    "BechmarkTimelimit"     => 50000,                     # int sec
    "WindowSize"            => nil,                       # int byte
    "HeadMethodOnly"        => false,                     # true, false or nil
    "Postfile"              => nil,                       # './post.txt',
    "Putfile"               => nil,                       # './put.txt',
    "ContentType"           => nil,                       # 'application/x-www-form-urlencoded',
    "OutputGnuplotFile"     => nil,                       # './gnu.txt'
    "OutputCSVFile"         => nil,                       # './csv.txt'
    "AddCookie"             => nil,                       # 'Apache=1234'
    "AddHeader"             => 'User-Agent: ab-mruby',    # 'User-Agent: test'
    "BasicAuth"             => nil,                       # 'user:pass'
    "Proxy"                 => nil,                       # 'proxy[:port]'
    "ProxyAuth"             => nil,                       # 'user:pass'
    "OutputHtml"            => false,                     # true, false or nil
    "BindAddress"           => nil,                       # 'matsumoto-r.jp'
    "SSLCipher"             => nil,                       # 'DHE-RSA-AES256-SHA' or get from [openssl ciphers -v]
    "SSLProtocol"           => nil,                       # 'SSL2', 'SSL3', 'TLS1', 'TLS1.1', 'TLS1.2' or 'ALL'
    "SilentMode"            => true,
  )

elsif get_config("TargetHost").to_s == "moblog.matsumoto-r.jp"

  add_config(
    "TotalRequests"         => 200,                       # int
    "Concurrency"           => 20,                        # int max 20000
    "KeepAlive"             => true,                      # true or false or nil
    "VerboseLevel"          => 1,                         # int 1 ~ 5
    "ShowProgress"          => false,                     # true, false or nil
    "ShowPercentile"        => false,                     # true, false or nil
    "ShowConfidence"        => false,                     # true, false or nil
    "WaitSocketError"       => true,                      # true, false or nil
    "RequestTimeOut"        => 30,                        # int sec
    "BechmarkTimelimit"     => 50000,                     # int sec
    "WindowSize"            => nil,                       # int byte
    "HeadMethodOnly"        => false,                     # true, false or nil
    "Postfile"              => nil,                       # './post.txt',
    "Putfile"               => nil,                       # './put.txt',
    "ContentType"           => nil,                       # 'application/x-www-form-urlencoded',
    "OutputGnuplotFile"     => nil,                       # './gnu.txt'
    "OutputCSVFile"         => nil,                       # './csv.txt'
    "AddCookie"             => nil,                       # 'Apache=1234'
    "AddHeader"             => 'User-Agent: ab-mruby',    # 'User-Agent: test'
    "BasicAuth"             => nil,                       # 'user:pass'
    "Proxy"                 => nil,                       # 'proxy[:port]'
    "ProxyAuth"             => nil,                       # 'user:pass'
    "OutputHtml"            => false,                     # true, false or nil
    "BindAddress"           => nil,                       # 'matsumoto-r.jp'
    "SSLCipher"             => nil,                       # 'DHE-RSA-AES256-SHA' or get from [openssl ciphers -v]
    "SSLProtocol"           => nil,                       # 'SSL2', 'SSL3', 'TLS1', 'TLS1.1', 'TLS1.2' or 'ALL'
    "SilentMode"            => true,
  )

elsif get_config("TargetHost").to_s == "registerunderflow.org"

  add_config(
    "TotalRequests"         => 300,                       # int
    "Concurrency"           => 30,                        # int max 20000
    "KeepAlive"             => true,                      # true or false or nil
    "VerboseLevel"          => 1,                         # int 1 ~ 5
    "ShowProgress"          => false,                     # true, false or nil
    "ShowPercentile"        => false,                     # true, false or nil
    "ShowConfidence"        => false,                     # true, false or nil
    "WaitSocketError"       => true,                      # true, false or nil
    "RequestTimeOut"        => 30,                        # int sec
    "BechmarkTimelimit"     => 50000,                     # int sec
    "WindowSize"            => nil,                       # int byte
    "HeadMethodOnly"        => false,                     # true, false or nil
    "Postfile"              => nil,                       # './post.txt',
    "Putfile"               => nil,                       # './put.txt',
    "ContentType"           => nil,                       # 'application/x-www-form-urlencoded',
    "OutputGnuplotFile"     => nil,                       # './gnu.txt'
    "OutputCSVFile"         => nil,                       # './csv.txt'
    "AddCookie"             => nil,                       # 'Apache=1234'
    "AddHeader"             => 'User-Agent: ab-mruby',    # 'User-Agent: test'
    "BasicAuth"             => nil,                       # 'user:pass'
    "Proxy"                 => nil,                       # 'proxy[:port]'
    "ProxyAuth"             => nil,                       # 'user:pass'
    "OutputHtml"            => false,                     # true, false or nil
    "BindAddress"           => nil,                       # 'matsumoto-r.jp'
    "SSLCipher"             => 'AES256-SHA',              # 'DHE-RSA-AES256-SHA' or get from [openssl ciphers -v]
    "SSLProtocol"           => 'SSL3',                    # 'SSL2', 'SSL3', 'TLS1', 'TLS1.1', 'TLS1.2' or 'ALL'
    "SilentMode"            => true,
  )
else

  add_config(
    "TotalRequests"         => 100,                       # int
    "Concurrency"           => 10,                        # int max 20000
    "KeepAlive"             => true,                      # true or false or nil
    "VerboseLevel"          => 1,                         # int 1 ~ 5
    "SilentMode"            => true,
  )

end

それぞれ、SSLの指定や同時接続数などを微妙に変えています。

次にテストスイート

テストは以下のように書きました。大体これぐらいのクオリティは満たしておいて欲しいなぁという値をテストケースにいれました。

$ cat ab-mruby.test.rb
module Kernel
  def test_suite &blk
    @@r = get_config
    @@t = blk
    @@result = true
  end
  def bln_color val
    (val) ? "[¥e[33m#{val}¥e[m]" : "[¥e[36m#{val}¥e[m]"
  end
  def should_be val
    ret = @@r[self] == val
    puts "[TEST CASE] #{bln_color ret} #{self} (#{@@r[self]}) should be #{val}"
    @@result = ret if ret == false
  end
  def should_be_over val
    ret = @@r[self] > val
    puts "[TEST CASE] #{bln_color ret} #{self} (#{@@r[self]}) should be over #{val}"
    @@result = ret if ret == false
  end
  def should_be_under val
    ret = @@r[self] < val
    puts "[TEST CASE] #{bln_color ret} #{self} (#{@@r[self]}) should be under #{val}"
    @@result = ret if ret == false
  end
  def test_run
    @@t.call
    puts "¥ntest suites: #{bln_color @@result}¥n"
  end
end

print <<EOS
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                            TEST PHASE

======================================================================
EOS

if get_config["TargetHost"] == "blog.matsumoto-r.jp"

  test_suite do
    "TargetServerHost".should_be                 "blog.matsumoto-r.jp"
    "TargetServerPort".should_be                 80
    "TargetDocumentPath".should_be               "/"
    "TargetServerSoftware".should_be             "nginx/1.4.1"
    "FailedRequests".should_be                   0
    "KeepAliveRequests".should_be                0
    "WriteErrors".should_be                      0
    "HTMLTransferred".should_be_over             0
    "TargetDocumentLength".should_be_over        0
    "TotalTransferred".should_be_over            0
    "CompleteRequests".should_be                 100
    "TransferRate".should_be_over                4000
    "TimeTakenforTests".should_be_under          5
    "RequestPerSecond".should_be_over            40
    "TimePerRequest".should_be_under             20
    "TimePerConcurrentRequest".should_be_under   200
    "TotalBodySent".should_be                    0
    "ConnetcErrors".should_be                    0
    "ReceiveErrors".should_be                    0
    "LengthErrors".should_be                     0
    "ExceptionsErrors".should_be                 0
    "Non2xxResponses".should_be                  0
  end

elsif get_config["TargetHost"] == "moblog.matsumoto-r.jp"

  test_suite do
    "TargetServerHost".should_be                 "moblog.matsumoto-r.jp"
    "TargetServerPort".should_be                 80
    "TargetDocumentPath".should_be               "/"
    "TargetServerSoftware".should_be             "nginx/1.4.1"
    "FailedRequests".should_be                   0
    "KeepAliveRequests".should_be                0
    "WriteErrors".should_be                      0
    "HTMLTransferred".should_be_over             0
    "TargetDocumentLength".should_be_over        0
    "TotalTransferred".should_be_over            0
    "CompleteRequests".should_be                 200
    "TransferRate".should_be_over                500
    "TimeTakenforTests".should_be_under          5
    "RequestPerSecond".should_be_over            40
    "TimePerRequest".should_be_under             30
    "TimePerConcurrentRequest".should_be_under   500
    "TotalBodySent".should_be                    0
    "ConnetcErrors".should_be                    0
    "ReceiveErrors".should_be                    0
    "LengthErrors".should_be                     0
    "ExceptionsErrors".should_be                 0
    "Non2xxResponses".should_be                  0
  end

elsif get_config["TargetHost"] == "registerunderflow.org"

  test_suite do
    "TargetServerHost".should_be                 "registerunderflow.org"
    "TargetServerPort".should_be                 443
    "TargetDocumentPath".should_be               "/"
    "TargetServerSoftware".should_be             "nginx/1.4.1"
    "FailedRequests".should_be                   0
    "KeepAliveRequests".should_be                0
    "WriteErrors".should_be                      0
    "HTMLTransferred".should_be_over             0
    "TargetDocumentLength".should_be_over        0
    "TotalTransferred".should_be_over            0
    "CompleteRequests".should_be                 300
    "TransferRate".should_be_over                500
    "TimeTakenforTests".should_be_under          30
    "RequestPerSecond".should_be_over            10
    "TimePerRequest".should_be_under             100
    "TimePerConcurrentRequest".should_be_under   3000
    "TotalBodySent".should_be                    0
    "ConnetcErrors".should_be                    0
    "ReceiveErrors".should_be                    0
    "LengthErrors".should_be                     0
    "ExceptionsErrors".should_be                 0
    "Non2xxResponses".should_be                  0
  end

else

  test_suite do
    "FailedRequests".should_be                   0
    "WriteErrors".should_be                      0
    "CompleteRequests".should_be                 100
    "TransferRate".should_be_over                500
    "RequestPerSecond".should_be_over            1000
    "TimePerRequest".should_be_under             100
    "TimePerConcurrentRequest".should_be_under   3000
    "ConnetcErrors".should_be                    0
    "ReceiveErrors".should_be                    0
    "LengthErrors".should_be                     0
    "ExceptionsErrors".should_be                 0
    "Non2xxResponses".should_be                  0
  end

end

test_run

あとは継続的にテスト

今回はシェルスクリプトで適当に書きました。

本当はRakefileみたいなの作って、あるディレクトリ以下にURL毎にディレクトリを作って、その中にそれぞれ上記のテストケースやベンチマークパターンをいれて、rake hogeとかした方がだいぶカッコいいと思いますが、僕の場合はこれで十分です。

$ cat ab-mruby-test.sh
#!/bin/sh

TARGET="
  http://blog.matsumoto-r.jp/
  http://moblog.matsumoto-r.jp/
  https://registerunderflow.org/
"

AB_MRUBY="./ab-mruby"
AB_MRUBY_CONFIG="./ab-mruby.conf.rb"
AB_MRUBY_TEST="./ab-mruby.test.rb"

for url in $TARGET
do
  $AB_MRUBY -m $AB_MRUBY_CONFIG -M $AB_MRUBY_TEST $url
done

そいてこれを実行すると、以下のようにテスト結果が出力されました。

$ sh ab-mruby-test.sh
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                          CONFIG PHASE

======================================================================
  Target Information  URL: http://blog.matsumoto-r.jp/
  Target Information HOST: blog.matsumoto-r.jp
  Target Information PORT: 80
  Target Information PATH: /
  Target Information  SSL: false
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                            TEST PHASE

======================================================================
[TEST CASE] [true] TargetServerHost (blog.matsumoto-r.jp) should be blog.matsumoto-r.jp
[TEST CASE] [true] TargetServerPort (80) should be 80
[TEST CASE] [true] TargetDocumentPath (/) should be /
[TEST CASE] [true] TargetServerSoftware (nginx/1.4.1) should be nginx/1.4.1
[TEST CASE] [true] FailedRequests (0) should be 0
[TEST CASE] [true] KeepAliveRequests (0) should be 0
[TEST CASE] [true] WriteErrors (0) should be 0
[TEST CASE] [true] HTMLTransferred (8875600) should be over 0
[TEST CASE] [true] TargetDocumentLength (88756) should be over 0
[TEST CASE] [true] TotalTransferred (8899200) should be over 0
[TEST CASE] [true] CompleteRequests (100) should be 100
[TEST CASE] [false] TransferRate (2738.769489) should be over 4000
[TEST CASE] [true] TimeTakenforTests (3.173186) should be under 5
[TEST CASE] [false] RequestPerSecond (31.5140681) should be over 40
[TEST CASE] [false] TimePerRequest (31.73186) should be under 20
[TEST CASE] [false] TimePerConcurrentRequest (317.3186) should be under 200
[TEST CASE] [true] TotalBodySent (0) should be 0
[TEST CASE] [true] ConnetcErrors (0) should be 0
[TEST CASE] [true] ReceiveErrors (0) should be 0
[TEST CASE] [true] LengthErrors (0) should be 0
[TEST CASE] [true] ExceptionsErrors (0) should be 0
[TEST CASE] [true] Non2xxResponses (0) should be 0

test suites: [false]
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                          CONFIG PHASE

======================================================================
  Target Information  URL: http://moblog.matsumoto-r.jp/
  Target Information HOST: moblog.matsumoto-r.jp
  Target Information PORT: 80
  Target Information PATH: /
  Target Information  SSL: false
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                            TEST PHASE

======================================================================
[TEST CASE] [true] TargetServerHost (moblog.matsumoto-r.jp) should be moblog.matsumoto-r.jp
[TEST CASE] [true] TargetServerPort (80) should be 80
[TEST CASE] [true] TargetDocumentPath (/) should be /
[TEST CASE] [true] TargetServerSoftware (nginx/1.4.1) should be nginx/1.4.1
[TEST CASE] [true] FailedRequests (0) should be 0
[TEST CASE] [true] KeepAliveRequests (0) should be 0
[TEST CASE] [true] WriteErrors (0) should be 0
[TEST CASE] [true] HTMLTransferred (6698600) should be over 0
[TEST CASE] [true] TargetDocumentLength (33493) should be over 0
[TEST CASE] [true] TotalTransferred (6759000) should be over 0
[TEST CASE] [true] CompleteRequests (200) should be 200
[TEST CASE] [true] TransferRate (4062.9241454) should be over 500
[TEST CASE] [true] TimeTakenforTests (1.62459) should be under 5
[TEST CASE] [true] RequestPerSecond (123.1079841) should be over 40
[TEST CASE] [true] TimePerRequest (8.12295) should be under 30
[TEST CASE] [true] TimePerConcurrentRequest (162.459) should be under 500
[TEST CASE] [true] TotalBodySent (0) should be 0
[TEST CASE] [true] ConnetcErrors (0) should be 0
[TEST CASE] [true] ReceiveErrors (0) should be 0
[TEST CASE] [true] LengthErrors (0) should be 0
[TEST CASE] [true] ExceptionsErrors (0) should be 0
[TEST CASE] [true] Non2xxResponses (0) should be 0

test suites: [true]
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                          CONFIG PHASE

======================================================================
  Target Information  URL: https://registerunderflow.org/
  Target Information HOST: registerunderflow.org
  Target Information PORT: 443
  Target Information PATH: /
  Target Information  SSL: true
======================================================================
This is ab-mruby using ApacheBench Version 2.3 <$Revision: 1430300 $>
Licensed to MATSUMOTO Ryosuke, https://github.com/matsumoto-r/ab-mruby

                            TEST PHASE

======================================================================
[TEST CASE] [true] TargetServerHost (registerunderflow.org) should be registerunderflow.org
[TEST CASE] [true] TargetServerPort (443) should be 443
[TEST CASE] [true] TargetDocumentPath (/) should be /
[TEST CASE] [true] TargetServerSoftware (nginx/1.4.1) should be nginx/1.4.1
[TEST CASE] [true] FailedRequests (0) should be 0
[TEST CASE] [true] KeepAliveRequests (0) should be 0
[TEST CASE] [true] WriteErrors (0) should be 0
[TEST CASE] [true] HTMLTransferred (17557800) should be over 0
[TEST CASE] [true] TargetDocumentLength (58526) should be over 0
[TEST CASE] [true] TotalTransferred (17709600) should be over 0
[TEST CASE] [true] CompleteRequests (300) should be 300
[TEST CASE] [true] TransferRate (686.5145302) should be over 500
[TEST CASE] [true] TimeTakenforTests (25.191792) should be under 30
[TEST CASE] [true] RequestPerSecond (11.9086407) should be over 10
[TEST CASE] [true] TimePerRequest (83.97264) should be under 100
[TEST CASE] [true] TimePerConcurrentRequest (2519.1792) should be under 3000
[TEST CASE] [true] TotalBodySent (0) should be 0
[TEST CASE] [true] ConnetcErrors (0) should be 0
[TEST CASE] [true] ReceiveErrors (0) should be 0
[TEST CASE] [true] LengthErrors (0) should be 0
[TEST CASE] [true] ExceptionsErrors (0) should be 0
[TEST CASE] [true] Non2xxResponses (0) should be 0

test suites: [true]

あっあっ、なんだかblogの処理性能が少し落ちてテストがコケていますね。アクセス集中でもしていたのでしょうか。でもこれで簡単かつ継続的にWebサーバの品質をabコマンドベースのベンチマークで確認できますね。

最後に

このように、abコマンドベースで自由にベンチマークパターンやテストケースを書けるab-mrubyを自分のサイトで継続的になんとかしてみました。

こんな風に簡単にかけてしまうとやはり楽しいものですね。

Apacheについて, プログラムについて, 研究について

Leave a Comment