Release Infrastructure/BOSS/Performance/Status

Contents 1 Memory Leak Showing in BOSS Performance Testing 2 Memory Leak Investigation Methods 3 Investigation in Ruby Application Space 4 Investigation in C Extensions

Memory Leak Showing in BOSS Performance Testing

• From our test results, we found there are memory leak in BOSS(http://wiki.meego.com/Release_Infrastructure/BOSS/Performance/Results)
• Memory occupation increased from initial 20M to over 600M in two days running
• Memory occupation increasing trend is linearity

Memory Leak Investigation Methods

• Generally, memory leak could happen in 3 domains of a Ruby program:
  • Bug in Ruby application(object space - Heap)
  • Broken C extensions
  • Ruby interpreter bug
• Basic Method
  • Running same test cases for different number of loops, then observing the memory difference
• For Ruby interpreter bug, it hardly to track and fix, so we focus on other domains with following methods
  • Bug in Ruby application: Observing the Heap growing by some code profiler tools("ruby-prof", "bleak_house").
  • Broken C extensions: Using valgrind

Investigation in Ruby Application Space

• Using "ruby-prof" with "memory" mode
  • we don't get useful data by this tools - it seems only showing the accumulative memory usage for each object
• Using "bleak_house"
  • Running different cases:
    • Case: load: 1k, iteration: 3

Displaying top 20 most common line/class pairs 188079 total objects 188078 filled heap slots 224574 free heap slots

  5750 /usr/lib/ruby/gems/1.8/gems/yajl-ruby-0.7.6/lib/yajl.rb:39:String
  3052 __null__:__null__:__node__
  2672 (eval):4:__node__
  1976 /usr/lib/ruby/vendor_ruby/1.8/rubygems/specification.rb:186:String
  1373 /usr/lib/ruby/1.8/i586-linux/openssl.so:__null__:__node__
  1340 (eval):3:__node__
  1284 (eval):6:__node__
  1139 (eval):5:__node__
   738 /usr/lib/ruby/gems/1.8/gems/yajl-ruby-0.7.6/lib/yajl.rb:39:Hash
   692 /usr/lib/ruby/gems/1.8/gems/ruby_parser-2.0.5/lib/ruby_parser.rb:1960:__node__
   491 /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31:__node__
   448 /usr/lib/ruby/1.8/fileutils.rb:1578:__node__
   .....

• • • Case: load: 1k, iteration: 10

Displaying top 20 most common line/class pairs 188078 total objects 188077 filled heap slots 224575 free heap slots

  5750 /usr/lib/ruby/gems/1.8/gems/yajl-ruby-0.7.6/lib/yajl.rb:39:String
  3052 __null__:__null__:__node__
  2672 (eval):4:__node__
  1976 /usr/lib/ruby/vendor_ruby/1.8/rubygems/specification.rb:186:String
  1373 /usr/lib/ruby/1.8/i586-linux/openssl.so:__null__:__node__
  1340 (eval):3:__node__
  1284 (eval):6:__node__
  1139 (eval):5:__node__
   738 /usr/lib/ruby/gems/1.8/gems/yajl-ruby-0.7.6/lib/yajl.rb:39:Hash
   692 /usr/lib/ruby/gems/1.8/gems/ruby_parser-2.0.5/lib/ruby_parser.rb:1960:__node__

• • From above test result, we can't observe the heap growth in Ruby application space
• Using Ruby internal util "ObjectSpace" to track heap objects - same conclusion as above

Investigation in C Extensions

• Using "valgrind":
  • case: load 1k, iteration 1:

File:Valgrind 1kx1.log

• • case: load 1k, iteration 5:

File:Valgrind 1kx5.log

• • From above results, we found the "yajl-ruby" library is losing memory(about 155K) in each iteration
  • Locate the real code bug and fix: https://github.com/brianmario/yajl-ruby/issues#issue/36)