48JIGEN Reloaded

RubyConf 2016

2016-11-12T00:00:00+09:00

I spent the past three days in Cincinnati, OH at RubyConf 2016. I was a first timer and I enjoyed a lot being a part of RubyConf community. To borrow Matz's RINSWAN(Ruby Is Nice So We Are Nice) from his keynote, IMO we were nice not only because Ruby is nice, but more importantly because we all were trying our best to be nice at any time throughout the conference. All things RubyConf community done during the conference such as inclusive attitude or tons of coordinations by both organizers and attendees made my experience great.

What made me feel most grateful was a few folks just talked to me and gave me some impression about my lightning talk. Despite my spoken English was so awkward I guess, there were many findings to me thanks to their feedbacks.

Thank you RubyConf. You all are awesome. I hope I can come next time again.

Calling Python and Julia libraries from Ruby

2016-10-05T00:00:00+09:00

Classifying documents using doc2vec in Python is already easy, but calling doc2vec from Ruby is even easier. Here is an example of preprocessing text data(in this case manpage) with Ruby and classifying them using doc2vec in Python.

One can call not only Python but Julia from Ruby. Following is an another example to perform K-means Clustering using Julia via Ruby (experimentally ported from official documentation of Clustering.jl).

VirtualModule on which I've been working lately enable you to call arbitrary Python or Julia code from Ruby. Here is yet another example to perform an SVM classification using cross-validation through scikit-learn from Ruby.

It looks perfectly sweet except for the fact that I needed to pass :_ Symbol whenever calling functions which take no arguments. This is required because Ruby syntax allows us to call functions without using parentheses, otherwise VirtualModule will never know if the message sent to the object is calling functions or accessing instance variables.

Playing with VirtualModule in REPL

In this section, I'm going to try my best to show you what's happening in VirtualModule internally. I'm assuming following are installed on the system already:

virtual_module gem (v0.3.0 or higher)
Python or Julia environment

Then here we go. Now we start to run irb on the terminal.

debussy:~ remore$ irb -r virtual_module
irb(main):001:0> po = VirtualModule.new(:python=>["sklearn"=>"datasets"])
=> #<Module:0x007fb7e1aee818>

Once you call VirtualModule#new, Python or Julia Process will be booted as a background job. If the background job successfully booted, then VirtualModule will return new Module instance, which behaves as a proxy to background job (In the example above, it's assigned to local variable po). To simplify, in this document we call it PO(Proxy Object).

irb(main):002:0> py.int(2.3)
=> 2
irb(main):003:0> po.unknown_method(2.3)
RuntimeError: An error occurred while executing the command in python process: ,name 'unknown_method' is not defined

What PO is doing internally is as simple as ABC. In the first example, PO receives int(2.3) method call and pass through this to the background process using msgpack. Consequently, Fixnum 2 is displayed on the terminal, which value is sent from the background job. If unknown method is called, background job will tell that there is no such method defined. Since data transfer is implemented using msgpack, at this moment only very limited types of data defined by msgpack spec is converted to Ruby's data type.

irb(main):004:0> po.datasets
=> #<Module:0x007ffd0906c030>
irb(main):005:0> po.datasets.load_iris(:_)
=> #<Module:0x007ffd09074500>
irb(main):006:0> po.datasets.load_iris(:_).vclass
=> "<class 'sklearn.datasets.base.Bunch'>"
irb(main):007:0> po.datasets.load_iris(:_).data[1].to_a
=> [4.9, 3.0, 1.4, 0.2]

What will happen if the data conversion by msgpack fails? The example above shows the behavior when one try to access unconvertible data type using msgpack. In the example above, a PO instance(local variable "po") creates new PO instance(#<Module:0x007ffd0906c030> etc) on every single method call, until the background process return values which are successfully converted to Ruby's data type by msgpack.

irb(main):008:0> po.datasets.vclass
=> "<type 'module'>"
irb(main):009:0> iris = po.datasets.load_iris(:_)
=> #<Module:0x007ffd09057568>
irb(main):010:0> iris.target.vclass
=> "<type 'numpy.ndarray'>"
irb(main):011:0> iris.target.vmethods
=> ["T", "__abs__", "__add__", "__and__", "__array__", "__array_finalize__", "__array_interface__", "__array_prepare__", "__array_priority__", "__array_struct__", "__array_wrap__", "__class__", "__contains__", "__copy__", "__deepcopy__", "__delattr__", "__delitem__", "__delslice__", "__div__", "__divmod__", "__doc__", "__eq__", "__float__", "__floordiv__", "__format__", "__ge__", "__getattribute__", "__getitem__", "__getslice__", "__gt__", "__hash__", "__hex__", "__iadd__", "__iand__", "__idiv__", "__ifloordiv__", "__ilshift__", "__imod__", "__imul__", "__index__", "__init__", "__int__", "__invert__", "__ior__", "__ipow__", "__irshift__", "__isub__", "__iter__", "__itruediv__", "__ixor__", "__le__", "__len__", "__long__", "__lshift__", "__lt__", "__mod__", "__mul__", "__ne__", "__neg__", "__new__", "__nonzero__", "__oct__", "__or__", "__pos__", "__pow__", "__radd__", "__rand__", "__rdiv__", "__rdivmod__", "__reduce__", "__reduce_ex__", "__repr__", "__rfloordiv__", "__rlshift__", "__rmod__", "__rmul__", "__ror__", "__rpow__", "__rrshift__", "__rshift__", "__rsub__", "__rtruediv__", "__rxor__", "__setattr__", "__setitem__", "__setslice__", "__setstate__", "__sizeof__", "__str__", "__sub__", "__subclasshook__", "__truediv__", "__xor__", "all", "any", "argmax", "argmin", "argpartition", "argsort", "astype", "base", "byteswap", "choose", "clip", "compress", "conj", "conjugate", "copy", "ctypes", "cumprod", "cumsum", "data", "diagonal", "dot", "dtype", "dump", "dumps", "fill", "flags", "flat", "flatten", "getfield", "imag", "item", "itemset", "itemsize", "max", "mean", "min", "nbytes", "ndim", "newbyteorder", "nonzero", "partition", "prod", "ptp", "put", "ravel", "real", "repeat", "reshape", "resize", "round", "searchsorted", "setfield", "setflags", "shape", "size", "sort", "squeeze", "std", "strides", "sum", "swapaxes", "take", "tobytes", "tofile", "tolist", "tostring", "trace", "transpose", "var", "view"]
irb(main):012:0> iris.target.to_a
=> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]

Just like Object in Ruby provides #class and #methods to know much about object instance, VirtualModule provides corresponding methods - #vclass and #vmethods. #vclass will tell you what data type a proxy object is referring to, and #vmethods will show you a list of functions you can request. If you'd like to see more examples, please consider to visit my GitHub repo and refer the other usages. And if you have any question, please feel free to post Issue.

Currently, VirtualModule has numerous drawbacks, as many other OSS software at the early stage has. For example, VirtualModule cannot handle Julia's good parts very well such as multidimensional arrays and Functions as first-class objects. Although I know this series of attempts to make it possible to call Python and Julia from Ruby are not enough to satisfy data scientist's needs yet, but somehow exciting enough to get my attention in the last few months. Moreover, in my opinion both the environment and the situation around scientific computing in Ruby has started to get improved these days, as some sources described about this as well. If you are interested in seeing what's happening these days around Ruby, I strongly recommend joining SciRuby community. You will find many other interesting projects and activities there.

Bytecode cache is experimentally released in Ruby2.3

2016-01-05T00:00:00+09:00

In RubyKaigi2015 held at Tokyo last month, Koichi Sasada has announced the experimental release of bytecode cache in Ruby2.3. You are likely to be able to test this experimental feature as follows for example, by verifying bytecode cache(.yarb file) is generated successfully.

$ ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
$ ls /usr/local/lib/ruby/gems/2.3.0/gems/activesupport-4.2.5/lib/
active_support  active_support.rb
$ export RUBYOPT='-ryomikomu' && export YOMIKOMU_AUTO_COMPILE='true'
$ rails -v
Rails 4.2.5
$ ls /usr/local/lib/ruby/gems/2.3.0/gems/activesupport-4.2.5/lib/
active_support  active_support.rb  active_support.rb.yarb

All you need to do to make this feature enabled is just to require 'yomikomu' rubygem and set some environmental variables introduced at here as you can find two export commands in the example above.

It may look a bit magical why VM-level bytecode cache is enabled only by requiring 'yomikomu' rubygem. Koichi described about this at his ticket.

The key interface is RubyVM::InstructionSequence.load_iseq(fname). When MRI try to load any script named fname, then call this method with fname if defined. The return value is an ISeq object, then MRI use this ISeq object instead of parsing/compiling fname file.

In short, 'yomikomu' rubygem defines RubyVM::InstructionSequence.load_iseq while VM(MRI) checks if #load_iseq method is defined at its boot timing and thus VM decides cache is enabled or not. We can find corresponding lines of code in ruby/iseq.c.

Here is a quick benchmark result of current bytecode cache implementation. I used 'bundle version' command with benchmark-ips on Ubuntu machine. Source

$ ruby measure.rb
Comparison:
        yomikomu(fs):        5.0 i/s
         yomikomanai:        3.6 i/s - 1.40x slower

It may vary depending on what command is used but from my quick check bundler was the most optimized in terms of speed. For more information including more benchmark score, please refer Koichi's presentation material presented at RubyKaigi2015 as well as his speech video.

Now is the time to deep-dive for speeding up MRI towards Ruby3.0!

Setting Up Blog With Octopress on Github Pages

2012-11-04T00:00:00+09:00

Here is my command log to setup this this blog with octopress on github pages.

# Create new EC2 intance(micro)
yum update
yum install make git ruby-devel gcc rdoc zlib-devel

# Install rvm
curl -L https://get.rvm.io | bash -s stable
source "$HOME/.rvm/scripts/rvm"

# Install ruby1.9.3
rvm get head
rvm install 1.9.3
rvm use 1.9.3

# Install bundler
gem install bundler

# Install octpress
git clone git://github.com/imathis/octopress.git octopress
cd octopress
bundle install --path vendor
rake setup_github_pages # answer your github page url
rake install

# Build and deploy
rake generate
rake deploy

Note that user permission and current folders are not considered carefully.

Mock Object in PHPUnit ROCKS

2012-11-03T00:00:00+09:00

Suppose you fake it like this:

class TwitterClient
{
    public function tweet($message)
    {
        return true;
    }
}

$foo = new TwitterClient();
$foo->tweet('Spot ko!'); // returns true

With PHPUnit, you’re able to use Mock Object instead. Now your code looks like:

class ClientContainerTest extends PHPUnit_Framework_TestCase
{
    public function test_MockPractice()
    {
        $foo = $this->getMock('TwitterClient', array('tweet'));
        $foo->expects($this->any())
            ->method('tweet')
            ->will($this->returnValue(true));

        $foo->tweet('Spot ko!'); // returns true
    }
}

It’s quite easy to understand how method(), will() and returnValue() work. How about expects()? It restricts the number of invocation count. Set any() then no restriction will be set. If you would like some method to be invoked only once, then set once() for example.

This is how you create Mock Object in PHPUnit, however it’s still hard to find out why Mock Object is useful. Let’s take another example, which accesses with database server via PDO.

Supporse you would like to run your test but you can’t run database server. I’m showing you a bad example first:

class PDO
{
    public function __construct($dsn)
    {
        return true;
    }

    public function query($sql)
    {
        return new PDOStatement();
    }
}

$pdo = new PDO('sqlite:memory'); // Fatal Error
$pdo->query('SELECT * FROM users;');

Run the code above, then you will see error message on your display which says “Cannot redeclare class PDO”. As PDO is embedded class and declared already, you cannot redeclare PDO. To use sqlite:memory which enable you to create database on memory was pretty neat though.

This is the case where you should remember Mock Object.

class PDOContainerTest extends PHPUnit_Framework_TestCase
{
    public function test_StubQuery()
    {
        $foo = $this->getMock('PDO', array('query'), array('sqlite:memory'));
        $foo->expects($this->any())
            ->method('query')
            ->will($this->returnValue(new PDOStatement()));

        $foo->query('SELECT * FROM users;') // instantiated PDOStatement class will be returned
        $foo->getAvailableDrivers(); // take note that methods originally declared in PDO class is available too
    }
}

With this test code, $foo will have stubbed query() method as well as methods originally declared in PDO class such as getAvailableDrivers() in the example above. This is how Partial Mock is made.

If you only need to prepare fixture for DB for example, then you have other even better choises like Database Extension of PHPUnit or Phactory etc. But how about your legacy code is doing for example:

Calling some webservice method(API) and depending on the behavior of the external module
Depending on the volatile data which memcached keeps at the production environment

would you still like to search the web and run out of your limited time?

To learn mock framework which doesn’t focus on solving problem in specific domain is pretty useful. Whatever issues you are facing with, mock framework gives you higher testability of the desgin you are making, which means refactoring becomes much easier for you.

Go on the next chapter and let’s take a look at the various mock frameworks written in PHP.

Mock Frameworks Written in PHP

Here is the result of my quick investigation on Oct 2012.

Framework	Required PHP Environment	Website	Note
PHPUnit	PHPUnit3.7: require PHP 5.3.3 and up(PHP 5.4.7+ is recommended) PHPUnit3.6: require even with PHP 5.2	Website	Sample Code
Mockery	PHP 5.3.2 and up	Website	Sample Code
Phake	PHP 5.2.4 and up	Website	Sample Code
SimpleTest	SimpleTest1.1: require PHP 5.0.3 and up Older version of Simpletest work with PHP 4	Website	Sample Code
Phockito	(Not Yet Confirmed)	Website	N/A
yayMock	(Not Yet Confirmed)	Website	N/A

From my quick skim, PHPUnit and Mockery look useful enough to me. And then, What is the pros/cons of these two mock frameworks and finally how to pick one?

The table below shows what feature is simple comparison of mock framework features.

Functionality	PHPUnit 3.7	Mockery 0.7.2	Note
Invocation Count Constraint	OK	Love It	Mockery is better. atMost() and atLeast() methods are available.
Partial Mocking	OK	Very Good	Constructer can be much simpler in Mockery.
Argument Matchers	OK	OK	All mathers at table 4.3 are available in PHPUnit by with() call. About Mockery, refer README.md
Cascading Mocks\|center	OK	Excellent	Just a bit hustle with PHPUnit but doable or (like this). With Mockery, here is example in README.md
Ordered Expectations	Can’t	OK	With Mockery it’s possible but not comfortable

As you can see the table, Mockery 0.7.2 is definitely better in terms of functionality. Actually Mockery is better at other aspects as well:

Mocking Public Properties
Preserving Pass-By-Reference Method Parameter Behaviour
(Since Mockery supports only PHP5.3 and up so this means closure is available, )Mock Object Recording

Then what about PHPUnit3.7? In my opinion, it’s still useful if you are just newbie on using mock framework.

Lately PHPUnit is broadly used in PHP development and therefore, once you learn to write mock object then for sure, you will be able to apply your experience to the design issue your team will face.

Reference

Difference between the Mock Object in PHPUnit and Mockery / Why Mockery?
Code exmamples of Mockery, Phake, Phockito and PHPUnit / gist: 1753804 / Example temperature webservice

Code To Learn

You must know about the book “Growing Object-Oriented Software Guided by Tests” a.k.a #goos by Steve Freeman and Nat Pryce. And you also know that there was a book Steve and Nat contributed about same topic before goos?

“Mock Roles, not Objects” is it. The pdf just consists of 11 pages, however the concept and the idea they explained in the book is worth reading. I promise that you will not waste your time to read this book deliberately.

Various Sample Codes Are Available

I wrote already in both PHPUnit nad Mockery and they are available at my github account. Kindly feel free to send me pull request if you find something wrong about my code. Other than that, @iakio seems wrote in PHP as well.

Or if you are Java developer, then the example by @digitalsoul0124 may help. There are tons of example in the world wide web so that you can find your favorite example with your favorite programming language. ( Let me add that in Java the example in jmock1 and the test, or there is also example in jmock2)

Tips On Coding “Mock Roles, not Objects”

Better to know basics about OOP and design patterns. Text implies useful and practical idea of OOP such as “object oriented style based on composition rather than inheritance” or “Programming by Composition” but if you don’t know OOP, these words will not make sense at all.
To set up PHPUnit environment, tips by suin works to me.(Sorry it’s written only in Japanese, but you can guess by reading just commands and source codes;) ) If you would like to use under 5.2 where you can’t use composer unfortunately, you need to use PEAR instead. – Take note that phar(PHP Archive) becomes available from PHPUnit3.7.5
It would be small point but let me tell you as It was a bit surprising to me. In PHPUnit, once you call getMock() then it increments number of assertions of final test run result even without calling any expects to the instantiated mock object. Guessing verify() equivalent function would be called at the end of test running. This unexpected things robbed me from a few hours to get understand well.

You CANNOT do method chaining with using getMock() instance directly(I found someone at stackoverflow did completely same as me). For example, this code will cause error:

      $foo = $this->getMock('PDO', array('query'))
          ->expects($this->any())
          ->method('query')
          ->will($this->returnValue(array('id'=>7, 'name'=>'Ichiro Suzuki')));

HashMap doesn’t make sense to most of PHP programmers. In PHP, let’s say it’s just a Array.(* I know strictly speaking Hashmap in Java is different from Array in PHP AT ALL but here just let me please explain just for simplification. [[Go stackoverflow>http://stackoverflow.com/questions/6841379/is-there-java-hashmap-equivalent-in-php]] if you would like to keep talking about this)
In the original pdf document, you’ll frequently see verify() method are called, but in PHPUnit, verify() equivalent function will be called automatically at the end of test running, so you do not need to take care about verify() calling in case you would use PHP. ( use mockery_verify in Mockery and verify() in Phake instead)
Inner class is used at some code snippet but in PHP inner class is not supported. Just ignore and take those class out of class is enough so far.

That’s all for today. Mock object in PHPUnit ROCKS.

Setting Up PHP5.4/Composer Application on Windows With #Chocolatey

2011-10-07T00:00:00+09:00

I liked Chocolatey concept which had been talked at HN two weeks ago: “Kinda like apt-get, but for Windows”. But how much is it useful actually?

Let’s see it by getting PHP5.4 application work on Windows environment.

Setting Up PHP5.4 On Windows

Download zipped PHP 5.4.XX binary
Unzip them and rename the folder(assuming “C:\php54” is the folder name)
Rename “C:\php54\php.ini-development” to “C:\php54\php.ini”

And lastly, edit C:\php54\php.ini and make following two extensions enable by trimming semicolon:

extension=php_curl.dll
extension=php_openssl.dll

Don’t forget to set the PATH system variable from system settings. For example:

...;C:\php54;

Getting Chocolatey And Composer Ready

Hit the command to get chocolatey ready:

C:\> @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

Install curl, git and svn commands with chocolatey, as applications available on Composer often depend on them.

C:\> cinst curl
C:\> cinst git
C:\> cinst svn

Here You Go

This is just an example to install BEAR.Package. See the official wiki if you need more detailed instruction.

C:\tmp> curl -s http://getcomposer.org/installer | php
C:\tmp> php composer.phar create-project -s dev --dev bear/package ./bear
C:\tmp> php .\bear\apps\Helloworld\public\min.php
HelloWorld!

Note

At this moment(Feb 17th, 2013), Installing PHP5.4 via cinst failed due to 404 error:

C:\> cinst curl php5-dev

Either cinst or php5-dev seems not maintainanced very well. I already commented this to the owner page of php5-dev.

48JIGEN *Reloaded*