Showing posts with label Software Architecture. Show all posts
Showing posts with label Software Architecture. Show all posts

2012-05-30

CQRS + Event Sourcing Distribute Architecture Design

Preface

Very good I got a chance to attend the DevTeach 2012 at Vancouver. Even though I only get one day to be there, I am still so happy that we will see what other company are going in Architecture level. 

First I must say thanks to those excellent speakers to show my respect:
Rob Daigneau
Dylan Smith
Ted Neward 
Vincent Grondin

This blog is what I learned form Dylan, and the same time added some my thought to make it better in my opinion.

Content

1. What is CQRS 

It stands as Command Query Responsibility Segregation
Strong point:
The core concepts is split write and read behavior to Command Handler and Query Handler, and it is really good to increase scalability for our server.
Imagine, a cluster which have 2 writing server and 20 read server based on 10 database, we can easily make it.

Weak point:
Command/Query handler is easy to implements, but it less extensibility and flexibility for reuse existing under layer modules. For examples: we want detect a behavior if user did command A, and then B, then D, we should do some specific thing for the case.

2. Event Sourcing

We all know the best benefit for Event-driven is we have flexibility and extensibility to hook up new features into the existing software. Based on CQRS's major weakness, we know if we mix this 2 architecture style together, we will have a round result.

3. CQRS + Event Sourcing

Based on Dylan's model, I added some specific part(red circle) in bellow overall figure:
Architecture illustrate for CQRS+ES

Core classes:
CommandHandler
Queryhandler 
DomainObject
Repository
Subscriber

EventHandler
DTO


I do not have time to write a small prototype for this, but I believe it is is good to deal with a large scale system.

4 Benifits

  • Simple
    • E=mc^2. Everything  should be made as simple as possible, not simpler. In this architecture design, a new feature we can easily implement a bunch of classes, e.g. Command, Query, Repository...
  • Single Responsibility Principle
  • Scalability
    1. Split write and read
    2. Support sync and notification evens
  • Flexibility & Extensibility
    • Event Bus allow us plug in new module to handler new events
  • Can be noSQL System 
    • Writing to a file server, QUEUE
  • Testing 
    • Each class are single responsibility, we can easily write unit test with mock


.........................................
Any comments are welcome.

2009-02-24

Memcahced setup


1. libevent-1.3e.tar.gz       libevent-1.3e.tar.gz [GPG Sig] - ChangeLog - Release 2007-09-24



One of the dependencies of memcache is libevent, so firstly download the source files for Libevent http://www.monkey.org/~provos/libevent/.
tar -xvf libevent-1.3b.tar.gz
cd libevent-1.3b
./configure;make;make install;

2. memcached-1.2.2.tar.gz  

Download the latest Memcached source code from danga.com  http://www.danga.com/memcached/
gunzip memcached-1.2.1.tar.gz
tar -xvf memcached-1.2.1.tar
cd memcached-1.2.1
./configure;make;make install;

Often libevent.so cannot be found when executing memcache. A useful command LD_DEBUG, is very helpful to determine where libraries are being loaded from.
 LD_DEBUG=help memcached -v
 LD_DEBUG=libs memcached -v 2>&1 > /dev/null | less
 18990: find library=libevent-1.3b.so.1 [0]; searching
  ...
 18990: trying file=/usr/lib/libevent-1.3b.so.1
 18990:
 memcached: error while loading shared libraries: libevent-1.3b.so.1: cannot open shared object file: No such file or directory
 Simply place the library where memcached will find it and execute memcached.
 ln -s /usr/local/lib/libevent-1.3b.so.1 /lib/libevent-1.3b.so.1
 memcached -d -u nobody -m 512 127.0.0.1 -p 11211
  The options for memcached are:
      -l <ip_addr>
      Listen on <ip_addr>; default to INDRR_ANY. This is an important option to consider as there is no other way to secure the installation. Binding to an internal or firewalled network interface is suggested.
      -d
      Run memcached as a daemon.
      -u <username>
      Assume the identity of <username> (only when run as root).
      -m <num>
      Use <num> MB memory max to use for object storage; the default is 64 megabytes.
      -M
      Instead of throwing items from the cache when max memory is reached, throw an error
      -c <num>
      Use <num> max simultaneous connections; the default is 1024.
      -k
      Lock down all paged memory. This is a somewhat dangerous option with large caches, so consult the README and memcached homepage for configuration suggestions.
      -p <num>
      Listen on port <num>, the default is port 11211.
      -r
      Maximize core file limit
      -M
      Disable automatic removal of items from the cache when out of memory. Additions will not be possible until adequate space is freed up.
      -r
      Raise the core file size limit to the maximum allowable.
      -h
      Show the version of memcached and a summary of options.
      -v
      Be verbose during the event loop; print out errors and warnings.
      -vv
      Be even more verbose; same as -v but also print client commands and responses.
      -i
      Print memcached and libevent licenses.
      -P <filename>
      Print pidfile to <filename>, only used under -d option.

3. memcache-2.2.0.tgz

To install the pecl package for PHP   http://pecl.php.net/get/memcache
wget http://pecl.php.net/get/memcache-2.1.2.tgz
gzip -df memcache-2.1.2.tgz
tar -xvf memcache-2.1.2.tar
cd memcache-2.1.2
phpize
./configure;make;make install;
Add memcache.so to the php.ini file
extension=memcache.so
Then run
php -i | grep -i 'memcache'
memcache should be listed and then restart the web server.

4. SHELL script


#!/bin/bash
#script memcacheDaemon.sh
LANG="zh_CN.utf8"
DAEMON_TIME=10
stopProcess()
{
        DATESTR=`date '+%Y-%m-%d %H:%M:%S'`
        echo "$DATESTR (INFO)  : STOP memcache begin.."
        PID=`ps -ef --cols=200 |grep $1 | awk '{print $2 " " $8}'| grep $1 | sed -e '/grep/d' -e 's/^  *//' -e 's/ .*//'`
        if [ -n "$PID" ]
        then
                echo "$DATESTR (ERROR) : $1 ID $PID, FORCE KILL NORMAL RUNNING PROCESS!"
                kill -9 $PID
        else
                echo "$DATESTR (INFO)  : $1 IS NORMAL DOWN!"
        fi
        echo "$DATESTR (INFO)  : STOP apache end."
}
runDaemon()
{
        DATESTR=`date '+%Y-%m-%d %H:%M:%S'`
        PID=`ps -ef --cols=200 |grep memcached | awk '{print $2 " " $8}'| grep memcached | sed -e '/grep/d' -e 's/^  *//' -e 's/ .*//'`
        if [ -n "$PID" ]
        then
                echo "$DATESTR (INFO)  : memcached NORMAL running. "
        else
                echo "$DATESTR (INFO)  : RELOAD memcached START. "
                if [ "`id -un`" = "root" ] ; then
                        echo "killall memcached"
                        killall memcached
                        echo "su - kangq -c '/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211'"
                        su - kangq -c '/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211'
                else
                        echo "killall memcached"
                        killall memcached
                        echo "/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211"
                        /usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211
                fi
                echo "$DATESTR (INFO)  : RELOAD END."
        fi
}
if [ "`id -un`" = "root" ] ; then
case "$1" in
start)
        DATESTR=`date '+%Y-%m-%d %H:%M:%S'`
        echo "$DATESTR (INFO)  : INTIAL memcached START. "
        if [ "`id -un`" = "root" ] ; then
                echo "killall memcached"
                killall memcached
                echo "su - kangq -c '/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211'"
                su - kangq -c '/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211'
        else
                echo "killall memcached"
                killall memcached
                echo "/usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211"
                /usr/local/bin/memcached -d -m 2048 127.0.0.1 -p 11211
        fi
        echo "$DATESTR (INFO)  : INTIAL END."
        while [ "  " != "" ]
        do
                runDaemon
                ##echo begin sleep ...
                sleep $DAEMON_TIME
                ##echo sleep over ...
        done
        ;;
stop)
        stopProcess memcached
        stopProcess memcacheDaemon.sh
        killall         memcacheDaemon.sh
        ;;
*)
        echo "usage: $0 (start|stop|help)"
        ;;
esac
else
        echo "ERROR: using root run this shell!"
fi
#!/bin/bash
#script memcacheStart.sh
SHELL_DIR=$(cd "$(dirname "$0")"; pwd)
$SHELL_DIR/memcacheDaemon.sh stop
nohup $SHELL_DIR/memcacheDaemon.sh start >>$SHELL_DIR/memcacheDaemon.log &
#!/bin/bash
#script memcacheStop.sh
SHELL_DIR=$(cd "$(dirname "$0")"; pwd)
nohup $SHELL_DIR/memcacheDaemon.sh stop >>/data/share/bin/memcacheDaemon.log &