Monday, August 22, 2016

Show version number in jenkins job history

Everybody handles versioning in a different way but if your jenkins job sends the version to the output stream you can use the below steps to show such version in the build history. This is handy specially for test jobs as you want to make sure that tests did pass for that specific version.

Let us assume that your job prints something like "Front-End Version: 1.2509". Here is how you print the version number in the jenkins job history:
  1. Install "Groovy Postbuild" plugin.
  2. Add a post build action type "Groovy postbuild"
  3. Insert the below in "Groovy Script" and save
    def m = manager.getLogMatcher("^.*Front-End Version: (.*)\$")
    // to debug using job logs
    // manager.listener.logger.println("found version:" + m.group(1))
    if(m != null && m.matches()) {
      version = m.group(1)
      manager.addShortText(version)
    }
    
  4. Run the job and you will get the version number next to the build.

Wednesday, August 03, 2016

Speeding up end to end (e2e) tests

How fast can you end to end test your application? It depends on how fast is your application.

You should parallelize your tests which of course must be idempotent. This means your tests should not step on each other toes. Your tests should take as long as the longest scenario you are trying to assert.

There are several ways to parallelize the tests. For example for web development when using protractor you can use {capabilities: {shardTestFiles: true, maxInstances: n}}. You should not use for n more than max number of processors - 1. You can use tools like jenkins or custom scripts that will spawn several tests at the same time but you will always face the limitations of the hardware used in each node. However you are clustering right? So why bother about the hardware for each node? Here are some numbers from a test run in a 4 processor VM using protractor and direct web driver configured to use 1,2 and 4 maxinstances:
1 instance: 87 sec
Wed Aug  3 09:53:25 EDT 2016
Wed Aug  3 09:54:52 EDT 2016

2 instances: 73 sec
Wed Aug  3 09:59:41 EDT 2016
Wed Aug  3 10:01:54 EDT 2016

4 instances: 117 sec
Wed Aug  3 10:10:06 EDT 2016
Wed Aug  3 10:12:03 EDT 2016
Not big savings really. We know better. Imposing a limitation on how long each test should take should be your premise. From there you act, build tests that include just enough scenarios that do not go beyond your acceptable time to run all e2e for your application, isolate them from each other, distribute the load. Divide and conquer.

I had an argument with somebody (I believe in stack overflow) over the efficiency of running browsers in the background (so called running the browser in headless mode) or in the foreground. When it comes to development, you should run it in the background most of the time so that your screen does not distract you from some other tasks that you are currently performing. When it comes to live testing it does not matter. There are no performance gains whatsoever in running your tests in the foreground or background. Of course if you run in headless mode you can leverage servers instead of desktops but it turns out that desktops allow you to easily debug what is going on in case you really need it: Just RDP into it and you see what is happening there. We must remember that we are testing the user interface/experience because that is how any program in the world ultimately functions: top-down.

Trying headless mode is straightforward. Here is how to do it in debian/ubuntu which I shared in stack overflow:
curl -k https://gist.githubusercontent.com/terrancesnyder/995250/raw/cdd1f52353bb614a5a016c2e8e77a2afb718f3c3/ephemeral-x.sh -o ~/ephemeral-x.sh
chmod +x ~/ephemeral-x.sh
date; ~/ephemeral-x.sh protractor … ; date
Here are my numbers for one instance running in headless mode. Compare the results between the previous GUI based versus the now headless mode. Having the real browser up and running ended up with a slightly better results. But I would not conclude from this that it is better. I would need to repeat this experiment several times before emitting such statement and I really do not have the time for it, but perhaps you do and want to report back your findings:
1 instance headless mode:

Wed Aug  3 10:35:41 EDT 2016
Wed Aug  3 10:37:14 EDT 2016
Bottom line, stop trying to resolve "test inefficiencies" or departing from UAT because "e2e are slow". Face the fact that perception is reality and take care of the performance of your application, make it fast. The application code is not just the runtime code but it is also the test code. All code should be as efficient as it can possibly be. All user interactions included in a particular scenario or group of scenarios should be wrapped in a unique test but only if those meet your test time objective (TTO). If the TTO = 1 minute then no test should take more that 1 minute. Then, spend money on a cluster of machines to perform the tests in parallel and get all tested in less than a minute. These machines can be spawned on demand of course but then account for the startup time as part of your TTO. Humans should be driven by goals and not by tools' deficiencies.

Monday, June 13, 2016

#NoProjects are needed for software development - Prioritize MMF based on multi-dimensional analysis (value, risk and cost of delay)

#NoProjects are needed for software development. Prioritize MMF based on multi-dimensional analysis (value, risk and cost of delay) instead.

I was tempted to share why I think that Projects in software development kill value rather than bring value but before posting I did my homework and I found that somebody wrote about it before.

Prioritization should be about selecting the minimal marketable features (MMF) that bring the biggest value, minimize the cost of delay and/or are bigger risk reducers (what I call multi-dimensional analysis). It should be done continuously just as software should be shipped. Accepting a big effort as more profitable than focusing on MMF delivery is a mistake. The devil is *always* in the details.

Saturday, June 04, 2016

AH00051: child pid ? exit signal Segmentation fault (11), possible coredump in ?

I ran into the below segmentation fault in apache accompanied by high CPU usage in the server:
[Fri Jun 03 06:00:01.270303 2016] [core:notice] [pid 29628:tid 140412199950208] AH00051: child pid 2343 exit signal Segmentation fault (11), possible coredump in /tmp/apache-coredumps
After debugging what the problem was, I concluded that mod proxy was the culprit. There is an apparent recursive bug. I decided not to report it because it was hard to replicate (this happened only late at night) and there was a workaround I could use. This error happened when there was a first request for a resource that the proxied worker could not serve, apache tried to use a specific document for the error but a configuration issue would try to pull such document from the worker as well. Since the document is not in the remote server mod proxy will enter an infinite loop resulting in an apache crash. Most likely a recursion related bug. Clearly resolving the configuration issue to make sure the document is pulled from the local apache hides this potential mod proxy bug.
[Fri Jun 03 05:56:40.831871 2016] [proxy:error] [pid 2343:tid 140411834115840] (502)Unknown error 502: [client 192.168.0.180:58819] AH01084: pass request body failed to 172.16.2.2:8443 (sample.com)
[Fri Jun 03 05:56:40.831946 2016] [proxy:error] [pid 2343:tid 140411834115840] [client 192.168.0.180:58819] AH00898: Error during SSL Handshake with remote server returned by /sdk
[Fri Jun 03 05:56:40.831953 2016] [proxy_http:error] [pid 2343:tid 140411834115840] [client 192.168.0.180:58819] AH01097: pass request body failed to 172.16.2.2:8443 (sample.com) from 192.168.0.180 ()
[Fri Jun 03 05:56:40.844138 2016] [proxy:error] [pid 2343:tid 140411834115840] (502)Unknown error 502: [client 192.168.0.180:58819] AH01084: pass request body failed to 172.16.2.2:8443 (sample.com)
[Fri Jun 03 05:56:40.844177 2016] [proxy:error] [pid 2343:tid 140411834115840] [client 192.168.0.180:58819] AH00898: Error during SSL Handshake with remote server returned by /html/error/503.html
[Fri Jun 03 05:56:40.844185 2016] [proxy_http:error] [pid 2343:tid 140411834115840] [client 192.168.0.180:58819] AH01097: pass request body failed to 172.16.2.2:8443 (sample.com) from 192.168.0.180 ()

Why do I get a Linux SIGSEGV / segfault / Segmentation fault ?

To respond to this question you will need to get a coredump and use the GNU debugger (gdb).
The application generating the segmentation fault must be configured to produce a coredump or should be manually run with gdb in case a segmentation fault can be manually replicated.
Let us pick Apache to go through the debugging steps with a real world example. In order to produce a coredump apache2 needs to be configured:
$ vi /etc/apache2/apache2.conf
...
CoreDumpDirectory /tmp/apache-coredumps
...
Also the OS must not impose limits on the size of the core file in case we don't know how big it would be:
$ sudo bash
# ulimit -c unlimited
In addition the core dump directory must exist and must be owned by the apache user (www-data in this case)
sudo mkdir /tmp/apache-coredumps
sudo chown www-data:www-data /tmp/apache-coredumps
Make sure to stop and start apache separately
$ sudo apachectl stop
$ sudo apachectl start
Look into running processes:
$ ps -ef|grep apache2
Confirm that the processes are running with "Max core file size" unlimited:
$ cat /proc/${pid}/limits
...
Max core file size        unlimited            unlimited            bytes
...
Here is a quick way to do it with a one liner. It lists the max core file size for the parent apache2 process and all its children:
$ ps -ef | grep 'sbin/apache2' | grep -v grep | awk '{print $2}' | while read -r pid; do cat /proc/$pid/limits | grep core ; done
To test that your configuration works just force a coredump. This can be achieved by sending a SIGABRT signal to the process:
$ kill -SIGABRT ${pid}
Analyze the coredump file with gdb. In this case it confirms that the SIGABRT signal was used:
$ gdb core  /tmp/apache-coredumps/core
...
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f1bc74073bd in ?? ()
...
Leave apache running and when it fails with a segmentation fault you can confirm the reason:
$ gdb core  /tmp/apache-coredumps/core
...
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fc4d7ef6a11 in ?? ()
...
Explore then deeper to find out what the problem really is. Note that we run apache now through gdb to get deeper information about the coredump:
$ gdb /usr/sbin/apache2 /tmp/apache-coredumps/core
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fc4d7ef6a11 in apr_brigade_cleanup (data=0x7fc4d8760e68) at /build/buildd/apr-util-1.5.3/buckets/apr_brigade.c:44
44 /build/buildd/apr-util-1.5.3/buckets/apr_brigade.c: No such file or directory.
Using the bt command from the gdb prompt gives us more:
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fc4d7ef6a11 in apr_brigade_cleanup (data=0x7fc4d8760e68) at /build/buildd/apr-util-1.5.3/buckets/apr_brigade.c:44
44 /build/buildd/apr-util-1.5.3/buckets/apr_brigade.c: No such file or directory.
(gdb) bt
#0  0x00007fc4d7ef6a11 in apr_brigade_cleanup (data=0x7fc4d8760e68) at /build/buildd/apr-util-1.5.3/buckets/apr_brigade.c:44
#1  0x00007fc4d7cd79ce in run_cleanups (cref=) at /build/buildd/apr-1.5.0/memory/unix/apr_pools.c:2352
#2  apr_pool_destroy (pool=0x7fc4d875f028) at /build/buildd/apr-1.5.0/memory/unix/apr_pools.c:814
#3  0x00007fc4d357de00 in ssl_io_filter_output (f=0x7fc4d876a8e0, bb=0x7fc4d8767ab8) at ssl_engine_io.c:1659
#4  0x00007fc4d357abaa in ssl_io_filter_coalesce (f=0x7fc4d876a8b8, bb=0x7fc4d8767ab8) at ssl_engine_io.c:1558
#5  0x00007fc4d87f1d2d in ap_process_request_after_handler (r=r@entry=0x7fc4d875f0a0) at http_request.c:256
#6  0x00007fc4d87f262a in ap_process_async_request (r=r@entry=0x7fc4d875f0a0) at http_request.c:353
#7  0x00007fc4d87ef500 in ap_process_http_async_connection (c=0x7fc4d876a330) at http_core.c:143
#8  ap_process_http_connection (c=0x7fc4d876a330) at http_core.c:228
#9  0x00007fc4d87e6220 in ap_run_process_connection (c=0x7fc4d876a330) at connection.c:41
#10 0x00007fc4d47fe81b in process_socket (my_thread_num=19, my_child_num=, cs=0x7fc4d876a2b8, sock=, p=, thd=)
    at event.c:970
#11 worker_thread (thd=, dummy=) at event.c:1815
#12 0x00007fc4d7aa7184 in start_thread (arg=0x7fc4c3fef700) at pthread_create.c:312
#13 0x00007fc4d77d437d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
I was miss led by the messages above until I started changing apache configurations and re-looking into generated coredumps. I realized Apache would fail at any line of any source code complaining about "No such file or directory" in a clear consequence of its incapacity to access resources that did exist:
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb44328d992 in ap_save_brigade (f=f@entry=0x7fb4432e5148, saveto=saveto@entry=0x7fb4432e59c8, b=b@entry=0x7fb435ffa568, 
    p=0x7fb443229028) at util_filter.c:648
648 util_filter.c: No such file or directory.
Disabling modules I was able to get to the bottom of it. An apparent bug in mod-proxy when a miss configuration causes recursive pulling of unavailable resources from the proxied target. Once the issue is found comment or eliminate the configuration from apache:
$ sudo vi /etc/apache2/apache2.conf
...
# CoreDumpDirectory /tmp/apache-coredumps
...
$ sudo apachectl graceful

Increasing swap size in Ubuntu Linux

Here is how to depart with parted (no pun intended) from a swap partition to a swap file in Ubuntu Linux. When in need to increase the swap file it is way easier to increase the size of a swap file than to increase the size of a partition. Starting with the 2.6 Linux kernel, swap files are just as fast as swap partitions. Remove the current swap partition:
$ sudo parted
…
(parted) print                                                            
…
Number  Start   End     Size    Type      File system     Flags
 1      1049kB  40.8GB  40.8GB  primary   ext4            boot
 2      40.8GB  42.9GB  2145MB  extended
 5      40.8GB  42.9GB  2145MB  logical   linux-swap(v1)
…
(parted) rm 2                                                             
(parted) print                                                            
…
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  40.8GB  40.8GB  primary  ext4         boot
…
Delete the entry from stab
# swap was on /dev/sda5 during installation
# UUID=544f0d91-d3db-4301-8d1b-f6bfb2fdee5b none            swap    sw              0       0
Disable all swapping devices:
$ sudo swapoff -a
Create a swapfile with correct permissions, for example the below creates a 4GB one:
$ sudo fallocate -l 4G /swapfile
$ sudo chmod 600 /swapfile
Setup the linux swap area:
$ sudo mkswap /swapfile
Enable the swap device:
$ sudo swapon /swapfile
Confirm that it was created:
$ sudo swapon -s
…
Filename    Type  Size       Used Priority
/swapfile            file  4194300   0     -1
…
Add the entry to /etc/fstab:
/swapfile       none    swap    sw      0       0
Restart:
$ sudo reboot now
Confirm that the swap is active:
$ free
             total       used       free     shared    buffers     cached
Mem:       4048220    1651032    2397188        536      66648     336164
-/+ buffers/cache:    1248220    2800000
Swap:      4194300          0    4194300

$ sudo swapon -s
…
Filename    Type  Size       Used Priority
/swapfile            file  4194300   0     -1
…

Wednesday, May 25, 2016

From subversion to git with the help of gitlab

Here are the steps I recently followed to migrate a subversion repo into git:

Thursday, May 19, 2016

Filter output of Linux top command

To filter by process name:
top -cb | grep processName
Or if you want to interact with top just start it using 'top -c', then press "o" and use as filter "COMMAND=processName"

Thursday, April 21, 2016

Streaming and saving your presentation video

I need to write these steps so often that I rather have them handy
  1. go to https://www.youtube.com/
  2. Click on menu | mychannel
  3. Click on video Manager
  4. Click on Live Streaming | Events
  5. Click on New Live event
  6. Title: ${Name of presentation} ${part number}
  7. Select Option "Unlisted" if you want it to be available to only certain people
  8. Click on "go live now"
  9. Share screen and start broadcast
  10. When done stop broadcast
  11. From the video manager get the link and share it with the relevant people

Followers