What seniority means for a software engineer? Is it familiarity with frameworks, libraries or something different? In this post, I will try to summarise my perspective on how it is changing for me. Got a lot of view during the last three years when I have lead or participated in a decent amount of interviews for engineering roles, including the C-level executives. Let state the obvious: fair and good hiring is super hard. During the hiring process, you try to assess the seniority of the candidate(and other qualities for sure) according to the company career ladder. As a reference, industry-respected and well-known ladder description can be used Google career ladder. If you haven’t had an opportunity to familiarise yourself with it, take the time and do so, it is definitely worth understanding. Google career ladder construction anchored in Google perfect team study. This post doesn’t attempt to mimic Google research but rather provide a starting point and raise curiosity. Excellent management is where the resulted outcome of the team is far greater than the pure sum of individual contributions. 

Seniority levels

Seniority levels are usually a combination of technical “hard” skills and other skills collectively referred to as “soft” skills. Unfortunately, relatively few companies go beyond this level of explanation. 

Junior level is characterised by a strong desire to learn new tools, frameworks and techniques. Often connected with black and white view on problems and when solving a task, junior developers can come up with a single possible solution or more with slight modifications. It is nothing wrong to be at that stage, and healthy teams often contain some junior developers as they bring fresh trends and passion to the teams. However, having more than 25% of junior developers in the team is challenging as they require more attention and slows the team down or affect the quality. Be careful with this mix.

Gaining seniority

Gaining seniority goes on a few axes – technical skills, context developer is using during the evaluation of possible solutions and ability to communicate effectively. Junior developer has limited technical skills, and his context is limited to code he is writing and communicates solely in terms of code, requirements or tickets. Context is a vital vehicle for guiding a developer in two remaining axes. As technical skills are growing number of the possible solution raises as well. Context can be divided but not limited to the following scopes:

Codebase

Limits the knowledge to the current codebase, used technologies and design principles and thinking in those low-level terms. This definition can be confusing as project codebase is involved every time as the ultimate source of truth but what differs is the level of abstraction. Developers are growing by operating bigger codebase and experience on different project codebase.

Development habits and practices

This area covers topics like where to apply which kind of tests. Where are the areas you can compromise if chasing time, and what are the real costs? Tech debt management. Essential features for long term supportability. Those items can be categorised under some risk management and damage control practices.

System design

Designing a system for performance, scalability, security and similar system-wide aspects. Defining thread zones and expected failure scenarios. Think in terms of consistency and ability to enforce those policies on the platform level. What can be handled on infrastructure level vs what is necessary to address in the application code?

Different mindset or roles

Understanding of other functions and associated mindsets of people participating in the development process (system engineers, quality assurance, machine learning engineers, project managers, etc.). What helps the best in this area is to try out the majority of those roles. Getting the role mindset will simplify communication during cross-team communication.

Software development life cycle

Ability to foresee and expect requirements which might not be implicitly obvious or stated in the desired functionality description. Understanding of product lifecycle stages starting from ideation till end-of-life. An item like supportability, troubleshooting etc. What helps in those aspects is to take the perspective of technical support personnel and think in terms of what-if. How difficult would it be to discover that component/feature X doesn’t work as expected.

Technology overview

Knowledge of technologies and alternatives gives you lego blocks for building a final solution architecture. This is especially important for the cloud-based product as you are trading a cost for a speed of delivery. Similarly, the same applies to programming languages. Those have specific features, design patterns etc. 

Cross disciplines

This context is pretty similar to `Different mindset roles` with an only difference; it is focused on actual skills necessary in those roles and effects the tools have on team dynamics. Disciplines like development, operation, infrastructure, marketing and similar. 

Cross-level and cross-department awareness

Understanding main objectives, areas of competence and levels of abstractions those people operate on. Unfortunately, even though role titles repeat competencies and responsibilities differs significantly between companies. That is what makes the execution different for each company and is one of the secret sauce of success. Effects are visible in accounting and balance sheets. 

Management and leadership skills

Guide the team without command and control approach. Leading the change throughout the team, getting the buy in to support the change and move things forward.

Business domain

Understanding of industry you operate in helps you to drive better value for your customers by combining your technical abilities with domain knowledge.

Business models

Understanding how the business makes money on the product you build helps you to understand key aspects of the technical solution in terms of desired points of flexibility, scalability and cost management.

Wrap up

Knowledge and experiences in those scopes provide a better context for making decisions in the bigger picture and finding a more strategic solution for given conditions. Helps you to drive communication more effectively as you can think more like your counter-party and understand his point of view. Lastly, you should be able to come up with a plan of delivery iterations with a value-added in each iteration and mitigated risk. 

The more senior you get on technical skills and proceed on the career ladder, the more the role start rely on your leadership skills. The vast amount of companies than forces people to manager roles but some (including Google, Facebook, etc.) allows to continue on technical track and label those positions as Principal or Staff Engineers. The role of those engineers is usually to improve the engineering culture and best practices of the teams apart from solving complex engineering tasks. 

I neglected at least one aspect in my writing, the effect of personal characteristics and perspectives for particular roles. I am not going into those but instead refer to great Neil blog with people topologies

What’s the number one lesson you’ve learned since starting in tech as a junior developer? Tweet me @jak_sky and share your story.

Resources for further study

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

Team Topologies: Organizing Business and Technology Teams for Fast Flow

The Five Dysfunctions of a Team

Managing Humans

The Invincible Company

Value Proposition Design: How to Create Products and Services Customers Want

The Business Model Navigator

If you like the content or have some experiences or questions don’t forget to leave a comment bellow and follow me on Twitter.

Teams invest significant effort in making their solution transparent, easy to troubleshoot and stable by instrumenting their product and technology which is then connected to their monitoring system (e.g. Prometheus as one of the popular solutions). The only question remains: Who monitors the monitoring? If the monitoring is down you won’t get any alarms about the malfunction of the product and you are at the begging. So you add monitoring of a monitoring system. Availability tough game starts. If both components have 95% availability you effectively achieve 90%. I believe that you got the rules of the game we play.

Dead man’s switch

How it is monitoring of monitoring setup in practice? Well, the implementation differs. But the core idea is the same and works well even in the army! It is called Dead man switch. It is based on the idea that if we are supposed to receive a signal for triggering an alarm in an unknown moment we need to guarantee that the signal can trigger an alarm anytime. Reversing the logic for triggering the alarm will give us that guarantee so having a signal which we receive constantly and the alarm is triggered when we don’t receive a signal. So simple! This principle (heartbeat) is used in multiple places e.g. clustering. In the army, they use it as well.

Prometheus dead man switch

Some monitoring tools they have this capability built-in but Prometheus doesn’t. So how to achieve this in order to sleep well that the watcher is watching.

Prometheus alert rule

We need to set up a rule that is constantly firing in Prometheus. There can be a rule like this:

    - name: monitoring-dead-man
      rules:
      - alert: "Monitoring_dead_man"
        expr: vector(1)
        labels:
          service: deadman
        annotations:
          summary: "Monitoring dead man switch should always fire alert"
          description: "Monitoring dead man switch for probing alert path"

where service label will be used further in the next step (or similar alternative).

Alertmanager dead man switch Route

Now we need to create a heart beating. The rule on its own would fire once then it would be propagated to Prometheus Alert Manager (component responsible for managing alerts) and all would be over. We need a regular interval for our check-ins. Interval is given by availability you want to achieve as that all adds to reaction time you need. You can achieve this behaviour by special route for your alert in an Alert Manager:

      - receiver: 'DEAD-MAN-SNITCH'
        match:
          service: deadman
        repeat_interval: 5m

Dead man snitch Reciever

Now we need to achieve an alarm trigger reverse logic. In our particular case, we use Dead Man Snitch  which is great for monitoring batch jobs e.g. data import to your database. It works in a way that if you do not check in in the specified interval it triggers with the lead time given by interval.

You can specify the rules when to trigger but those are the details of the service you use. All you need to add to Prometheus is a receiver definition checking in particular snitch as follows:

       - name: 'DEAD-MAN-SNITCH'
          webhook_configs:
            -  url: 'https://nosnch.in/your_snitch_id'

URL is taken for created snitch as an example here

Pagerduty dead man’s switch

The last thing you need to do is integrate the trigger with the system you use for on-call rotta management e.g. PagerDuty . For this example integration between dead man snitch and pagerduty or you can integrate it via pager duty email trigger but that will be less reliable as it can change unnoticed. 

Summary

Prometheus doesn’t have heartbeat service by default but we achived a desired functionality by using functionality of alert manager for resending alerts in combindation with 3rd party service which reverse the trigger logic. Let me know if is it working for you or whether you fund a better way how to achive the desired functionality. You can leave a comment below or you reach me on twitter.

Helm package manager

Kubernetes Helm is a package manager for Kubernetes deployments. It is one of the possible tools for deployment management on Kubernetes platform. You can imagine it as an RPM in Linux world with package management on top of it like an apt-get utility.

Helm release management, ability to install or rollback to a previous revision, is one of the strongest selling points of Helm and together with strong community support makes it an exciting option. Especially the number of prepared packages is amazing and make it extremely easy to bootstrap a tech stack on the kubernetes cluster. But this article is not supposed to be a comparison between the kubernetes tools but instead describing an experience I’ve made while working with it and finding the limitations which for some else might be quite ok but having an ability to use the tool in those scenarios might be an additional benefit.
Helm is written in GO lang with the usage of GO templates which brings some limitation to the tool. Helm works on the level of string literals, and you need to take care of quotation, indentation etc. to form a valid kubernetes deployment manifest. This is a strong design decision from the creators of the Helm, and it is good to be aware of it. Secondly, Helm merges two responsibilities: To render a template and to provide kubernetes manifest. Though you can nest templates or create a template hierarchy, the rendered result must be a Kubernetes manifest which somehow limits possible use cases for the tool. Having the ability to render any template would extend tool capabilities as quite often the kubernetes deployment descriptors contain some configuration where you would appreciate type validation or possibility to render it separately. At the time of writing this article, the Helm version 2.11 didn’t allow that. To achieve this, you can combine Helm with other tools like Jsonnet and tie those together.
While combining different tools, I found following Helm advanced features quite useful which greatly simplified and provide some structure to the resulting Helm template. Following list enumerates those which I found quite useful.

Useful Helm advanced features

Template nesting (named templates)

Sub-templates can be structured in helper files starting with an underscore, e.g. `_configuration.tpl` as files starting with underscore doesn’t need to contain valid kubernetes manifest.

{{- define "template.to.include" -}}
The content of the template
{{- end -}}

To use the sub-template, you can use

{{ include "template.to.include" .  -}}

Where “.” passes the actual context. When there are problems with the context trick with $ will solve the issue.

Include file

To include a file all you need is specify a path

{{ $.Files.Get "application.conf" -}}

Embed file

Embedding file as configuration

{{ (.Files.Glob "application.conf").AsConfig  }}

it generates key-value automatically which is great when declaring a configMap

Validate input and fail quickly

To provide a reasonable error message and make some config values mandatory, use required function

{{ required "Error message if value not specified" .Values.component.port }}

Escaping dots

Accessing a key from value file which contains a dot e.g. application.conf​

{{- index .Values.configuration "application.conf" }}

Combined logical expressions

IF statements combining multiple values

{{- if or (.Values.boolean1) (.Values.boolean2) (.Values.boolean3) }}

Wrapping reference to value into braces solves the issue.

Definitevly I didin’t list all features but those I see as a moving a code and structure to the next level. Do you fell that I have missed some? Please let me know here in the comment section or reach me on twitter.

Some time ago I wrote a post about tools I found useful when working in a distributed team. Working habits are changing and companies which grasp it will benefit from access to the broader talent pool and possible benefits when managed rightly. However, managing remote teams pose significant challenges. This post isn’t going to discuss how to structure teams or how to address araising challenges but rather focusing on things I found useful when working in a remote distributed team.

In the past years, Slack become defacto standard in this area even though I keep hearing from various teams that it didn’t meet quite their needs. If you have experience with the different tool I will be happy if you can share your experience in the comment section below the post.

Following section lists features I discovered really useful during working in a remote distributed development team.

1. Snooze notifications –  to protect your focus time and boost your productivity. Break day to focus units where all distraction is disabled.
2. Highlight words – can save your time to skim through unread messages available via Slack > Preferences > Notifications > My Keywords
3. Reminder tools – on each message you can set a reminder
4. To quickly start google hangout meeting – use command /hangout and the call will be initiated
5. If you want to append ¯\_(ツ)_/¯ to channel – type /shrug command
6. Use advanced slack search – start in search box with plus sign
7. Create Poll – use the command: /poll “Question?” “Option1” “Option2” “Option3” …
8. Convert direct communication to a private channel. To direct communication must be included more than one party involved and the resulted channel is always private with no option to make it public. You can, later on, invite new participants with the option to see the channel history.
9. Channels for dedicated groups or around topics of interest are standard but what I found really useful is User groups feature which allows communicating with a group of people across the channels.
10. Use stars to create an instant to-do list. You can mark any message or phrase with a star next to the time-stamp. It’s almost the same as bookmarking. Hint: if you mark messages with the task then you’ll get a personal to-do list which is really convenient to form during the conversation.

This concludes my top 10 slack features I found really useful while working in a remote distributed team. If you have other tips and tricks please let me know. Also if you found other useful tools please also share that experience in the comment section below.

JVM Internals

HotSpot Java jvm internals

hotspot jit

class IncWhile {

  def main(): Int = {
    var i: Int = 0
    var limit = 0

    while (limit < 1000000000) {
      i = inc(i)
      limit = limit + 1
    }
    i
  }

  def inc(i: Int): Int = i + 1
}

[info] Result "com.jaksky.jvm.tests.jit.IncWhile.main":
[info] 2112778741.540 ±(99.9%) 9778298.985 ns/op [Average]
[info] (min, avg, max) = (2040573480.000, 2112778741.540, 2192003946.000), stdev = 28831537.237
[info] CI (99.9%): [2103000442.555, 2122557040.525] (assumes normal distribution)
[info] # Run complete. Total time: 00:08:03
[info] Benchmark Mode Cnt Score Error Units
[info] IncWhile.main avgt 100 2112778741.540 ± 9778298.985 ns/op

When inlining enabled JVM JIT also capable to use next optimizations like loop optimizations which might case that our whole loop is eliminated as it is easily predictable. We would get time around 3 ns which are for 1GHz processor unreal to perform billions of operations. To disable most of loop optimizations use -XX:LoopOptsCount=0 JVM option.

[info] Result "com.jaksky.jvm.tests.jit.IncWhile.main":
[info] 332699064.778 ±(99.9%) 3485503.823 ns/op [Average]
[info] (min, avg, max) = (316312877.000, 332699064.778, 358738827.000), stdev = 10277087.396
[info] CI (99.9%): [329213560.955, 336184568.600] (assumes normal distribution)
[info] # Run complete. Total time: 00:04:55
[info] Benchmark Mode Cnt Score Error Units
[info] IncWhile.main avgt 100 332699064.778 ± 3485503.823 ns/op

so the performance gain by inlining a method body can be quite significant 2 seconds vs 300 milliseconds.

Conclusion
In this post, we discussed the mechanics of Java JIT compilation and some optimisation techniques used. We particularly focused on the one of the simplest optimisation technique called method inlining. We demonstrated performance gain brought by eliminating a method call represented by invokevirtual bytecode instruction. Scala also offers a special annotation @inline which should help us with performance aspects of the code under the development. All the code for running the experiments is available online on my GitHub account.
Share this:
Tweet
Reddit
Pocket
Like Loading...
					

Java-based applications run in Java Runtime Environment (JRE) which consists of a set of Java APIs and Java Virtual Machine (JVM). The JVM loads an application via class loaders and runs it by the execution engine.

JVM design principles

In this post dedicated to Big Data I would like to summarize hadoop file formats and provide some brief introduction to this topic. As things are constantly evolving especially in the big data area I will be glad for comments in case I missed something important. Big Data framework changes but InputFormat and OutputFormat stay the same. Doesn’t matter what’s big data technology is in use, can be hadoop, spark or …

File formats basic concepts

Let’s start with some basic terminology and general principles. The key term in mapreduce paradigm is split which defines a chunk of the data processed by single map. Split is further divided into the record where every record is represented as a key-value pair. That is what you actually know from mapper API as your input. The number of splits gives you essentially the number of map tasks necessary to process the data which is not in clash with the number of defined mappers for your mapreduce slots. This just means that some map tasks need to wait until the map slot is available for processing. This abstraction is hidden in IO layer particularly InputFormat or OutputFormat class which contains RecordReader, RecordWriter class responsible for further division into records. Hadoop comes with a bunch of pre-defined file formats classes e.g. TextInputFormat, DBInputFormat, CombinedInputFormat and many others. Needless to say that there is nothing which prevents you from coming with your custom file formats.

Described abstraction model is closely related to mapreduce paradigm but what is the relation to underlying storage like HDFS? First of all, mapreduce and distributed file system (DFS) are two core hadoop concepts  which are “independent” and the relation is defined just through the API between those components. The well-known DFS implementation is HDFS but there are several other possibilities(s3, azure blob, …). DFS is constructed for large datasets. The core concept in DFS is a block which represents a basic unit of the original dataset for a manipulation and processing e.g. replication etc. This fact puts also additional requirements on dataset file format: it has to be splittable – that means that you can process a given block independently from the rest of the dataset. If the file format is not splittable and you would run a mapreduce job you wouldn’t get any level of parallelism and the dataset would be processed by a single mapper. Splittability requirement also applies if the compression is desired as well.

What is the relation between a block from DFS and split from mapreduce? Both of them are essentially key abstractions for parallelization but just in different frameworks and in ideal case they are aligned. If they are perfectly aligned that hadoop can take full advantage of so-called data locality feature which runs the map or reduce tasks on a cluster node where the data resides and minimize the additional network traffic. In case of imprecise alignment, a remote reads will happen for records missing for a given split. For that reasons file formats includes sync markers or points.

To take an advantage and full power of hadoop you design your system for big files. Typically the DFS block size is 64MB but can be bigger. That means that biggest Hadoop enemy is a small file. The number of files which lives in DFS is somehow limited by the size of Name Node memory. All the datasets metadata are kept in memory. Hadoop offers several strategies how to avoid of this bad scenario. Let’s go through those file formats.

Hadoop HDFS data file formats

Hadoop archive (HAR)

HAR file (stands for Hadoop archive) – is a specific file format which essentially packs a bunch of files into a single logical unit which is kept on name node. HAR files don’t support additional compression and as far as I know are transparent to mapreduce. Can help if name nodes are running out of memory.

Sequence file

Sequence file is a kind of file-based data structure. This file format is splittable as it contains a sync point after several records. The record consists of key – value and metadata. Where key and value is serialized via class whose name is kept in the metadata. Classes used for serialization needs to be on CLASSPATH.

MAp file

Map file is again a kind of hadoop file based data structure and it differs from a sequence file in a matter of the order. Map file is sorted and you can perform a look up. Behavior pretty similar to java.util.Map class.

AVRO data file

Avro data file is based on avro serialization framework which was primarily created for Hadoop. It is a splittable file format with a metadata section at the beginning and then a sequence of Avro serialized objects. Metadata section contains a schema for an Avro serialization. This format allows a comparison of data without deserialization.

Google Protocol buffers

Google Protocol buffers are not natively supported by Hadoop but you can plug the support via libraries as elephant-bird from twitter.

Text files

So what about file formats as XML and JSON? They are not natively splittable and so “hard” to deal.  A common practice is to store then into text file a single message per line.

For textual files needless to say that those files are the first class citizens in Hadoop. TextInputFormat and TextOutputFormat deal with those. Byte offset is used as key and the value is the line content.

Conclusion

This blog post just scratches the surface of Hadoop file formats but I hope that it provides a good introduction and explain the connection between two essential concepts – mapreduce and DFS. For the further reference book, Hadoop Definite guide goes into the great detail.

RPM vs jar

Java applications archives as jar, war and ear files are elementary distribution blocks in the java world. At the beginning managing all of these libraries and components were a bit cumbersome and error prone as the project dependencies depends on another libraries and all those transitive dependencies creates so called dependency hell. In order to ease developers of this burden Apache Maven (maven like tools) were developed. Every artefact has so called coordinates which uniquely identifies it and all dependencies are driven by those coordinates in a recursive fashion.

Maven ease the management at the stage of the artefact development but doesn’t help that much when we want to deploy the application component. Many times that’s not such a big deal if your run time environment is clustered J2EE aplication server e.g. Weblogic cluster. You hand the ear or war over to your ops team and they deploy it to the cluster via cluster management console to all nodes at once. They need to maintain an archive of deployed components in case of roll back etc. This is the simplest case (isolated component and doesn’t solve dependencies e.g. libraries provided in the cluster etc.) where management is relatively clean but relying on the process a lot. When we consider different run time environment like run the application as a server less java process (opposed to J2EE cluster) then the stuff gets a bit more complicated even for a simplest case. Your java applications are typically distributed as a jar file and you need to distribute it to every single linux server where instance of this process is running. Apart of that standard jar file doesn’t contain dependencies. One possible solution to that would be to create shaded (fat) jar file which has all dependencies embedded. I suppose that you have a repository where all builds are archived. Does it make sense to store those big archives where the major part are 3rd party libraries? This is probably not the right way to go.

Another aspect of roll out process is ability to automate it. In case of J2EE clusters like weblogic there is often scripting tool provided (WLST ~ weblogic scripting tool). The land of pure jar is again a lot worse. You can take some advantage of maven but that doesn’t solve all the problems. Majority of production environments in java world run on linux operating system so why not to try to take advantage of linux server standard distribution management like yum, apt etc. for distributing rpm linux packages. This system provides atomicity, dependency management between rpm linux packages, easy way to roll back (keeps track of versions), minimise the number of manual steps – potential of human error is reduced and involves native auditing. It is pretty easy to get an info about installation history.

Pack java application to RPM

rpmbuild tool introduction

To pack your java jar file application you need a tool called rpmbuild which creates a linux package from a SPEC file. SPEC file is something like pom in maven world plus it contains instruction how to install, uninstall etc. Packages containing a required plus handy tools are rpmdevtool and rpmlint. On linux OS it is simple to install it. On Windows OS you need a cygwin installed with the same tool set. In order to build your rpm work space run the following command. It is highly recommended not to run it under root account if there is no special need for it.

rpmdev-setuptree

This command creates rpmbuild folder – that is the place where all linux RPM packaging will happen. It contains sub-folders: BUILD, RPMS, SOURCES, SPECS, SRPMS. For us the important ones are RPMS – will contain final linux rpm and SPECS – this is the place we need to put our SPEC file describing installation and content of our application.
Create spec file
This file is the core of the linux rpm packaging. It contains all the information about version, dependencies, installation, un-installation, upgrade etc. We can create our skeleton SPEC file by running following command:
rpmdev-newspec

Majority of directives in this file are clear from its name, e.g. Name, Version, Summary, BuildArch etc. BuildRoot require special attention. It is a sort of proxy which mimics a root of system under the construction e.g. If I want to install my [application_name] (replace this place holder with actual name) to /usr/local/[application_name] location I have to create this structure under BuildRoot during the installation. Then there are important sections which corresponds to various phases of installation: %prep, %build, %install – which is the most important for as as we do not build from sources but just pack already built jar file to linux rpm package. Last very important section of this file is %files which lists all files which will be in the final linux rpm package and hence installed in the target machine. Apart from that there can be additional hooks to installation and un-installation process as %post, %preun, %postun etc. which allows you to customize a process as you need. Sample SPEC file follows:
%define _tmppath /home/virtual/rpmbuild/tmp
Name: [application_name]
Version: 1.0.2
Release: 1%{?dist}
Summary: Processor component which feed data into DB
Group: Applications/System
License: GPL
URL: https://jaksky.wordpress.com/
BuildRoot: %{_topdir}/%{name}-%{version}-%{release}-root
BuildArch: noarch
Requires: jdk >= 7
%description
Component which process incoming messages and store them to DB.

%prep

%build

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/usr/local
cp -r %{_tmppath}/[application_name] $RPM_BUILD_ROOT/usr/local
mkdir -p $RPM_BUILD_ROOT/usr/local/[application_name]/logs
mkdir -p $RPM_BUILD_ROOT/etc/init.d
cp -r %{_tmppath}/[application_name]/bin/[application_name] $RPM_BUILD_ROOT/etc/init.d
mkdir -p $RPM_BUILD_ROOT/var/run/[java application]

%files
%defattr(644,[application_name],[application_name])
%dir %attr(755, [application_name],[application_name])/usr/local/[application_name]
%dir %attr(755,[application_name],[application_name]) /usr/local/[application_name]/lib
/usr/local/[application_name]/lib/*
%attr(755,[application_name],[application_name]) /usr/local/[application_name]/logs
%dir %attr(755,[[application_name],[application_name]) /usr/local/[application_name]/conf
%config /usr/local/[application_name]/conf/[application_name]-config.xml
%config /usr/local/[application_name]/conf/log4j.properties
%dir %attr(755,[application_name],[application_name]) /usr/local/[application_name]/deploy
/usr/local/[application_name]/deploy/*
%doc /usr/local/[application_name]/README.txt
%dir %attr(755,[application_name],[application_name]) /usr/local/[application_name]/bin
%attr(755,[application_name],[application_name]) /usr/local/[application_name]/bin/*
%attr(755,root,root) /etc/init.d/[application_name]
%dir %attr(755,[application_name],[application_name]) /var/run/[application_name]

%changelog
* Wed Nov 13 2013 Jakub Stransky <Jakub.Stransky@jaksky.com> 1.0.2-1
- Bug Fixing wrong messages format
* Wed Nov 13 2013 Jakub Stransky <Jakub.Stransky@jaksky.com> 1.0.1-1
- Bug Fixing wrong messages format
* Mon Nov 11 2013 Jakub Stransky <Jakub.Stransky@jaksky.com> 1.0.0-1
- First relaese of [application_name]

Several things to highlight in the SPEC file example: tmppath points to the location where the installed application is prepared, that is essentially what is going to be packed to rpm package. %defattr set the standard attributes to files if special one are not specified. %config denotes configuration files which means that for the first installation those standard one are provided but in case of upgrade those file will not be overwritten as they are probably customized to this particular instance.
Create PRM package
Now we are ready to create the linux rpm package just the last step is pending:
rpmbuild -v -bb --clean SPECS/nameOfTheSpecFile.spec

TEst RPM package
Created package can be found in RPMS subfolder. We can test the package locally by
rpm -i nameOfTheRpmPackage.rpm

To complete the smoke test lets remove the package by
rpm -e nameOfTheApplication

Creating a SPEC file should be pretty straightforward process and once you create your SPEC file for the application building of linux rpm package is one minute job. But if you want to automate it there is a maven plugin which generates a SPEC file for you. It is essentially wrapper of rpmbuild utility which means that plugin works fine on linux with tool set installed but on windows machine you need have cygwin installed and create wrapper bat file to mimic rpmbuild utility for the plugin. Detailed manual can be found for example here.
Couple things to highlight when creating a SPEC file. Prepare the linux package for all scenarios – install, remove, upgrade and configuration management right from the beginning. Test it properly. It can save you a lot of troubles and manual work in case of large installations. Creating a new version of java application is only about about replacing jar file, re-packaging rpm bundle.
Conclusion
In this quick walk through I tried to show that creating of linux rpm package as a unit for software deployment of the java application is not that difficult and can neaten a roll out process. I just scratch the surface of linux rpm packaging and I was far away from showing all capabilities of this approach. I will conclude this post by several links which I found really useful.
If you have some tips or wan’t share your experience please find the comment section bellow or reach me on twitter
References
Great tutorial on RPM packaging in general

Good rpmbuld manual pages

Maven rpm plugin

Maximum RPM book
Share this:
Tweet
Reddit
Pocket
Like Loading...
					

Desire for high availability

Scalability, Availability, Resilience – those are just common examples of computer system requirements which forms an overall application architecture very strongly and have a direct impact to “indicators” such as Customer Satisfaction Ratio, Revenue, Cost, etc. The weakest part of the system has the major impact on those parameters. The topic of this post availability is defined as the percentage of time that a system is capable of serving its intended function.

In BigData era Apache Hadoop is a common component of nearly every solution. As the system requirements are shifting from purely batch-oriented systems to near-to-real-time systems this just adds pressure on systems availability. Clearly, if the system in batch mode runs every midnight than 2 hours downtime is not such a big deal as opposed to near-to-real-time systems where result delayed by 10 min is pointless.

I this post I will try to summarize Hadoop high availability strategies as a complete and ready to use solutions I encountered during my research on this topic.

Hadoop 1.x

In Hadoop 1.x the well-known fact is that the Name Node is a single point of failure and as such all high availability strategies try to cope with that – strengthen the weakest part of the system. Just to clarify widely spread myth – Secondary Name Node isn’t a backup or recovery node by nature. It has different tasks than Name Node BUT with some changes, Secondary Name Node can be started in the role of Name Node. But neither this doesn’t work automatically nor that wasn’t the original role for SNN.

High Availablity strategies

High availability strategies can be categorized by the state of standby: Hot/Warm Standby or Cold Standby. This has a direct correlation to failover(start up) time. To give a raw idea(according to doc): Cluster with 1500 nodes with PB capacity – the startup time is close to one hour. Startup consists of two major phases: restoring the metadata and then every node in HDFS cluster need to report block location.

Hadoop 1.x HA strategies

The typical solution for Hadoop 1.x which makes use of NFS and logical group of name nodes. Some resources claim that in case of NFS unavailability the name node process aborts what would effectively stop the cluster. I couldn’t verify that fact in different sources of information but I feel important to mention that. Writing name node metadata to NFS need to be exclusive to a single machine in order to keep metadata consistent. To prevent collisions and possible data corruption a fencing method needs to be defined. Fencing method assures that if the name node isn’t responsive that he is really down. In order to have a real confidence, a sequence of fencing strategies can be defined and they are executed in order. Strategies range from simple ssh call to power supply controlled over the network. This concept is sometimes called shot me in the head. The failover is usually manual but can be automated as well. This strategy works as a cold standby and Hadoop providers typically provide this solution in their High Availability Kits.

Because of the relatively long, start up time of back up name node some companies (e.g. Facebook) developed their own solutions which provide hot or warm standby. Facebook’s solution to this problem is called avatar node. The idea behind is relatively simple: Every node is wrapped to so-called avatar(no change to original code base needed!). Primary avatar name node writes to shared NFS filer. Standby avatar node consists of secondary name node and back up name node. This node continuously reads HDFS transaction logs and keeps feeding those transactions to encapsulated name node which is kept in safe mode which prevents him from performing any active duties. This way all name node metadata are kept hot. Avatar in standby mode performs duties of secondary name node. Data nodes are wrapped to avatar data nodes which send block reports to both primary and standby avatar node. Failover time is about a minute. More information can be found here.

Another attempt to create a Hadoop 1.x hot standby coming to form China Mobile Research Institute is based on running synchronization agents and sync master. This solution brings another question and it seems to me that it isn’t so mature and clear as the previous one. Details can be found here.

Hadoop 2.x HA strategies

An ultimate solution to high availability brings Hadoop 2.x which removes a single point of failure from a different architecture. YARN (Yet Another Resource Negotiator) also called MapReduce 2. And for HDFS there is another concept called Quorum Journal Manager (QJM) which can use NFS or Zookeeper as synchronization and coordination framework. Those architectural changes provide the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby.

Conclusion

This post just scratches the surface of Hadoop High Availability and doesn’t go deep in detail daemon but I hope that it is a good starting point. If someone from the readers is aware of some other possibility I am looking forward to seeing that in the comment section.

Linux service vs J2EE container

Using standard J2EE containers for application deployment is not always suitable option. Time to time you need to run an java application (jar file) as a server less, more light weight linux process. Using standard java -cp …. MainClass is feasible but sooner or latter you will reveal that there is something important missing. Especially if you are supposed to run multiple components in this way. I becomes relly messy and hard to manage pretty soon. On linux system there is a solution which is a lot better – run the component as a linux service.

Linux service controlling process

Lets make is simple and easy to understand. Linux service is essentially a “process” which is driven by init script and has defined API – set of standard commands for management of the underlying linux process. Those linux service commands looks as following(processor represents actual name as defined in init script, see latter):

service processor start
service processor status
service processor stop
service processor restart


That’s a lot simpler, easy to manage and monitor, right? You don’t need to know where particular jar file is located etc. Examples of init scripts can be usually located /etc/init.d/samples or just simply read scripts in /etc/init.d which contains various init scripts for different kinds of linux services already present on the system.
Linux Service wrappers
For java applications there is a bunch of projects which acts as a service wrappers. That enables you to quickly and easily turn jar file to regular linux service as a program daemon. There are wrappers even for windows OS. For some reasons I was directed to use just linux server standard tools so the reminder of this post will be about making the linux service program daemon in a common way via shell scripts.
Startup and shutdown scripts
First of all there is a necessity to create startup and shutdown script with a need to properly manage pid (process id) file accordingly. A good practice is to have a dedicated user to run a particular linux services and have them installed under /usr/local/xxx .

startup script follows:

#!/bin/sh
#
# Script parameters: [Instalation_Foleder]
#
# JAVA_HOME Must point at your Java Development Kit installation.
# Required to run the with the "debug" argument.
#
# JRE_HOME Must point at your Java Runtime installation.
# Defaults to JAVA_HOME if empty. If JRE_HOME and JAVA_HOME
# are both set, JRE_HOME is used.
#
# JAVA_OPTS (Optional) Java runtime options used when any command
# is executed.

# Check the way the script has been called and set current directory as PROCESSOR_HOME
if [ "X$1" = "X" ]
then
  cd .. >/dev/null
  pwd >/dev/null
  PROCESSOR_HOME=$PWD
  SERVICE_INVOKE="no"
else
  PROCESSOR_HOME=$1
  SERVICE_INVOKE="yes"
fi
echo PROCESSOR_HOME set to $PROCESSOR_HOME
# Load confing
source $PROCESSOR_HOME/bin/config.sh
# Check if the invocation is according to configuration [asService | asProcess]
if [ ! "$SERVICE_INVOKE" == "$RUN_AS_SERVICE" ]
then
  echo "ERROR - Invocation is not according to configuration - run as a Lunux Service= $RUN_AS_SERVICE"
  exit 6
fi
# check installation
if [ ! -d "$PROCESSOR_HOME/bin" \
-o ! -f "$PROCESSOR_HOME/bin/config.sh" \
-o ! -d "$PROCESSOR_HOME/conf" \
-o ! -d "$PROCESSOR_HOME/deploy" \
-o ! -d "$PROCESSOR_HOME/lib" \
-o ! -f "$PROCESSOR_HOME/conf/log4j.properties" \
-o ! -f "$PROCESSOR_HOME/deploy/test1-1.0-SNAPSHOT.jar" ];
then
echo
echo ERROR - Installation is not correct!
echo Expected installation package looks:
echo "$PROCESSOR_HOME/bin"
echo "$PROCESSOR_HOME/bin/config.sh"
echo "$PROCESSOR_HOME/conf"
echo "$PROCESSOR_HOME/conf/log4j.properties"
echo "$PROCESSOR_HOME/deploy"
echo "$PROCESSOR_HOME/deploy/test1-1.0-SNAPSHOT.jar"
echo "$PROCESSOR_HOME/lib"
exit 1
fi
# clean up
CLASSPATH=
JAVA_OPTS=
JAVA_PATH=
JAVA_EXEC=

# set JAVA
REQUIRED_JVM_VERSION=1.7
if [ -z "$JAVA_HOME" ];
then
  if [ -z "$JRE_HOME" ];
    then
      echo ERROR - either JAVA_HOME or JRE_HOME is not set!!!
      exit 1
    else
    echo Java JRE used $JRE_HOME
    JAVA_PATH=$JRE_HOME
  fi
else
  echo Java used $JAVA_HOME
  JAVA_PATH=$JAVA_HOME
fi

# set JAVA_EXEC
JAVA_EXEC=$JAVA_PATH/bin/java
#check Java bin
if [ ! -x "$JAVA_EXEC" ];
then
  echo Java binaries not found $JAVA_EXEC
  exit 1
fi
# checkJavaVersion
JVM_VERSION=$("$JAVA_EXEC" -version 2>&1 | awk -F '"' '/version/ {print $2}')
#echo version "$JVM_VERSION"
if [[ "$JVM_VERSION" < "$REQUIRED_JVM_VERSION" ]];
then
  echo ERROR - $JAVA_EXEC doesnt point to propper java version $REQUIRED_JVM_VERSION
  exit 1
fi
# setBDHISTP_MAIN
BDHISTP_MAIN=cz.jaksky.PROCESSOR.PROCESSOR
# setClasspath
CLASSPATH=$PROCESSOR_HOME/deploy/*:$PROCESSOR_HOME/lib/*
# echo Classpath set to: $CLASSPATH

# setJAVA_OPTS
JAVA_OPTS=-Dbdconf=$PROCESSOR_HOME/conf
JAVA_OPTS="$JAVA_OPTS -Dlog4j.configuration=file:$PROCESSOR_HOME/conf/log4j.properties"
#echo JAVA_OPTS set to: $JAVA_OPTS
# This is nasty as in the code there is hardcoded location to actual config file for the process
cd $PROCESSOR_HOME
runProgram() {
echo $JAVA_EXEC $JAVA_OPTS -classpath $CLASSPATH $BDHISTP_MAIN
$JAVA_EXEC $JAVA_OPTS -classpath $CLASSPATH $BDHISTP_MAIN & PROCESS_PID=$!
echo $PROCESS_PID > $PIDDIR/$PID_FILENAME
echo "new application instance started as process $PROCESS_PID"
}

if [ ! -f "$PIDDIR/$PID_FILENAME" ]
then
  echo "I will try to start new process ..."
  runProgram
else
  PID=$(cat $PIDDIR/$PID_FILENAME)
  if ps -p $PID >/dev/null
    then
      echo "WARNING $APP_NAME already running as process $PID"
    else
      echo "process $PID is not running - will try to start a new instance of the application"
      echo " "
      runProgram
  fi
fi
exit 0



shutdown script follows:

#!/bin/sh
# Script usage:
# this script can be invoked either directly in bin folder or from different location with passing information where to locate installation folder
#
# Check the way the script has been called and set current directory as PROCESSOR_HOME
if [ "X$1" = "X" ]
then
  cd .. >/dev/null
  pwd >/dev/null
  PROCESSOR_HOME=$PWD
  SERVICE_INVOKE="no"
else
  PROCESSOR_HOME=$1
  SERVICE_INVOKE="yes"
fi
echo PROCESSOR_HOME set to $PROCESSOR_HOME
# Load confing
source $PROCESSOR_HOME/bin/config.sh
if [ -z "$PIDDIR" ]
then
  echo "ERROR - Installation configuration file config.sh not found at $PROCESSOR_HOME/bin"
  exit 1
fi
# Load confing
source $PROCESSOR_HOME/bin/config.sh
# Check if the invocation is according to configuration [asService | asProcess]
if [ ! "$SERVICE_INVOKE" == "$RUN_AS_SERVICE" ]
then
  echo "ERROR - Invocation is not according to configuration - run as a Lunux Service= $RUN_AS_SERVICE"
  exit 6
fi
if [ -f "$PIDDIR/$PID_FILENAME" ]
then
  PID=$(cat $PIDDIR/$PID_FILENAME)
  kill $PID
  RC=$?
  rm $PIDDIR/$PID_FILENAME
  echo "Application $APP_NAME - process $PID shut down successfull"
  exit $RC
else
  echo "pid file not exist $PIDDIR/$PID_FILENAME, nothing to shut down"
  exit 0
fi


Config script
Those scripts relies on existence of installation configuration shell script – config.sh located in bin folder of installation as follows:

#!/bin/sh
RUN_AS_SERVICE="yes"
APP_NAME="Processor"
APP_LONG_NAME="Processor instance"
PIDDIR="/var/run/processor"
PID_FILENAME="processor.pid"


Startup script creates pid file located /var/run/processor – user under which the installation is running needs to have appropriate privileges.
Init.d script
Finally the init script which needs to be placed into /etc/init.d folder:

 ### BEGIN INIT INFO
# Provides: processor
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: processor daemon
# Description: processor daemon
# This provides example about how to
# write a Init script.
### END INIT INFO
# Config to edit if needed
INSTALL_HOME=/usr/local/Processor
JAVA_HOME=/usr/java/default
SERVICE_USER="processor"
# No modification allowed from here
# Using the lsb functions to perform the operations.
. /lib/lsb/init-functions
#
# If the daemon is not there, then exit.
test -x $INSTALL_HOME/bin/startUp.sh || exit 5
test -x $INSTALL_HOME/bin/shutDown.sh || exit 5
test -x $INSTALL_HOME/bin/config.sh || exit 5
# Load confing
source $INSTALL_HOME/bin/config.sh
export JAVA_HOME
PIDFILE=$PIDDIR/$PID_FILENAME
# Process name ( For display )
NAME=$APP_NAME
CURRENT_USER=`id -nu`
start(){
  echo "Starting $NAME under $SERVICE_USER user..."
  if [ "$CURRENT_USER" == "$SERVICE_USER" ]
  then
    $INSTALL_HOME/bin/startUp.sh $INSTALL_HOME >/dev/null
    RC=$?
  else
    su --preserve-environment --command="$INSTALL_HOME/bin/startUp.sh $INSTALL_HOME >/dev/null" $SERVICE_USER
    RC=$?
  fi
}
stop(){
  echo "Stoping $NAME running under $SERVICE_USER user ..."
  if [ "$CURRENT_USER" == "$SERVICE_USER" ]
  then
    $INSTALL_HOME/bin/shutDown.sh $INSTALL_HOME >/dev/null
    RC=$?
  else
    su --preserve-environment --command="$INSTALL_HOME/bin/shutDown.sh $INSTALL_HOME >/dev/null" $SERVICE_USER
    RC=$?
  fi
}
case $1 in
start)
  start
  exit $RC
;;
stop)
  stop
  exit $RC
;;
restart)
  stop
  start
  exit $RC
;;
status)
  if [ ! -f "$PIDDIR/$PID_FILENAME" ]
  then
    echo "$NAME is NOT RUNNING"
    exit 1
  else
    PID=$(cat $PIDDIR/$PID_FILENAME)
    if ps -p $PID >/dev/null
    then
      echo "$NAME is RUNNING $PID"
      exit 0
    else
      echo "$NAME is NOT RUNNING"
      exit 1
    fi
  fi
;;
*)
# For invalid arguments, print the usage message.
echo "Usage: $0 {start|stop|restart|status}"
exit 2
;;
esac


In the init script there is a need to change to appropriate java apps installation folder JAVA_HOME if not default and SERVICE_USER to user which is supposed to run this service. Service can be started under root account or SERVICE_USER without password specification or any other user with knowledge of credentials.
Conclusion
If you have a production like experience with java service wrappers mentioned at the beginning of the article don’t hesitate and share it! This way it serves the purpose at given situation.
Share this:
Tweet
Reddit
Pocket
Like Loading...
					

BPMS in production environment