Kit Plummer's Blog Feed

Ruote-Stomp Ping Pong

Mon, 20 Jun 2011 00:00:00 CST

Ruote-Stomp Ping Pong

Just returning from some time with the DevOpsDays crew and a bit of an extending “vacation” at the Taos Holy Cross Hospital (after my son’s attempt to fly a mountain bike down the Angel Fire ski resort :)). I’ve been meaning to get this tidbit blogged for a bit, but it just hasn’t happened - seemingly more typical these days. Ah well.

Quickly, here are the players:

Ruote
Stomp
stompserver or ActiveMQ
ruote-stomp
daemon-kit - for generating the simple “workers”

This story began with the AMQP-based Ping Pong exercise as I was looking hard at Ruote for a project needing remote, distributed work units, similar to what is possible to Resque that could be attached to managed workflows.

Ruote itself is deserving of an entire blog post. It is an amazing framework, that captures the need for a simple, lightweight engine that could - do more than basic state machine stuffs - accommodating standard workflow patterns.

Although I think AMQP is definitely up to the task of brokering the work units, I need something that can be run from within a JVM, and is a bit easier to test in isolation. Since the only content that is passed between Ruote’s “engine” and “participants” is textual, and JSON at that I figured the Stomp protocol was a logical choice. Stomp is supported in many messaging brokers (ActiveMQ, RabbitMQ, stompserver). So, I set off to port the AMQP ping pong setup to Stomp.

I shamelessly started with the work Kenneth Kalmer did in the ruote-amqp gem, simply swapping the appropriate AMQP calls for Stomp calls. And, thus ruote-stomp was born. It tests pretty easily with the embedded ruby-based stompserver.

I then needed to get the ping and pong participants to use stomp. The AMQP-based ping pong version uses daemon-kit to build participants that consume work off of queues. So, I then forked Kenneth’s work, and added the stomp capability.

The end result, my Ruote-Stomp Ping Pong experiment, can be found at https://github.com/kitplummer/ruote-stomp-ping-pong. To run it is pretty straightforward:

Clone the ruote-stomp-ping-pong project.
Open 3 terminal windows and cd into each project subdirectory (ping/, pong/, ping-pong-engine/).
Ping

➜  ping bundle

➜  ping bin/ping

Pong

➜  pong bundle

➜  pong bin/pong

Ping-Pong-Engine

➜  ping-pong-engine bundle

➜  ping-pong-engine stompserver &

➜  ping-pong-engine ruby lib/main.rb

In Metrics We Trust

Wed, 04 May 2011 00:00:00 CST

In Metrics We Trust

As I’ve stated in the past Trust is the Cornerstone of DevOps Culture and I think we’re getting closer at the root apparatus for creating trust. I do, now, believe that it is more than just QA, providing assurance of functional and non-functional requirements. The problem, that is becoming clearer to me, is the state of change (I know it is a great oxymoron) that is always in play on both sides of the lifecycle (dev and ops). Not only are requirements a moving target, but so is the solution. Agile has taught us to treat the solution as a dynamic entity that is reacquiring the end-state - based on requirements, technology, and people. So, there’s no way QA (process and people) can keep up with the lot, in an effective way that allows for continuous delivery, or even “more” rapid delivery without some means for providing a feedback loop.

That feedback loop is created by metrics. But, metrics that identify the performance of the applications (UI, internal, external, asynchronous processes, etc.) are well beyond just access and availability. Effective Code Instrumentation is really more than “live” tracking a block of code, or capturing log files. Instrumentation also goes beyond failure cases. Metrics provided by instrumentation need to look beyond just a single application, or a single relationship (front-end to database query) and into the full architecture - all tiers.

DevOps is here to help. By creating DevOps-minded environment for Operations to work directly with Developers, early in the cycle (and each cycle) there’s an opportunity to build better applications/services that include the required instrumentation triggers. QA can be included to define the appropriate test cases, and monitoring patterns that will identify trends (especially those associated with changes to infrastructure, business activities, and general load). Obviously, the actual strategy for which metrics are best suited is part of the problem. The Dev, QA, Operations triad is required to work the strategy from the start, and to continuous evolve the strategy - learning from all angles as the solution matures.

Did you know there’s actually an open standard for instrumenting code? Wonder if anybody actually uses it in their Apache HTTP server runtimes.

I’m sure we’ll be hearing more about it at Devopsdays 2011 - Mountain View

Help us out - and vote on a t-shirt design for the event too:

In Metrics We Trust

could be a really cool t-shirt design. (I think this was actually Patrick Debois’ idea. :))

No QA and No Ops Needed - Yeh Right

Sat, 26 Feb 2011 00:00:00 CST

No QA and No Ops Needed - Yeh Right

This is a great example of NoOps & NoQA.

I can’t remember exactly how, but I was pointed to this (above) video of Dub FX that just totally blew me away. I’m a fan of anyone with exceptional talent…and after having watched it a couple of times I began to envision the parallel between what he’s doing on the street with what modern software devers are doing with webapps (the dev-to-deploy) process. For both the awesome sound and metaphor to “NoQA NoOps” I highly recommend taking the few minutes to watch him.

If you don’t get the same vision that I did, let me break it down real quick. He records a single track (read feature development), then immediately listens to it (read feature QA), then promtes the track loop into his operational environment (live). He (by himself, fulfilling the three cultures of dev, QA, and ops) does this a few times over, continuously deploying new tracks to the song - in realtime.

Cage Rattling

Mike Gualtieri has written a couple great “cage rattling” posts, that have generated an excellent dialog.

NoOps?

The first one, I Don’t Want DevOps. I Want NoOps., is a look at the desire/need for developers to be truly abstracted from their target platforms - to include the need for deep understanding of the hardware constraints.

Mike’s post also pushed on the aspect of deployment, suggesting that cloud-based providers will proved the “Ops” needed, to ops teams who can then pass through to developers. In saying this he’s obviously added a touch of irony. Ops is still needed. Yesterday at DevOpsDay (at SCALE9x) I heard Christophe Louvion say explicitly that he gives his developers a credit card and the “go ahead” to manage their own cloud resources. That’s great for “Dev”and is a critical “Yes” required in the DevOps culture to empower developers to gain performance efficiencies and a better understanding of Ops, as long as they work with Ops to utilize cloud resources in the best way that will optimize the deliver/deployment process in the greater release function.

I’m a huge fan of Heroku - especially how it enables me to quickly deploy, within my existing software engineering workflow by simply ’git push’ing to Heroku’s remote repo that represents my target environment. I’m already committing my to my local (in-work branches) repo, pushing to remote branches and trunk/mainline. It really is nothing to push my “prod” branch to Heroku - simplicity. I’m doing something very similar with this blog, using Github’s Pages (I use Jekyll - a cool static site generator).

NoOps for the “One-Liners”

Ok, here’s the reality. This flow makes great sense for my simple (in the sense of both functional and non-functional/operational requirements), single dever, self-QA’d, environment. This flow might make sense for “hot” app dev-to-production for large-scale infrastructures, where the app will have a short life expectancy and won’t change much. For anything more complex, or that is being produced and managed by a multi-person, multi-organizational environment, forget it. Sure, you could use SCM and a binary repo to manage the steps and promotions - but that will be a struggle where there is a “wares” hierarchy with transitive dependencies and version releases along the way. Note, I’m not trying to imply that the simple use case is any less professional, this is just the difference between manufacturing a skateboard and a car.

NoQA?

Mike continues in another post Want Better Quality? Fire Your QA Team., where here acknowledges a client’s of his identification that getting rid of their QA team actually resulted in higher quality software. Mike notes the reason that it worked was because “Dev” is required to take responsibility for the quality of the “deliverable” and not rely on QA as a crutch. Makes perfect sense, and I’ve seen the same things in many different types of organizations and products.

Testing is…Well, it is Hard!

There are problems with pushing the responsibility for testing forward. Testing is a skill. And the greater the complexity of the product/project, the more complex the testing environment gets. Most developers think testing is an impedance - not discounting it’s importance - just identifying that testing can break creative progress. Even though I have fully swallowed the value of *DD (specifically BDD - it is very easy to end up with a brittle test harness. This is even more of an issue where there are multiple customers - each injecting unique requirements.

Continuous integration isn’t enough anymore; functional testing isn’t enough; measuring and monitoring isn’t enough. Truth is always changing. Wether it is a changing customer requirement, or a version change on an internal service, or a turn-over in a key team role, the proof that is quality assurance is a moving target - a true byproduct of the need for agility.

We do need to make QA a shared responsibility across the DevOps culture, and grow the skills that will allow for realistic accountability. But, at the end of the day, we can’t short-change the complexity and importance of focusing on Quality Assurance tasks.

A Couple of Closing Thoughts

To Ops: give devers and QA testing environment that mirror upper-level staging and production environments.

To Devers: ask for help, but do yourself a favor and dig a little and be prepared to offer some help too. Never demand.

To QA: spread your knowledge of requirements, processes and tools to Dev and Ops, and make an effort to get a deeper understanding of the overall methodologies and practices.

Vagrant is My New Best Friend

Fri, 18 Feb 2011 00:00:00 CST

Vagrant is My New Best Friend

Vagrant is my new best friend. Well, really it is the latest tool I’m adding to my development toolbox - or I should say my DevOps toolbox.

Vagrant is a Ruby-based utility for simple interaction with VirtualBox (Oracle’s open source virtualization product). Rather than posting a tutorial…this is how simple it is:

<code>$ gem install vagrant
$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init
$ vagrant up
$ vagrant ssh</code>

Of course, you have to have already installed VirtualBox. But, this gets you a basic Ubuntu Lucid (32-bit) environment, installed, running and logged in. How cool is that???

This works great for me, as one of my target environments is CentOS 5.5. Yep, you guessed it - theres an available CentOS5.5 box available for downloading/running. I’ve learned the hard way that you have to keep each environment in its own directory, because the VM settings are stored in a file called Vagrantfile.

<code>$ mkdir CentOS5.5 && cd CentOS5.5
$ vagrant box add centos http://dl.dropbox.com/u/15307300/vagrant-0.7-centos-64-base.box
$ vagrant init centos
$ vagrant up
$ vagrant ssh</code>

It’s also possible to configure the VM using Puppet - but again my intent isn’t for this to be a tutorial, so check the Vagrant docs if you’re interested. I do this so I can quickly get Subversion and a few other tools install really quickly.

Trust as the Cornerstone of DevOps Culture

Mon, 31 Jan 2011 00:00:00 CST

Trust as the Cornerstone of DevOps Culture

Patrick Debois’ How to Bootstrap a DevOps Mentality prezo

is so good, I can only wish that I’d have been there. Since, I can not catch the next hop to a Euro DevOps meetup, nor execute time-travel a follow-up blog post is the best I can do.

First, let me caveat with saying that I’m a software engineer, and come from a history of Agile attempts (some successful, and some not-so-much). I can’t help but feel like the current buzz around DevOps is tilted towards the Ops functions. Perhaps this is due to the well-deserved attention tools like Puppet and Chef are getting (and the hype revolving around all-things cloud). Automation in the Ops space is very much needed.

I completely agree with Patrick, DevOps is a cultural issue, and subsequently a behaviorial one. Having spent many years working in realtime, embedded software, I can assure you that the current development-operations flux is no different than the current software-hardware riff in the longer running product/widget space. And yes, respect is the “correct” answer. However, changing culture in most organizations with dev and ops shops is an unrealistic expectation. Every organization regardless of a product or service focus, in widget or web form, will be challenged with respect, responsibility, and accountability issues - by the pure dynamic nature of culture. Again yes, there are exceptions to every rule, and there are a few organizations that can overcome culture issues, but they are the minority (especially as the size/complexity of the organization increases).

Trust is paramount.

I love this: trust = f(character,competence)

As both character and competency are fluid with changing teams, changes in organizational structure, and projects themselves it is very important that there is a way to ensure that the values can be made more concrete, improving over time. How we do this is the million [insert_currency_here, !pesos] question, for the DevOps community to debate.

I’ll go ahead and assert that the pivot point (for balancing the relationship between development and deployment) is testing. Testing is the proof of character and competence, the only practice that can provide concrete values to these variables. And, testing is something that can be proven over time. Proof, is the precursor to truth (read trust). Take safety-critical systems for example (e.g. an automated medicine injection system)…ok, bad example…take for example a mission critcal system like a stock transaction processing system. The development and operational activities aren’t left to culture, or behavior to ensure quality (or rate of deployability). These systems are tested, automatically and otherwise, out the wazoo - including the deployment operation. These tests provide the confidence/proof required to create an organizational culture of trust - not member behavior which is susceptible to too many influences.

Trust is derived from proof.

IMHO, bootstrapping a DevOps culture of trust is solely dependent on testing. And thus, DevOps is indeed a technology problem - assuming that testing must be automated as much as possible in order to be of long-term value.

Even within the software engineering microcosm the same issues of trust exist. Agile practices, as stated in my favorite book on the subject, push for respect in the form of humans and technology. Source code management tools enable Agile practices like “commit early, commit often” and continuous integration, which lead to transparency and respect, and thus trust later. Without source control, trust would never survive developer turnover and external technology influences or organizational culture.

In order to build and grow DevOps cultures we need to be more “inclusive” of software development culture and practices, and extend to and accept the technology discussion. The technology solution needs to go all the way to the front-end (requirements/stories) and include automated quality assurance in the middle. If we don’t end-to-end trust and collaboration will be a pipe dream.

I’ll be speaking on this subject at DevOps Day at SCALE9X if you’d like to engage on the discussion. I’d love any feedback, comments, or WTFs.

Starting Up with MaestroDev Startup

Thu, 27 Jan 2011 00:00:00 CST

Starting Up with MaestroDev Startup

I’ve recently started the next chapter of my working life. I hesitate to call it a career, which seems more appropriate for doctors, teachers, and the like. I suppose I am fortunate, however, to be able to fit work into a path called Software Engineering. For the first time, I’ll be working as a software engineer, for software engineers (and more).

The company I am joining is called MaestroDev and our main product is Maestro. More on Maestro in future posts, but quickly it is an build and private cloud environment that automates the build-test-release-deploy lifecycle from a single web interface, targeting enterprise development and operations shops. How’s that for marketeering?

Aside from the excitement I have for Maestro, I’m very pleased to get to spread the DevOps gospel. In fact I’ll be speaking at DevOps Day in LA on 2/25 as part of the greater Southern California Linux Expo event. To top it off, I get to do all of this in the context of the big wide enterprise, and not just the Department of Defense industrial complex where I spent the majority of my waking hours. This is such a huge opportunity for me that I can only hope to be able to make my mark and add value to it.

I’m also very much looking forward to a new set of peers, meeting all of the people already committed to this space, as well as enlightening all of my current dever friends to the DevOps and modern software engineering ways.

Here’s to it!

Have been toying with how to use QR codes, personally and professionally. This one is just a link to the MaestroDev site, but we’re thinking it would be cool to have it point to a bio page on our corporate site.

Pax Exam and ActiveMQ...Testing OSGi Bundles and JMS

Sat, 18 Dec 2010 00:00:00 CST

Pax Exam and ActiveMQ…Testing OSGi Bundles and JMS

I’ve been working with OSGi, specifically Apache Felix/Karaf, a lot lately. I’m also building a few services that interact via JMS. Being test-minded I was looking for a good way to automate testing of my bundles, possibly even deployed against multiple OSGi frameworks. I found PaxExam and became enamored with its great features for integration testing OSGi bundles, or multiples thereof. Pax Exam provides a great API-based scheme for declaring dependent bundles, and means for fetching them from Maven.

There’s already some great posts on using PaxExam, like this one - so I won’t get too much into that. But here’s how I was able to test a couple of Blueprint -based bundles that require a javax.jms.ConnectionFactory to be injected. I also am using ActiveMQ to run an embedded (within the test harness) JMS broker, and expose a ConnectionFactory to the OSGi service framwork.

First, I create an embedded broker in the “setup” method, and create and register the ConnectionFactory:

Note the disabling of the JMX in the broker - otherwise ActiveMQ will fail to launch because some OSGi framework initiate the same 1099 JMX port.

Use the @Configuration annotation to setup the broker’s required dependencies:

Now, any @Test cases can send messages to bundles, and listen on topics or queues which bundles produce to.

Lastly, I use the @After annotation and to shutdown the broker:

Busted Update...Open Development/Government.

Sun, 14 Nov 2010 00:00:00 CST

Busted Update…Open Development/Government.

Hey OpTucs.

Just wanted to give props to Lucas Taylor for making a small, and needed, updated to the Busted feed service.

https://github.com/kitplummer/Busted/pull/1

The details of the fix aren’t really all that important or exciting here. What is really cool, and worth sharing, is the actual process of collaboration.

This is how I think it went down. At the last Open Tucson shindig, which Lucas and I attended, we discussed a few ideas about how the group can participate in the opening of Tucson’s data sources and promote greater transparency within the local governments. One of the solutions, that actually started during our Great American Hackathon last year, was the retrofitting of the SunTran bus feed into a more usable and accessible format - and has been dubbed Busted (by me). Busted has since turned into an opportunity for further development as a data source for a few different ideas revolving around local-business advertising and mobile applications.

Because Busted is open source Lucas was not only able to access the data feed, but also the code that actually does the raw data transformations. Lucas found a small issue with one of the bus feeds. (In my opinion this is where things get really cool.) Because the code is hosted at http://github.com, a social coding and collaboration site, Lucas was able to “clone” my repository into his own personal environment and make the required changes. The key here is that he was able to do so without me even knowing - or having to gain permission to my repository. Lucas then made the fixes.

Obviously it doesn’t end here. Because most open source development is done, well, in the ope n - there are typical geographic distribution challenges to overcome. The software development world, long familiar with this requirement, has create various tools to help work with distributed code. One of those, and the backbone of Github, is called git. Git is distributed version control system (DVCS) that provides the “cloning” capability that allowed Lucas to copy my repository and work within his own environment. Git also provides an apparatus for distributing the changes that Lucas made, back to me, or anyone else for that matter (who might have also cloned my repository). Git calls this transaction a “pull request” and Github makes this dead simple by generating the request and sending it to me as message (email too). For me to incorporate Lucas’ fix it took a couple steps and voila, we’re now in sync and my repository has been updated and Busted has been made better. It took a single additional step for me to send the new Busted to t he “cloud” and affect the “live” feed.

All of this sounds really complex, or really easy - but let me assure it is closer to the latter. The value of data transparency is only the starting point for improving our governments. Transparency is required all the way out to the organizations and communities who are creating solutions. Open Tucson and all of the other “Government 2.0” organizations need more of this type of collaboration. Thanks to Lucas for the opportunity to highlight the coolness of “openness”.

Kit

[This is a copy of a post made to the Open Tucson Google group’s discussion forum, but it doesn’t have public access yet.]

Busted - Now Feeding JSON

Fri, 15 Oct 2010 00:00:00 CST

Busted - Now Feeding JSON

It’s been a bit…been too busy (yep, broken record :)).

Here’s the service’s site: http://busted.kitplummer.apigee.com

http://busted.kitplummer.apigee.com/beta/route/2.json shows the feed for Route #2: “Cherrybell/Country Club”

The list of all routes can be fetched too: http://busted.kitplummer.apigee.com/beta/routes.json

I’m starting to look adding support for the General Transit Feed Specification and publication to Google’s transit feeds. Stay tuned…

Busted - A Tucson SunTran WebWatch Feed Proxy Service

Tue, 29 Jun 2010 00:00:00 CST

Busted - A Tucson SunTran WebWatch Feed Proxy Service

It’s been a bit…been too busy. Changes coming. Anywho…

Back in December, we kicked off the Open Tucson working group with the aspiration to push Tucson into the “open” limelight with many other forward-leaning municipalities. The city has some cool initiatives, services like SeeClickFix, that are making there way into public use.

SunTran, the city’s mass transit service, has made its bus service information, available in a close-to-realtime view via Google Maps or Microsoft Virtual Earth maps. Even the data is being distributed in map form it would be really cool if SunTran made the raw information available for the opportunity of mobile apps, mashups or any innovative use.

A few of us from the Open Tucson group met with SunTran last week to discuss. While they are willing to help, they are somewhat handcuffed by the system that homes all the data behind the maps. Ian Johnson, from the Moia Group, and I poked around the client-side SunTran WebWatch code a bit and found the data feed that the maps consume. It’s ugly, really ugly - take a peek here.

So, I built a simple Sinatra app to home a proxy service, to repackage the data into a more usable format. I’ve put the service up under the Heroku IaaS/PaaS, hey it’s free. In order to test, and more importantly capture analytics for, the proxy feed I’ve put the proxy behind Apigee. Apigee is a API proxy, that provides Google Analytics-like information about usage. Cool stuff.

Here’s the service’s site: http://busted.kitplummer.apigee.com

There’s some basic instructions there, but I think it is easy to see that http://busted.kitplummer.apigee.com/alpha/route/2 is a bit more developer friendly than the SunTran feed.

We’re talking about getting the Open Tucson group together again on the 8th of July. Check the Open Tucson site for the deets.