Click here to see my first writeup
Once again I convinced $dayjob to spring for (most of) my trip to LISA13 I did a big write up for management last year, and did several training sessions with other employees as it was very much worth the cost / time / effort in my opinion.
This year was quite a bit further for me to travel, as I am on the west coast, and LISA is on the east coast. Google maps says that it’s 4421 km (2747 miles). I don’t think I can complain about the travel time as there are people I met who came from the UK, Sweden, Germany, Denmark, etc.
Once again I did one day of training and 3 days of the technical sessions, and again I did get all of the training material (which is totally worth the day of training, regardless of what training you take).
My Training
The first course I took was Theodore Ts’o’s “Recovering from Linux Hard Drive Disasters” which, was a very informative course on the history of file systems but I would have preferred some demos of fixing some broken file systems. Enter debugfs and fix some deleted files, or show WHEN it is better to hit no in a fsck, etc. I think that Ted Ts’o is well informed and a great speaker, it was just the history material was a little long, and not really on topic.
The second course I took was Joshua Jensen’s “High-Availability Linux Clustering” which was a pretty good course. My only comments is, I would have liked to see a demo, even if it was a short one.
My session highlights
The Disney animation studio presentation was fantastic, I was very impressed with what they have done with traditional management roles and how they are evolving to something that I think will work a lot better for pretty much all medium to large IT organizations and I do hope that more people will follow their example. Disney Animation studios is now on my list of “cool companies I would like to work for someday”.
The “surprise plenary” session on flamegraphs was also very good. It was a surprise because the scheduled speaker got sick and had to go to hospital (she is okay) so they needed a plenary session and had 1 hour notice. Branden had one hours notice to take a 30 minute talk and turn it into a hour and half talk and he was up till 4 in the morning coding on a new type of graph (Chain graphs) I think he did a really good job.
Dan Kaminsky’s talk “Rethinking Dogma: Musings on the Future of Security” was very good (and thanks to Matt Simmons for the recommendation, I wasn’t sure which talk to go to for that time block).
The closing plenary
The closing plenary session was called “Post Ops: A Non-surgical (personal) tale of fragility and reliability” by Todd Underwood (Google). It was a fairly amusing talk with a lot of interesting points. Todd is offering that sysadmins are going to go the way of the dodo bird, or more specifically that the ops part of devops is going to go away and that everybody should become developers.
There is a part of me that agrees a little but I think there is a rather large disconnect with the industry that is, and what google is doing. I think Todd may have his google glass blinders on and is only seeing things from the google perspective. While I’m sure it’s nice place to live I don’t think it is grounded in reality.
There are people who are using 10 year old software and hardware now, a lot of people. The same is going to be true in 10 years from now. In 10 years there will still be people using windows 7 and 8, and just starting to get a plan together to adopt windows 2023 / windows 18 or whatever it is called. Administration of those devices is going to be the same then as it is now.
I can understand his point, if you think about a small office with 20 employees who are, say real estate agents who have 20 chromebooks or 21 with a spare, everything is wireless and they microsoft office 360 / google docs. I don’t see the need for an IT person. In the last 10 years or perhaps even today there would be a tipping point. Maybe when the office reaches 50 people or 100, an IT person would eventually be hired. Will the tipping point drastically move or be eliminated entirely? People are still going to be terrible with computers and need help. I think helpdesk people are safe at least. :)
Other highlights
The so called “Hallway track” was really great this year. I got to have breakfast with Tom Limoncelli and Tobias Oetiker and out for dinner with Matt Simmons and many others. We had some cool conversations about system administration as a whole and where we think it is going.
Birds of a feather
The birds of a feather this year were very good, although I did find that there were more vendor bofs than there were last year. Not sure if that is good or bad.
Books
Books that were recomended at the conference (That are not on safari bookshelf):
I’ve ordered:
Structure of scientific revolutions
Title Transition
It seems as though we are transitioning from the title “System Administrator” to “Site Reliability Engineer”. I think this is kind of an odd choice. The SRE name has been around for a long time, it’s just now instead of google using it many other companies are switching as well.
The old:
“System” is a broad term, and it fits. Most system administrators I know are in charge of anything that has blinking lights on it, and often many other things that don’t.
“Administrator” is a person who manages (takes charge of). Again I think this makes sense for what we do.
The new:
A site reliability engineer is a odd title to me. “Site” in the modern sense of the word infers a web-site, which can be very complicated, or very simple. In the old sense of the word like a job site, implies you are concerned with the reliability of an entire physical location (job-site).
“Reliability” makes sense, we you want to make sure the website/job-site is reliable.
“Engineer” is somewhat of an issue for me personally, but I think this is more to do with a Canadian upbringing. Where I am from, one doesn’t up and call themselves an engineer without having a degree in engineering. This comes from the Ritual of the Calling where after (some|most) Canadian engineers wear the iron ring. I am not an engineer, and around here people would consider it disingenuous if I called myself a site reliability engineer without having a proper engineering degree.
As well, I’m not just concerned with the reliability of the site, I’m also worried about capacity planning, future proofing, scaling, security, updates, etc.
I wonder how long it will be before someone wants to call me an SRE instead of sysadmin?
Small Infrastructure Bof
I attended the small infrastructure Bof, even though my infrastructure is not really that small. I did the same last year; I find them interesting. One of the topics that came up was:
Is LISA still relevant for the small / medium system administrators?
I think system administration is splitting into groups of complexity. According to me it looks something like this:
< 100 boxes/vms/nodes/devices/whatever you want to call it small shops
< 1,000 medium shops
< 5,000 large shops
< 50,000 very large shops
> 50,000 massive shops
LISA has a tendency to prefer talks from massive shops or contractors that work with massive shops. This makes sense because the massive shops have a tendency to be the ones who are pushing the industry forward. Also, the L in LISA refers to large. However when LISA started in 1986 a large system was what we now consider to be small.
I don’t currently think that we need to split LISA up, but I do think the LISA organizers need to be somewhat mindful to the small shops and what they are gaining from the conference. The company I work for fits somewhere between medium and large shops, and I was in the lisa session: Enterprise Architecture Beyond the Perimeter, and it WAS interesting information but this was more a massive shop problem. I didn’t find the talk relevant to anything I was doing or would be doing in the small / medium term. I could have got up and left the talk, but like I said, it was interesting information.
Random notes
Just some random notes I took:
- Check out ktap
- test gluster 3.4 kvm integration
- debate going to confu in 2014
- watch attack driven defense talk
- get familiar with docker containers
- Play with maltego
- play with qcow2 snapshots
- use qemu-nbd (mounts qemu disks locally)
- look into HPM (hardware platform monitor / hpmstat)
Final Thoughts
I think attending LISA is a fantastic experience; and am looking forward to LISA 2014 in Seattle.