Senior Systems Administrator at the Linux Foundation

Hello, Fedorans!

There is a fairly large project joining the Linux Foundation in the near future and we are looking for a US- or Canada-based senior systems administrator to join our Linux Foundation IT team. We are looking for the following skills:

• Excellent knowledge of RHEL-6
• Familiarity with NetApp appliances
• Good knowledge of networking
• Vlans, iptables, ipv4, ipv6, etc
• HP ProCurve switches
• Juniper routers a plus
• Yum, RPM
• Puppet + Func
• SELinux
• Load balancing using nginx, haproxy, LVS
• OpenVPN
• Git (and gitolite)
• KVM+Qemu+libvirt/virsh
• Postfix+Dovecot
• Apache, PostgreSQL

Perks:

• Work from home
• Receive excellent benefits
• Attend LF conferences in fun places
• Do cool things with cool people
• Feel awesome about your work

If that sounds good, please send your resume to tbd-job@linuxfoundation.org.

My simple IT support rules

1. Don't be an asshole.
2. If someone is being an asshole to you, see #1.
3. Are they upset at something? Identify and fix. Repeat as required.
4. Assholes are infectious and easy to flare. Avoid them, unless #3.

Higher education as investment opportunity

It strikes me that most people in favour of tuition hikes view higher education as a net loss paid by their taxes, rather than as an investment that will bring high dividends in the future. It is my wish that more people approached higher education funding like venture capitalists approach startups -- as an investment rather than as a cost. Let me explain what I mean.

Statistically, 9 8 out of 10 startups will fail, costing venture capitalists millions. However, the 2 that succeed will more than cover the losses on the other 8, with lots of extra profit on top, which is why the VCs continue to do it.

Opponents to free higher education tend to point out how many students have trouble finding jobs after they graduate, especially those who chose to major in humanities. However, if we look at it from the same perspective as venture capitalists, it doesn't matter that many students who receive higher education end up working minimal-wage jobs. We as a society reap our monetary and cultural benefits from those few who do succeed.

Averages can be tricky, but we shouldn't ignore them. On average, people with higher education tend to earn more money. People who earn more money pay more in taxes. A province with more highly educated people will be on average more prosperous than a province with fewer highly educated people. This goes beyond just monetary prosperity -- a lot has to be said about culture and just plain joie-de-vivre. Money doesn't tend to stay for very long in dull, desolate places.

But this isn't to say that we shouldn't be smart about investing our money. It doesn't make sense to invest tens of thousands into someone's doctorate degree just to see them move to some other province or country, leaving us to foot the bill. Similarly, it doesn't make sense for all universities to charge the same tuition fees. Some are world-renowned institutions, while others are humble community colleges. One size doesn't fit all.

By changing how we allocate funds, we can both allow universities to charge tuition based on their needs, and hedge our bets to ensure that the money we spend on higher education brings dividends back into our pockets, instead of the pockets of some other provinces or countries.

So, how do we do it?

The government will provide students with a no-interest, inflation-indexed loan to complete their studies. Upon graduation, the government will establish a simple repayment scheme:

1. If the person resides in the province but has no income, the government simply re-indexes the loan to keep up with inflation.
2. If the person resides in the province and has income, the government repays a part of their loan. The sum repaid reflects the amount paid in provincial taxes.
3. If the person leaves the province, the loan is sold to a commercial bank of their choice. The taxpayers are fully reimbursed and repaying the loan becomes that individual's responsibility.

Simple enough?

Let's say Pierre goes to McGill to become a family doctor:

• McGill charges him $10,000 a year for 5 years of studies, resulting in a $50,000 loan with the government. 
• After graduating, Pierre has trouble finding work for a couple of years, so his loan remains with the government and grows by 1-2% a year, reflecting inflation. 
• In his third year, Pierre finds a job and earns $60,000 in income, paying $10,000 in provincial taxes. That year, the government pays a $2,000 instalment towards the loan. 
• Five years later, Pierre earns $150,000 a year, paying $30,000 in provincial taxes and the government pays a $6,000 instalment towards the loan. 
• A year later, Pierre finds a job in the US and moves to work there. The remainder of his loan is sold to a commercial bank, which pays off the loan to the taxpayers.

As you see, this is a net win to the government (and us, as taxpayers), since not only did we not lose any money on Pierre's education, but earned many times as much in taxes, not to mention benefited greatly from his skills as a practising doctor.

Let's take another example:

• Pauline goes to UQAM to become a teacher. UQAM charges her $8,000 a year for 3 years. Her loan with the government is $24,000. 
• Upon graduation, she finds a job right away, but only earns $35,000 a year for the first few years, paying $4,000 in provincial taxes. The government pays the minimal instalment of $1,000 towards her loan. 
• 10 years later, Pauline earns $60,000 a year and pays $10,000 in taxes. Her loan is reimbursed at $2,000 annually. 
• Pauline never leaves Quebec and her loan is fully paid off in 17 years, except she'd completely forgotten that she had a loan at all, as she's never had to worry about payments. 
• As far as Pauline is concerned, she never paid a dime for going to school (which, of course, she did, via her taxes).

And, let's take a third example:

• Michael goes to Duke university in North Carolina and finishes his physics degree in 4 years, taking out a loan of $150,000 to pay for his studies. 
• He finds out about a "Come work for us, we'll help pay your student loans" program in Quebec and finds a job with Hydro for $90,000 a year, paying $20,000 in provincial taxes. Quebec government gives him a $4,000 annual tax credit towards his student loan payments.

Lastly, to prevent anyone from "perpetually staying in school," we can choose an arbitrary ceiling, such as $80,000, after which no more loans will be issued by the government.

This scheme is simple, transparent, and assures that any amount we spend on higher education is a true investment into the province that stays in the province, all without weighing down our youth with heavy loans at precisely the age when what they need most is a boost.

Puppet eyes

Have you ever worked in an environment where some files are in puppet, but not all of them? E.g. when you aren't starting from scratch but have to "puppetize" an existing infrastructure?

Have you ever had to wonder "wait, is this file in puppet?"

Put puppet-eyes.vim in /root/.vim/plugin (or in /usr/share/vim/vimfiles/plugin if you want to enable this globally) and VIM will display an alert when you are trying to edit a file managed by puppet.

Note: To work, the file definition in puppet must have "checksum = md5".

Project page on github

Restrict the gitolite user with SELinux

One of the things I have been working on is adding SELinux user profiles to all of our non-system users. Most recently, I wrote a custom SELinux role for the gitolite user, to further restrict what it is able to do. As I've had a hard time finding the right resources online, I figured I'll write it up here.

By default, all users are running as unconfined, meaning that things act pretty much as if SELinux was disabled. Included in the SELinux policy are 3 SELinux user profiles:

• staff_u: can sudo
• user_u: can do most things regular users can do
• xguest_u: can run X and some applications, but not get on the network
• guest_u: can't even run X, but can move files around

I'm not going into much detail -- see Fedora SELinux docs for more detailed distinctions between these roles. For the gitolite user, I wanted to put the same restrictions as guest_u, except allow it to transition to the gitosis_t domain (gitolite used to be known as "gitosis," so the policy name stuck).

Let's start by writing a new user policy for our gitolite user. I call it mygitoliteuser_u, and the policy will be in the file mygitoliteuser.te:

policy_module(mygitoliteuser, 1.0.0)

require {
    type system_mail_t;
    type postfix_postdrop_t;
}

role mygitoliteuser_r;

role mygitoliteuser_r types { system_mail_t postfix_postdrop_t };

userdom_restricted_user_template(mygitoliteuser)
gitosis_run(mygitoliteuser_t, mygitoliteuser_r)
gen_user(mygitoliteuser_u, user, mygitoliteuser_r, s0, s0)

A few things going on here:

1. We base it off the userdom_restricted_user_template(), which is what guest_u uses.
2. We allow it to run gitolite via the gitosis_run() interface.
3. We additionally let it send email. Note, that theoretically this should be covered by the mta_role() interface, but it wasn't doing the right thing for me.

To compile and load the policy, run:

make -f /usr/share/selinux/devel/Makefile mygitoliteuser.pp
semodule -i mygitoliteuser.pp

Now set up the contexts for the new mygitoliteuser_u:

cd /etc/selinux/targeted/contexts/users
cat guest_u | sed 's/guest_u/mygitoliteuser_u/g' > mygitoliteuser_u

Now you need to assign this profile to the gitolite user:

usermod -Z mygitoliteuser_u gitolite

Now here is where things get annoying. Once you do this, don't try to run "restorecon /var/lib/gitolite", as this will screw up the labels on everything in that directory and label it as user_home_t. You see, all currently released versions of semanage assume that if a user has a real shell, its home directory needs to be labelled as user_home_t, which is sane reasoning, but doesn't work for things like gitolite user. There is a fix for this behaviour in libsemanage 2.1.5 -- you can set ignoredirs=/var/lib/gitolite in /etc/selinux/semanage.conf, but this is not helpful on RHEL6.

Anyway, the only real solution currently is to set up a cronjob that would make sure that everything in /var/lib/gitolite is labelled as gitosis_var_lib_t. I used puppet for this purpose:

file  { '/var/lib/gitolite':
  seltype => 'gitosis_var_lib_t',
  recurse => true,
}

That's about it. I may as well share my tweaks to the default gitosis policy here:

policy_module(mygitosis, 1.0.0)

require {
  type gitosis_t;
  type gitosis_exec_t;
  type tmp_t;
  type ssh_home_t;
  type bin_t;
  type fs_t;
}

# required by fork
allow gitosis_t gitosis_exec_t:file execute_no_trans;

# used by hooks (usually here-docs)
allow gitosis_t tmp_t:dir { write remove_name add_name };
allow gitosis_t tmp_t:file { write getattr read create unlink open };

# these appear bogus
dontaudit gitosis_t bin_t:file setattr;
dontaudit gitosis_t fs_t:filesystem getattr;

optional_policy(`
  mta_send_mail(gitosis_t)
')

Linux Foundation

I just accepted a systems and network administrator position at the Linux Foundation. I will be starting in 2 weeks.

I am thrilled to be able to work full-time on open source again.

CrudMiner: find (some) known-vulnerable software in a web root

A while ago I inherited a large webserver full of user-installed PHP software. As it is nearly always the case, when clients are allowed to install their own software, they never actually bother to keep it patched and updated. I looked for a solution that would help me keep an eye on all the crud that my clients are installing, and either notify me when something is known to be vulnerable, or preferably first nag them for a while, and then notify me if they don't update it.

I couldn't find anything, so I wrote CrudMiner to fill that gap.

https://github.com/mricon/CrudMiner

I need your help, though. The crud.ini file is basically just a drop in the bucket. I need help collecting more information and updating the file with the latest info. Any volunteers? :)

I have submitted it to Fedora for package review, if anyone is interested:
https://bugzilla.redhat.com/show_bug.cgi?id=748446

Pirates (humorous kids story)

This is a children's story I wrote about 3 years ago but never posted on my blog. Enjoy! :)

---

Billy was standing in the doorway with a pillow in his one hand, a swimming noodle in the other, and an excited expression on his face. Kate knew immediately that he had some sort of an idea.
"Hi, Kate!" he said. "Wanna play pirates?"
Kate gave this some thought. Her schedule for the afternoon was quite open.
"Sure," she said, letting him in. "What do we have to do?"
Billy thumped inside and swung around, nearly knocking things down with his swimming noodle.
"Hey, watch it!"
"Sorry," he said, tossing it on the couch.
"What's with the noodle, anyway?" asked Kate.
"This? It's called a 'compass,'" said Billy. "Every self-respecting pirate has one."
"Are you sure that's a compass?" asked Kate doubtfully.
"Oh, yeah."
"What does it do?"
Billy suddenly seemed less sure.
"I think it's supposed to point where you're going," he said.
Kate considered it.
"And you brought a swimming noodle?"
"Well, yeah," he said. "See, when I'm at the pool, it always points wherever I want to go," he explained.
"Oh... Okay, I guess that makes sense," agreed Kate. "Then what's with the pillow?"
"That's easy," said Billy. "Pirates are supposed to 'pillage.'"
"With pillows?"
"Oh yeah. Watch."
He grabbed the pillow with both hands and bonked Kate on the head with it.
"Hey!" Kate brushed the hair out of her face and glared at him. "Well... That's not very... you know... fearsome."
Billy bonked her again, this time right on the nose.
"Ow! Okay, okay, sheesh," she said and stepped back to be safely out of pillaging range. "What else do pirates do?"
Billy tossed the pillow next to the swimming noodle and thought hard.
"I think," he said, "I think we're supposed to wear an eye-patch and a wooden leg."
"That's silly," said Kate. "Why would you wear such things?"
"I think," said Billy, "the eye-patch is for safety."
Kate considered it. Her dad always wore goggles when he worked in the basement shop, and always talked to her about safety.
"That makes sense," she admitted.
"Yeah," said Billy. "That's why they say 'eye' all the time.
"They do?"
"Oh yeah."
Billy screwed up his face in a fierce expression and said in a gruff voice: "Eye, mateys, watch for the eye!"
He coughed furiously and then added in his usual voice: "They also say 'R' a lot and talk about 'curvy dogs.'"
"The what?"
Billy shrugged.
"Some kind of sea critter, I think."
Kate thought about it.
"Like a sea lion?" she suggested.
"I think so," nodded Billy. "Except it's a dog. That's why they need the wooden leg, see? When the curvy dogs attack, they take the leg off and throw it at them, like sticks. Then, when the curvy dogs chase after the wooden legs, the pirates sneak around and steal their treasure."
There was a pause while they both admired the mental image.
"And, if someone disobeys their captain," added Billy, "they get 'tarred and feathered.'"
"The what-and-feathered?"
Billy thought for a moment.
"I'm not sure what 'tarred' means," he admitted after a while.
Kate rubbed her nose.
"I know what a 'leotard' is," she said. "I have to wear one to my dancing lessons. Is that the same thing?"
Billy looked unsure.
"Does it have feathers?" he asked.
"No, but I have a feather boa," said Kate.
Billy was still unconvinced.
"Can I see?"
Kate dug deep into her closet and took out her sparkly green leotard and her purple feather boa.
Billy nodded vigorously.
"Yeah, that's it," he shuddered. "If you don't listen to your captain, he makes you wear this stuff."
He paused, then added with terror in his voice: "Ain't nothing you can do to a pirate that's worse than that."
Kate bit her lip. She wore the leotard every Tuesday and Friday to her dancing practice and did not think it was so bad, actually.
"What else do pirates do?" she asked, putting things back in the closet.
Billy breathed a sigh of relief once the offensive outfit was out of sight.
"They also have to sing 'sea-shanties,'" he said.
"What are 'shanties?'" asked Kate.
"Dunno," shrugged Billy.
Kate thought for a moment.
"I have an Auntie Sheila who likes to sing," she said.
Billy looked unconvinced.
"Is she popular with pirates?" he asked.
"She used to be a famous singer," said Kate. "I heard my dad say once that she was in something called 'The Urchins from Venus.'"
"Was it any good?" asked Billy doubtfully.
"I think so. She sang it at the Broadway music school in New York," said Kate. "Besides, if it has urchins in it, it's probably popular with pirates. Space pirates, anyway," she added.
Billy shook his head, doubtfully. He was pretty sure pirates didn't have much to do with either outer space or aunties from New York music schools.
"Do you know the words to these 'Urchins from Venus?'" he asked.
Kate screwed up her face trying to remember.
"I think I heard it once on the radio," she said. "It goes like 'Urchins, urchins! Emergency!'"
Billy nodded.
"The space urchins were probably attacking their ship."
"That makes sense," said Kate.
They stood quietly for a moment, admiring the mental image.
"Is there anything else pirates do?" asked Kate.
"I think that's it," said Billy. "No, wait, we have to pick pirate names!"
Kate thought for a moment.
"I think I'll be Uhura," she said.
Billy nodded.
"I already picked mine," he said. "I'll be Billy Big Bones."
"Huh?"
"My mom always says that I'm 'big-boned,'" explained Billy. "That's why I'm Billy Big Bones."
"I think you should be called Spock," said Kate.
"No way, that's not a pirate name!"
"Don't argue with me, I'm the captain," said Kate.
"Wait, I thought I was going to be the captain!" cried Billy.
"No, you're not, Spocky."
"No way you get to be the captain! And it's 'Billy Big Bones!' OW!"
He got pillaged squarely on the nose.

When Kate's mom looked in ten minutes later to check on the kids, she found them chasing each-other around the room singing disco music. Kate was brandishing a pillow, while Billy was clutching on to a swimming noodle. He was wearing Kate's sparkly green leotard with a purple feather boa tied loosely around his neck. His face, covered by a pair of large safety goggles, was a bright shade of pink.

The wrong way to ask

A little while ago McGill performed a survey asking whether the administrative staff would consider switching from Microsoft Office to OpenOffice.org. The results of the survey weren't shared with us, but seeing as there has been no movement on that front, I believe they were unfavourable, and it's hard to blame the responders -- switching from one software to another is always painful, especially when it adds to your other duties.

Nevertheless, I think that the responses would have been more positive if the question had been framed slightly differently. Now, I don't have any numbers on how much MS Office license costs or how many users it covers, but let's assume that we have 1000 people working in various administrative offices and that MS Office licenses cost us $200,000 annually (NB: I can be dramatically off here on both numbers).

I betcha if the question was "Would you consider switching from Microsoft Office to OpenOffice.org for a $200.00 annual bonus?" the answers to that survey would have been different. Add to that license costs for Oracle (PostgreSQL), Windows (Linux), MS Exchange (Zarafa), Sungard (OpenERP), etc, and you're talking real money going straight to your employees.

Just a thought.

Update: I'm not naive and this will, eventually, translate into savings for the entire company -- if your concern is that "execs" will never "go for it" if the savings all go to their employees. The workforce is not static and the payout bonuses will only go to existing staff to provide them with real monetary incentive to switch. Newly hired employees will not be eligible for the "platform migration bonus," so, depending on the company's attrition rate, "execs" will start seeing the savings come back into their budgets in only a few years. Look at it this way: in 10 years you can still be paying $200,000 per year to Microsoft, with extra money thrown into the pile every 3-4 years during major upgrade cycles, or you can switch to a free/libre solution, give employees a real monetary incentive to switch, and see these savings come back into your budget after a few years.

Linux Symposium

I'm off to the Linux Symposium (it's back in Ottawa this year). If you're not coming, well, then phooey on you. :)

BTW, I'm presenting a tutorial on web application security. Come see me talk in fake British accent. :)