Linux Foundation

I just accepted a systems and network administrator position at the Linux Foundation. I will be starting in 2 weeks.

I am thrilled to be able to work full-time on open source again.

CrudMiner: find (some) known-vulnerable software in a web root

A while ago I inherited a large webserver full of user-installed PHP software. As it is nearly always the case, when clients are allowed to install their own software, they never actually bother to keep it patched and updated. I looked for a solution that would help me keep an eye on all the crud that my clients are installing, and either notify me when something is known to be vulnerable, or preferably first nag them for a while, and then notify me if they don't update it.

I couldn't find anything, so I wrote CrudMiner to fill that gap.

https://github.com/mricon/CrudMiner

I need your help, though. The crud.ini file is basically just a drop in the bucket. I need help collecting more information and updating the file with the latest info. Any volunteers? :)

I have submitted it to Fedora for package review, if anyone is interested:
https://bugzilla.redhat.com/show_bug.cgi?id=748446

Pirates (humorous kids story)

This is a children's story I wrote about 3 years ago but never posted on my blog. Enjoy! :)

---

Billy was standing in the doorway with a pillow in his one hand, a swimming noodle in the other, and an excited expression on his face. Kate knew immediately that he had some sort of an idea.
"Hi, Kate!" he said. "Wanna play pirates?"
Kate gave this some thought. Her schedule for the afternoon was quite open.
"Sure," she said, letting him in. "What do we have to do?"
Billy thumped inside and swung around, nearly knocking things down with his swimming noodle.
"Hey, watch it!"
"Sorry," he said, tossing it on the couch.
"What's with the noodle, anyway?" asked Kate.
"This? It's called a 'compass,'" said Billy. "Every self-respecting pirate has one."
"Are you sure that's a compass?" asked Kate doubtfully.
"Oh, yeah."
"What does it do?"
Billy suddenly seemed less sure.
"I think it's supposed to point where you're going," he said.
Kate considered it.
"And you brought a swimming noodle?"
"Well, yeah," he said. "See, when I'm at the pool, it always points wherever I want to go," he explained.
"Oh... Okay, I guess that makes sense," agreed Kate. "Then what's with the pillow?"
"That's easy," said Billy. "Pirates are supposed to 'pillage.'"
"With pillows?"
"Oh yeah. Watch."
He grabbed the pillow with both hands and bonked Kate on the head with it.
"Hey!" Kate brushed the hair out of her face and glared at him. "Well... That's not very... you know... fearsome."
Billy bonked her again, this time right on the nose.
"Ow! Okay, okay, sheesh," she said and stepped back to be safely out of pillaging range. "What else do pirates do?"
Billy tossed the pillow next to the swimming noodle and thought hard.
"I think," he said, "I think we're supposed to wear an eye-patch and a wooden leg."
"That's silly," said Kate. "Why would you wear such things?"
"I think," said Billy, "the eye-patch is for safety."
Kate considered it. Her dad always wore goggles when he worked in the basement shop, and always talked to her about safety.
"That makes sense," she admitted.
"Yeah," said Billy. "That's why they say 'eye' all the time.
"They do?"
"Oh yeah."
Billy screwed up his face in a fierce expression and said in a gruff voice: "Eye, mateys, watch for the eye!"
He coughed furiously and then added in his usual voice: "They also say 'R' a lot and talk about 'curvy dogs.'"
"The what?"
Billy shrugged.
"Some kind of sea critter, I think."
Kate thought about it.
"Like a sea lion?" she suggested.
"I think so," nodded Billy. "Except it's a dog. That's why they need the wooden leg, see? When the curvy dogs attack, they take the leg off and throw it at them, like sticks. Then, when the curvy dogs chase after the wooden legs, the pirates sneak around and steal their treasure."
There was a pause while they both admired the mental image.
"And, if someone disobeys their captain," added Billy, "they get 'tarred and feathered.'"
"The what-and-feathered?"
Billy thought for a moment.
"I'm not sure what 'tarred' means," he admitted after a while.
Kate rubbed her nose.
"I know what a 'leotard' is," she said. "I have to wear one to my dancing lessons. Is that the same thing?"
Billy looked unsure.
"Does it have feathers?" he asked.
"No, but I have a feather boa," said Kate.
Billy was still unconvinced.
"Can I see?"
Kate dug deep into her closet and took out her sparkly green leotard and her purple feather boa.
Billy nodded vigorously.
"Yeah, that's it," he shuddered. "If you don't listen to your captain, he makes you wear this stuff."
He paused, then added with terror in his voice: "Ain't nothing you can do to a pirate that's worse than that."
Kate bit her lip. She wore the leotard every Tuesday and Friday to her dancing practice and did not think it was so bad, actually.
"What else do pirates do?" she asked, putting things back in the closet.
Billy breathed a sigh of relief once the offensive outfit was out of sight.
"They also have to sing 'sea-shanties,'" he said.
"What are 'shanties?'" asked Kate.
"Dunno," shrugged Billy.
Kate thought for a moment.
"I have an Auntie Sheila who likes to sing," she said.
Billy looked unconvinced.
"Is she popular with pirates?" he asked.
"She used to be a famous singer," said Kate. "I heard my dad say once that she was in something called 'The Urchins from Venus.'"
"Was it any good?" asked Billy doubtfully.
"I think so. She sang it at the Broadway music school in New York," said Kate. "Besides, if it has urchins in it, it's probably popular with pirates. Space pirates, anyway," she added.
Billy shook his head, doubtfully. He was pretty sure pirates didn't have much to do with either outer space or aunties from New York music schools.
"Do you know the words to these 'Urchins from Venus?'" he asked.
Kate screwed up her face trying to remember.
"I think I heard it once on the radio," she said. "It goes like 'Urchins, urchins! Emergency!'"
Billy nodded.
"The space urchins were probably attacking their ship."
"That makes sense," said Kate.
They stood quietly for a moment, admiring the mental image.
"Is there anything else pirates do?" asked Kate.
"I think that's it," said Billy. "No, wait, we have to pick pirate names!"
Kate thought for a moment.
"I think I'll be Uhura," she said.
Billy nodded.
"I already picked mine," he said. "I'll be Billy Big Bones."
"Huh?"
"My mom always says that I'm 'big-boned,'" explained Billy. "That's why I'm Billy Big Bones."
"I think you should be called Spock," said Kate.
"No way, that's not a pirate name!"
"Don't argue with me, I'm the captain," said Kate.
"Wait, I thought I was going to be the captain!" cried Billy.
"No, you're not, Spocky."
"No way you get to be the captain! And it's 'Billy Big Bones!' OW!"
He got pillaged squarely on the nose.

When Kate's mom looked in ten minutes later to check on the kids, she found them chasing each-other around the room singing disco music. Kate was brandishing a pillow, while Billy was clutching on to a swimming noodle. He was wearing Kate's sparkly green leotard with a purple feather boa tied loosely around his neck. His face, covered by a pair of large safety goggles, was a bright shade of pink.

The wrong way to ask

A little while ago McGill performed a survey asking whether the administrative staff would consider switching from Microsoft Office to OpenOffice.org. The results of the survey weren't shared with us, but seeing as there has been no movement on that front, I believe they were unfavourable, and it's hard to blame the responders -- switching from one software to another is always painful, especially when it adds to your other duties.

Nevertheless, I think that the responses would have been more positive if the question had been framed slightly differently. Now, I don't have any numbers on how much MS Office license costs or how many users it covers, but let's assume that we have 1000 people working in various administrative offices and that MS Office licenses cost us $200,000 annually (NB: I can be dramatically off here on both numbers).

I betcha if the question was "Would you consider switching from Microsoft Office to OpenOffice.org for a $200.00 annual bonus?" the answers to that survey would have been different. Add to that license costs for Oracle (PostgreSQL), Windows (Linux), MS Exchange (Zarafa), Sungard (OpenERP), etc, and you're talking real money going straight to your employees.

Just a thought.

Update: I'm not naive and this will, eventually, translate into savings for the entire company -- if your concern is that "execs" will never "go for it" if the savings all go to their employees. The workforce is not static and the payout bonuses will only go to existing staff to provide them with real monetary incentive to switch. Newly hired employees will not be eligible for the "platform migration bonus," so, depending on the company's attrition rate, "execs" will start seeing the savings come back into their budgets in only a few years. Look at it this way: in 10 years you can still be paying $200,000 per year to Microsoft, with extra money thrown into the pile every 3-4 years during major upgrade cycles, or you can switch to a free/libre solution, give employees a real monetary incentive to switch, and see these savings come back into your budget after a few years.

Linux Symposium

I'm off to the Linux Symposium (it's back in Ottawa this year). If you're not coming, well, then phooey on you. :)

BTW, I'm presenting a tutorial on web application security. Come see me talk in fake British accent. :)

How to get through to your MP about digital locks in bill C-32

The government is trying again to introduce "Digital Lock" into the Canadian copyright law with bill C-32. Previous attempts to do the same have failed, but there's immense pressure from the US to pass a DMCA-like law, so we will enjoy seeing this issue come back to us over, over, over, and over again, like "B" zombie movie sequels.

If you're interested in learning more, see Michael Geist's blog post on this issue. In this post I just want to offer a suggestion about one of the approaches that may convince your MP that digital locks are a bad idea.

Make it "about the children."

If you've ever been around kids, you know that "washing hands" and "being careful with stuff" is somewhere near the bottom of their priorities. When I buy a DVD, I am sure as heck not going to let my son handle it (well, he's only 18 months now, but the time is rapidly approaching when he's capable of picking a DVD, inserting it into a player, and pressing "play"). When such time comes, my strategy will be to rip the DVD and then either burn a copy, or stick the transcoded file onto our multimedia system with a large hard drive, attached to our living room TV.

Then I will store the DVD I bought somewhere far, far away from his reach for when the burnt copy gets scratched bad enough to be unplayable (probably a couple of weeks), or when orange juice enters the multimedia system.

Now, dear Member of Parliament -- do you really want to make criminals of all parents like me? Proposed bill C-32 makes it illegal to copy DVDs that are protected by digital locks -- even for purposes of making backups or transcoding the media I purchased and own into a different format ("format shifting").

Please don't turn your back on Canada's parents and children -- remove the digital lock restrictions from bill C-32.

Please write your MP. If you don't know who they are, find their mailing address here. Use your own words, and let your kids sign the letter. Let your message "hit where it matters most."

My Wii makes me feel sad

We got a Wii last week -- mostly because we heard a lot of good things about Wii Fit and it was cheaper than getting a gym membership. The Wii Fit part works awesome, and the two standard Sports games that came bundled are quite fun too. However, I find that I'm getting increasingly frustrated at how Nintendo goes out of the way to remove features for the fear that they might -- just might -- be used to pirate games or other content. My setup at home used to consist of two devices -- an old laptop that is used mostly to play Russian cartoons for my son (which I can only get over teh Internets), and a junky cheap DVD player that we got at Futureshop for $40 because the laptop doesn't have a DVD drive and because half of the DVDs we rent these days won't play in computers anyway (thanks, you jerks).

Now that we have the Wii, I have a third device to connect to this setup. Off-the-bat that means that now I have to futz around with RCA plugs when I need to switch between the Wii and the DVD player, since I only have one set of RCA connectors on the TV set. This is because Nintendo went out of their way to make sure that DVDs don't play on the Wii. There are, apparently, 3rd-party tools that you can install semi-legitimately to enable DVD playback, but, reportedly, Nintendo routinely makes a point of breaking that functionality with new updates, so I'm not even going to try.

Yesterday I thought of moving the Russian cartoons off the laptop and onto an SD card to see whether I can play them on the Wii -- so I can remove the ailing laptop out of the setup and put it to well-deserved rest (I've had it since 2002). But no, apparently the only SD-card video playback supported on the Wii is MJPEG+PCM, meaning that a 10-minute cartoon encoded at a barely-passable quality "80" ends up around 600M. If you're not familiar with MJPEG, then it's sufficient to know that each frame of the video is saved as a JPEG image, and if you've ever saved a JPEG image, then you know that "quality 80" means ugly blocky artefacts. Oh, and PCM is the codec used for WAV files -- you know how huge those are.

Anyway, a 32GB SDHC card will hold around 50 ten-minute cartoons (and cost me pretty money). That's not even a third of our cartoon library. This approach of dictating what I can and cannot do with devices I purchased and own is really freaking frustrating. Perhaps it's because I see it from a perspective of a Linux user, who is used to the fact that if my device is physically capable of supporting feature X, then there's probably software out there that already allows me to use it. Not so in the world of proprietary "walled gardens."

Okay, our Wii is a toy. We bought it to be used as a toy. It's annoying that I have to keep three devices when one of them is fully capable of performing the functions of the other two, but it is merely an annoyance -- I can live with it. However, if trends continue to replace general-purpose tools such as laptops with locked-down everything-must-be-approved-by-faceless-suits things like iWhatnots, well, I may end up becoming a Luddite. It has a nice ring to it.

Enjoy your walled gardents. Me, iDoNotWant.

pidgin-sipe

The pidgin-sipe package, which allows connecting to the MS Office Communication Server ("OCS") using pidgin, is feeling lonely and unloved. It's been sitting in the bugzilla review queue for the past 4 months.

Would some kind reviewer put it out of its misery?

Bugzilla entry for pidgin-sipe

Passwords in php scripts

Putting passwords in your php scripts is dangerous for a number of reasons, as for example:

1. the php files are usually world-readable, meaning that anyone with shell access to the server has full access to your db passwords, even if they only have an unprivileged account
   
2. if you are using version control, the passwords are needlessly replicated to all the tree checkouts. If someone's home machine gets compromised, you will have to change all the database passwords as well. Changing database passwords always involves a blip in uptime because changing the password in your db software is not simultaneous with changing it in your web scripts.
   
3. the same applies if an employee in your organization leaves.
4. if for some reason the apache configuration becomes screwed up and your php handler disappears, then the server will output the php code as text, exposing all passwords contained within.
   

Or, to illustrate, let's imagine the following scenario. You have two applications, one you trust (example.com/trusted), and the other not so much (example.com/untrusted). One day there is a remote exploit in the untrusted app that allows remote code execution. This would let an attacker read any files readable by the apache process, and therefore any passwords used by the trusted application would be accessible to the attacker, if they are stored in the application source. As a consequence, the data used by the trusted application is compromised because the attacker has access to the mysql database used by /trusted.

The solution is to store these passwords separately from your main app tree and only readable by root. Doing a simple php include from somewhere outside the web tree will problem #4 above, but it still won't address the other concerns. If you have full admin control of your apache server, you can go further than that.

Apache reads config files as root, so we'll take advantage of that.

1. Create a /etc/httpd/conf.d/passwords.conf
2. Put your password information in the form:
   
   <Directory "/var/www/path/to/your/app">
       # disable phpinfo, to prevent accidental leaks
       php_admin_value disable_functions phpinfo
   
       SetEnv mysql_user_password SecretPassXX
       SetEnv some_other_password AnotherPassXX
   </Directory>
   
   
3. Set permissions on passwords.conf by doing:
   chown root:root passwords.conf
   and
   chmod 0600 passwords.conf
   Now this file is only readable by root.
4. You can now access these passwords from your scripts using:
   $_SERVER['mysql_user_password'].

The benefits of this solution are:

1. The passwords are not accessible outside your application's immediate execution environment
2. The only way an attacker can get these passwords is to execute arbitrary code in the context of your application, which is quite a bit harder than simply being able to output arbitrary files (e.g. via directory traversal vulnerabilities)
3. Underprivileged accounts, including php scripts that are executed in another directory can't get to the passwords

In terms of our example with /trusted and /untrusted, even if /untrusted gets compromised, the attacker would not be able to obtain the passwords used by /trusted (the config file containing the passwords is only readable by root, and SetEnv directives with password values are only set in /trusted location -- unavailable in /untrusted even if the attacker has full access to $_SERVER).

There are caveats with this solution as well. E.g. you have to be careful about doing things like phpinfo() or var_dump($_SERVER) (which is why I include the code to disable the phpinfo() using the php_admin_value disable_functions phpinfo parameter).

I think overall this helps significantly tighten the security of web applications, especially in shared environments.

Different SMTP relay host depending on the location

Okay, after thinking about the problem and brainstorming a bit, I came up with a solution for my "smtp from two locations" problem. The solution is tied to NetworkManager and is actually pretty straightforward (if a bit kludgy).

This is what I wrote and placed in /etc/NetworkManager/dispatcher.d/10-relayhost, and it expects to have a default installation of postfix, with at least one uncommented relayhost= line (doesn't matter what's in the line itself). By default, postfix listens on localhost:25, which is fine for me. I have configured claws-mail to always use localhost:25 as the SMTP server, and it's working just fine, automatically switching my relayhost from mcgill's to my ISP's depending on where I am at the moment. It can probably be improved further, but this is the general idea.

Of course, a better solution would be to have mailhost.mcgill.ca accept mail relaying for authenticated connections, but that's a bit of a pie in the sky at the moment.

Hope this helps someone with a similar problem.

#!/bin/sh

CONF="/etc/postfix/main.cf"
HOME="smtp.teksavvy.ca"
WORK="mailhost.mcgill.ca"

if [ "$2" = "up" ]; then
 if `/sbin/ifconfig | grep -q 'inet addr:192.168.1.'`; then
     # we're at home
     sed -i -e "s/^relayhost=.*/relayhost=${HOME}/g" ${CONF}
 elif `/sbin/ifconfig | grep -q 'inet addr:132.2'`; then
     # we're at work
     sed -i -e "s/^relayhost=.*/relayhost=${WORK}/g" ${CONF}
 else
     # we're elsewhere, unset relayhost and hope for the best
     sed -i -e "s/^relayhost=.*/relayhost=/g" ${CONF}
 fi
 /sbin/service postfix reload
fi