Articles

  • Changing systemd services on startup

    Systemd is fast becoming the standard for PID 1 on Linux systems, replacing the ancient init script approach. It takes a bit of time to wrap your head around its concurrent and implicit ordering behaviour, but it does bring some very welcome improvements, with init.d files replaced with much simpler, declarative service definition files, along with integrated logging, eventing and automatic restarting among other things.

    One of the things you can't do any more is start services from a script during startup because systemd evaluates the service dependency tree on launch and starts services accordingly, so modifications made to it afterwards are ignored until systemd is reloaded, which you can't do while it is launching. This broke the ability for my fixed-image servers to change which server-specific services ran on boot (the servers boot from a common image, then a pre-configured service on startup mounts and applies an overlay containing configuration files and commands specific to it). As far as I know there are no hooks in systemd to allow modification before the dependency tree is determined, which is fair enough given my use case definitely isn't a common one.

    Without any hooks, in order for me to get my server-specific services started, I needed to reload systemd after the current run had completed. Luckily, the systemd on my server image has DBus support, meaning systemd will broadcast events during its execution. I wrote up a Python script that is executed by a custom service scheduled early on by systemd (Requires=local-fs-pre.target, After=local-fs-pre.target, Before=basic.target). That script hooks into DBus and waits for systemd's StartupFinished message. When it receives it, it tells systemd to reload, then starts the default target again, causing the service dependency tree to be evaluated and executed once more, this time with the added services. Systemd is smart enough to know which services are already running and won't reload them, and can also stop services that have since been disabled.

    Systemd service definition

    [Unit]
    Description=Local overlay loader
    DefaultDependencies=false
    Requires=local-fs-pre.target
    After=local-fs-pre.target
    Before=basic.target
    
    [Service]
    Type=oneshot
    ExecStart=/root/loadoverlay.sh
    RemainAfterExit=true
    
    [Install]
    WantedBy=local-fs.target

    The /root/loadoverlay.sh script mounts the overlay, executes the overlay script and finally runs the following Python script in the background (i.e. with &). The Python DBus bindings need to be installed for this to work.

    Note that because of the RemainAfterExit=True parameter, this service will not be re-executed when systemd is reloaded, because that flag tells systemd that this service is still considered to be running even though the executed process isn't.

    DBus StartupFinished hook script

    #!/usr/bin/env python
    
    ##
    # Adds a signal handler to the 'startup finished' signal on systemd.
    #
    # We then re-execute the default systemd target so it activates the systemd
    # changes made during the overlay script.
    ##
    
    import dbus
    import gobject
    from dbus.mainloop.glib import DBusGMainLoop
    
    import logging
    import subprocess
    
    logging.basicConfig(
        filename='/var/log/dbus_startup_finished_hook.log',
        level=logging.DEBUG,
        format='%(asctime)s [%(name)s][%(levelname)s] %(message)s'
    )
    
    def startup_finished_handler(firmware, loader, kernel, initrd, userspace, total):
        try:
            logging.info('systemd "startup finished" signal detected. Re-activating default.target...')
    
            logging.info('calling systemctl daemon-reload...')
            p = subprocess.Popen(['/usr/bin/systemctl', 'daemon-reload'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
            logging.info('systemctl daemon-reload completed.')
            if stdout:
                logging.info('systemctl daemon-reload stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl daemon-reload stderr:\n' + stderr)
    
            logging.info('calling systemctl start default.target...')
            p = subprocess.Popen(['/usr/bin/systemctl', 'start', 'default.target'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
            logging.info('systemctl start default.target completed.')
            if stdout:
                logging.info('systemctl start default.target stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl start default.target stderr:\n' + stderr)
    
            # handler completed; quit now
            logging.info('handler finished, quitting event loop.')
            loop.quit()
    
        except Exception as e:
            logging.exception('Exception occurred in startup_finished handler.', e)
    
    # hook up the handler
    dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
    
    bus = dbus.SystemBus()
    
    bus.add_signal_receiver(
        startup_finished_handler,
        dbus_interface='org.freedesktop.systemd1.Manager',
        signal_name='StartupFinished'
    )
    
    # start the event loop
    logging.info('Starting the DBus event loop...')
    loop = gobject.MainLoop()
    loop.run()

     

     
  • Big tech company problems

    For the last few years, I've been working at a large, multi-national software development contractor, developing and integrating solutions for a range of customers.  It was an experience I definitely don't regret, and I learnt a lot about how companies function and how things get done. One of the things I realised was that the things that employees gripe about the most at the larger companies aren't really that different.

    These problems seem to arise as a result of growth, and without a plan to counter them, they often snowball and become ingrained in the company culture. Addressing these issues early on is therefore key to minimising their effects, and may also head off other problems that may develop in the future, particularly problems with employee retention.

    Here's a selection of the problems that I have come across.

    A reactive training policy, i.e. no policy

    In the IT industry, keeping your knowledge and skillset up-to-date is an important part of not only your career, but also your value to the company and hence the company's health and ability to perform.

    Unfortunately, at many companies, training is often considered an ad-hoc perk, with the cost covered by your team/project, at your manager's discretion. This is often compounded by the lack of a guiding technological vision that helps steer employees in the direction which will help the company grow.

    Often this problem manifests itself in the mad scrambles to skill up staff or the rushed recruitment drives to fill gaps in knowledge when commencing new projects. Effectively, the company becomes monkeys for hire instead of being an industry leader.

    Companies should have company-wide training policies, with the cost coming from a separate 'bucket of money'. All employees should be guided throughout their career, training them as necessary to keep them ahead of the curve in line with the company vision, allowing the company to be more productive and grow. All employees must participate to reinforce the idea of continuous improvement, and also to ensure employees can take advantage of new developments without being held back by others.

    Crumbling basic facilities, equipment and services for employees

    I'm not talking about a shiny cafeteria complete with beer on tap or a company gym packed with every machine imaginable. I'm talking about basic facilities, like toilets that don't look worse than public toilets. Or buildings that aren't crawling with locusts on certain days of the year. Or offices that don't have broken air conditioning such that you could walk to the other side of the building and feel the need to put a jacket on.

    I'm also talking about having the right equipment for the job, whether that'd be multiple monitors, a particular piece of software, a level desk, a comfortable chair, a clean keyboard or appropriate privileges so you can do your work efficiently. Nothing extravagant, just the basics so employees can work productively in whatever way they work best.

    Finally, services. Along with outdated or inappropriate equipment, inefficient or deliberately hobbled services are an absolute pain, particularly when you have to use them regularly to carry out your job. Things like a broken timesheet web app - accidentally type in the timesheet code field and all the associated hours are gone. (I've never understood why internal organisation software is always terrible, worse than anything that gets produced for customers. At one company, an employee tried to fix this by coding an alternative implementation, only to be shut down by IT security when they found out.)

    Another commonly hobbled service that infuriates developers is access to the internet. I've seen companies with a wonderfully restricted system where many useful sites were blocked,  HTTP/S were practically the only allowed ports and HTTPS went via a man-in-the-middle (surreptitiously - many continued to do their internet banking without even knowing), while another had a rule that meant all access had to be done through IE.

    The blocking of sites was annoying (one company even blocks StackOverflow; those poor developers), but it was bearable. The stealthy HTTPS MITM is just plain evil, but once you know, you can avoid it (just use your phone instead). The worst was the last point - it meant it was not possible to use package managers, a common best practice. The way IE-only access was enforced was by checking the User-Agent string in the HTTP request, so I spun up a Squid proxy instance to inject the appropriate headers to bypass it. Worked like a charm. Another team, one with more money, just said 'screw it', bought a laptop and a Telstra 4G USB stick, used that to mirror the repository before burning it to a DVD.

    This isn't rocket science. Employees will not be efficient nor productive if the company passively and actively puts up roadblocks. Companies should have developer advocates in IT to ensure measures introduced don't unnecessarily and unreasonably impede developers, and give developers a budget each year to spend on whatever that would make them more productive (within reason).

    Excessive red tape

    TPS reports and red tape are a necessary evil. I actually don't mind it, because I know when I'm trying to sort something out for someone, I like to have all the necessary details and approvals to be able to do it efficiently.

    The problem is when the red tape becomes out of sync with the actual process or the people actioning them. It leads to situations where things that take minutes take days or even months to complete; where people get told to always tick 'no' to a certain box or put a certain bit of information in another box; where you end up pissing off half the org chart above you just to get their approval (ah, those wonderful email chains where half the organisation is CC`ed). Eventually, no one remembers why things are done that way, or worse some processes end up having a 'fast-track' if you happen to know that right people, which leads to a terrible exclusive, cliquey company culture. Ick.

    Companies should empower someone who works with a particular process regularly to change it, with simplification always the goal. Red tape is there to assist the process, not cause it to snowball into some unrecognisable monster.

    A one-size-fits-all, pay-focused performance management system

    If the only way a company can motivate me to stay is to pay me more, then they have already lost. In fact, I'd even consider it a demotivator if that was all that was offered, because it says that the company doesn't really care about my aspirations and complaints; they're happy to just throw me more money until I shut up or leave. It is one of the more expensive retention mechanisms with the least return in my opinion, and can often have a detrimental effect on the company culture.

    I'm not sure what I'd rather see in terms of the actual process itself, but all the ones I've seen are just silly games all employees (managers and developers) hate but play to make HR happy.

    A career ladder that always ends in management

    For some reason, HR seems to assume everyone who wants to progress in their career wants to join management and shoot for the top job. Newsflash - we don't. Besides, who will do the grunt work when everyone is in management? :)

    Companies should have progression paths that allow the employee to choose between entering management or furthering their engineering (or marketing or whatever) expertise, and make them equivalent in terms of pay grade. One should not have to become a manager just to earn more.

    Patronising 'memos' from above

    This is a pet peeve of mine. Maybe it is an American thing, but I hate reading emails about how 'you are our most important resource' (I'm a person damnit, not the office coffee machine), or how 'we would not have been able to do this without you', or 'this has been a tough and challenging year for all of us, and if not for our combined hard work, determination and dedication, we would not have been able to achieve this extraordinary result'. Why can't they just communicate like normal people do?

    Also, stop the stupid /s/ rubbish. This is an email. No one cares about your signature.

    Management who acknowledge issues but don't act

    Ah, employee surveys - the big scam management holds every year or two to make employees feel that they are valued. I filled my first one in meticulously, proof-reading and rewriting the free-text comment box before submitting. Then nothing, except a patronising 'we heard you and will discuss the results soon' email. So disappointed.

    A couple of the companies I worked with even had internal wiki pages making a mockery of the issues they faced (many part of the unofficial induction process). Others had a tag on the internal blog site dedicated to the parting thoughts of employees, with many recurring themes throughout the years.

    If your employees are telling you something is an issue for them, and the same issue is reported repeatedly by others… just maybe it is a problem that should be fixed?

    Silos of knowledge

    This is a problem that affects companies of any size. I saw it when I was working at a small 4-person company, where knowledge was siloed inside each employee, and again at much larger companies, where knowledge was siloed by teams. No one knew who had what skill set outside of the team or who had experience with which customer. Got a question about Perl? Or maybe a question about IT security? Google. Want to know if there's anyone with Puppet expertise? Email those you know and hope for the best.

    Knowledge transfer is important in any organisation, whether it be technical information, or information about the new travel process.  A decent intranet is a good way to solve this. But ultimately, the solution to this problem isn't technical. The solution to the problem lies with management. They need to be committed to the solution and gently force the workforce to use it by leading by example.

    Breaking down the silos also helps bring people from all over the organisation together and builds the company culture. People can choose to find out more about how another project is going, or organise themselves around common after-work interests.

    Sidenote - Email alone is not the solution. At one company, there was no mechanism to communicate between staff besides email. New charge codes for timesheets? Email. Details about the upcoming performance assessment? Email. Coffee van had arrived? Email. (Yes, one every day at 11am.) Does any one know who has the key to the stationary cupboard? Email. Welcome to inbox hell. Sure knowledge is being disseminated, but the mechanism just isn't conducive to making the knowledge accessible when needed.

    A lack of participation with the industry

    Any organisation, particularly an IT organisation, can no longer afford to isolate themselves. The world is now much more connected and integrated than ever before, and isolation will only make the organisation less competitive. Any employee in the IT industry can't afford to isolate themselves either, potentially missing out on future career opportunities, new skills and connections.

    Companies should communicate in plain English through their website and maintain a social media presence. Employees should be encouraged to engage the wider community, whether that be through social media, conferences, meetups, open source projects or something else to not only raise the profile of the company, but also to learn.

     
  • Declutterising

    I am a hoarder. Back at my parents' place, I still have boxes full of every exercise book I've had from prep until I finished university. My bookcase still has a shelf full of Goosebumps books; my desk has towers of music CDs and a significant part of my wardrobe contains things I can no longer fit into.

    All that clutter never bothered me until I moved out of home. Leaving all that clutter behind, I bounced around apartments and townhouses, only bringing things that I needed in the next week or so from my parents' place and making extra trips whenever I needed something.

    Through no plan at all, each place ended up being smaller than the last (the location did get better though, so that probably explains it). During the latest move, everything that I had with me didn't fit so I had to immediately start getting rid of things. It was only then that I realised 90% of the stuff I had back at my parents' place is stuff I don't actually need.

    I made the decision to let go and de-clutter my life. One would think that was the hardest step, but it wasn't. Actually disposing of things was. I didn't want to contribute to landfill if I could help it (yeh, warm and fuzzy feeling), so I set about trying to find new homes for all of my clutter.

    Selling items

    Making money from my clutter wasn't and isn't the primary goal, but hey, if I can make some money from it, why not.

    eBay and Gumtree were the logical starting points for me. Wow. Gumtree I can kind of understand given its roots, but eBay has a pretty horrendous user experience for such a large, well-known and well-used web property. From silly bugs, spammy looking UI elements, and update issues to incorrect error messages and unfriendly placement of elements, it had it all.

    Some more useful things to know about eBay:

    • After selling a few things on eBay and having some buyers try to be tricky, I discovered that eBay has a section called 'buyer requirements' that lets you restrict buyers to only those that meet certain requirements, e.g. have no policy violations or a feedback score of X or more. It is worth having a look at, particular when you sell more expensive items.
    • Know how much eBay will take in fees. Often the listing fee is free, but there is a final value fee which is a percentage of how much the item sold for. Also, if you use PayPal to accept payment, PayPal will take a cut as well.
    • It doesn't happen consistently, but I have found that items sell better when the auction ends at a time when people are free, rather than when most are at work or out. Unfortunately, scheduling an auction requires an extra cost (minimal though), so it is often easier to just list things later in the day (and hence they end at the same time some number of days later).

    Once things started to sell, postage was the next problem. Australia Post is bloody expensive, particularly for small or large packages. Sure we're a big country, but as many online shoppers will know, it is often cheaper to send something from other countries to Australia than it is to send something within Australia. Apparently this is because of the way international mail is charged between countries. (I wonder if local online retailers would be more competitive if our government subsidised the postage...) Because it is an effective monopoly (for good reasons, just not advantageous to me), there really isn't an alternative (courier companies roughly charge the same for small quantities).

    To make things worse, not all post offices seem to know all the rules. For example, if the parcel you're sending happens to be able to fit through the letter guide (e.g. an Xbox 360 game), it can be sent as a large letter, which is much cheaper than a parcel (starting at $6.95). Some however, don't bother checking and just assume it is a parcel. It is sometimes like going to a sandwich shop - you know you're ordering the same thing each time, but somehow, it never costs the same.

    Pick up is an alternative for buyers who live nearby, but unless I have to, I'd rather not have to wait around all day for someone to come. Plus buyers tend to try and convince you to accept cash on pickup, then try to bargain down the price when they see you in person. Ugh.

    All this of course takes time too - time to clean the item if necessary, take photos, type up an (appealing) description, determine a reasonable price, list it somewhere, hope it sells, package it for delivery, lodge it at the post office, write feedback etc. It took more time than I expected (at least 45 minutes per item), but unexpectedly, it is quite rewarding. There's a sense of relief, like a weight lifted off your shoulders when you get rid of something, reducing not only physical clutter, but also that mental clutter in the back of your mind.

    That sense of reward is addictive. It pained me initially to let go of things, even though I hadn't used it for ages. One trick that I found effective was just to keep things that I was undecided about and revisit them in a week or so. I found that over time, maybe after mentally digesting the idea of letting it go, it was easier to make a decision on what to let go and what to keep.

    Although making money was never a goal for me, I was surprised at what people will pay for and how much they will pay. I managed to sell a broken Xbox 360 for $20 excluding postage (it was somewhat fixable I guess) and a couple of working 2TB hard drives for $60 each excluding postage (over 50% of today's purchase price).

    Besides eBay and Gumtree, Cash Converters may also be worth trying.

    Donating things

    For things that probably won't sell or aren't worth the effort, there's always the local charities. Besides clothing, the Brotherhood of St. Lawrence accepts books which they resell on their website. I'm planning on donating a boxful of books next week. They don't take encyclopedias, dictionaries, textbooks or ex-library books though.

    GiveNow.com.au has a neat list of charities and what they accept. It isn't complete though; for example, The Salvation Army is mysteriously missing from the clothing section. Google will also help.

    I was hoping there would be a charity that would take old textbooks, maybe for students in third-world countries, but I haven't been able to find one yet. I suppose the content may be a bit dated, but mathematics or physics hasn't changed that dramatically in the past decade.

    Keeping the clutter down

    After doing this on-and-off for the past few weeks, I'm definitely happy with the stuff I've gotten rid of and haven't regretted it at all. As others have mentioned it is seriously therapeutic. Plus the freedom of knowing you are much more prepared to be able to leave and move elsewhere at the drop of a hat, even if you never go through with it, is pretty refreshing. I haven't gone all out and pared my belongings to a list of 100 things or a backpack, but I'm definitely going to try and keep the clutter down.

    One of the strategies that I have for that is to keep storage spaces to a minimum. I could easily get a few bookcases and chests of drawers or cabinets to maximise the storage in my place, but that just invites hoarding. Plus a perceived lack of storage (there's still plenty of space in the cupboards I already have) allowed me to use the excuse of not having space when friends and family try to give me things (my kitchen would be overflowing with gadgets and pots if I had said yes to everyone).

    Another strategy is to get rid of things sooner rather than later. If there's a computer that I no longer need, getting rid of it sooner means there's a higher chance that someone else would want it, which also means you might be able to get more for it. Worse case, you'll just have to buy something similar back from someone else.

    Others go further with rules about having to get rid of something before being able to buy something else, or have days set aside every so often to de-clutter. I've just decided to be more conscious of clutter whenever I'm about to get something new. This approach also causes me to think twice about buying cheap 'disposable' things, in favour of better quality, longer lasting things too.

    I still have all my exercise books though. One day...

     
  • Being funemployed

    Three years ago, I started working for megacorp, my first job after university (but not my first job in the industry). I was lucky - I had stability, I had job security, I was working on a challenging project with a small team of smart software developers and I had great overseas travel opportunities. It was pretty much all you could ask for in a graduate position.

    Of course, there were some negatives - working for megacorp sometimes felt like scenes right out of Office Space, and the office wasn't in the greatest location, requiring a 40 minute drive each way. The pros outweighed the cons though, so I happily stuck with it.

    The itch

    Fast forward one and a bit years and things started changing. I started feeling the itch again, the itch to do something different, to do what I enjoyed and make everything else work around that. My to-do list started growing and side-projects were started but never finished. With the time I was devoting to work and the associated transport time, I simply did not have the time to do the things I wanted to do.

    Sidenote - don't underestimate the effect transport to and from work has on your lifestyle. This is particularly true if you're driving. Even if it is against peak traffic, with few traffic lights and mostly on a freeway, it is still wasted time. About the only thing you can do is listen to the radio or podcasts (TOFOP/FOFOP, The Little Dum Dum Club, Stack Exchange and Risky Business were the main ones for me; I found startup and tech podcasts, while informative, also reminded me about what I disliked about my career, so I gave them a miss). Catching public transport is slightly better in that you can do other things, but there's only so much you can do with zero personal space during peak hours and or services that are infrequent enough in the off-peak to be very annoying.

    Megacorp was always a stepping stone for me; throughout university I had always dreamt of being involved in startup or a small company and the idea of being free from the constraints of a typical 9-5 job was always at the back of my mind.

    However, having spent the four years prior to megacorp with practically all the time in the world only to end up with not an awful lot to show in terms of career development, I decided I needed some real-world experience first. Megacorp was probably a bit of a leap to the other extreme, but they worked on some pretty cool stuff.

    There were some other factors at play at work, but ultimately, I came to the realisation that I had learnt all the things I wanted to from them and it was time to move on. I could stay, but I knew if I did, I'd just become more and more unhappy and frustrated with myself.

    The realization was easy; committing was the harder part, due to the fear of leaving a safe, stable and predictable environment for the great unknown. I decided that things couldn't go on like this though, so I set a deadline and started telling friends. The accountability was just what I needed, and by the end of last year, the wheels had been set in motion.

    The notification

    My boss had seen it coming and was understanding when I told him. Money was offered, but as nice as that would've been, I couldn't take it, stomach the situation for a few more months before attempting to bail again. Not only did that feel wrong, accepting it was really just me caving into procrastination.

    The notification period was long… eight months long. It didn't need to be anywhere near that long, but I felt it was the right thing to do. The date was chosen because it coincided with the release date for the new version and as I was responsible for a significant feature, it also gave me time to complete it as well as transition my work and knowledge over.

    In hindsight, I realised I must've been a very difficult employee for the company machine to control - money didn't motivate me, I happily worked unpaid overtime, I attempted to cause havoc with their performance management processes by giving monkey answers, I gave an excessive notice of resignation period and I actually told the truth about why I was leaving during my exit interview (I didn't have any issues with any particular person, just the processes). I left on good terms with everyone though.

    It's worth pointing out that I absolutely don't regret working for megacorp. There are things that working for a megacorp teach you first-hand that I don't think you can get elsewhere, things like dealing with processes and hierarchy, how to liaise and work with large customers and how to manage a large project with many legacy components across multiple teams. Plus there was lots of new, shiny, expensive tech to play with.

    The journey so far

    Today, it has been a bit over three months since I had a job. Armed with enough savings to sustain me for a while, I've been slowly working through the things on my todo list and working on a couple of projects ever since.

    One of those projects is close to being complete, at least for version 1. It uses all kinds of fun new stuff like AngularJS and d3.js, things that I've always wanted to learn but didn't have time for and couldn't work into projects at work. I've also started exercising more, going to a gym and learning Spanish for fun. There's no grand plan here; of course, I'd love for one of my projects to become profitable and help finance this lifestyle, but realistically I'm happy to accept that I'll be job-hunting again at some point next year and hopefully end up in something more aligned with my career aspirations and lifestyle.

    The ups and downs

    It hasn't exactly been smooth sailing - being completely in control of your entire day requires discipline, and lots of it. For me, it was extremely easy to let a day or even a week slip by without achieving much at all, yet not feel like I have been wasting time either. I found myself doing all the small tasks instead, then when they were done, thinking of more small tasks to do. As the weeks ticked by, this scared the hell out of me. I had plans with (unrealistic) timelines and I wasn't meeting any of them.

    This sounds like I'm just not interested in my projects, but I am. It is just that it takes a kind of mental preparation to get 'in the zone' and that's something I was conditioned to do at work, but not at home. Back at work, working in a shared quad office, I often used headphones to get 'in the zone'. On some days I can get there without, but as soon as anyone mentions anything mildly interesting like 'donuts!', the zone disappears. In the past few weeks, I've started wearing headphones at home to emulate this, and as silly as it sounds, it works. Even though there's barely any background noise at home, listening to music through speakers just doesn't have the same effect as listening through headphones.

    The plans were changed as well to include attainable tasks and I reminded myself that this gap year is as much of a learning experience as it is a productive one. Yes the projects were important, but so is being able to enjoy this time, reflect on how things are going and broaden my experiences. I didn't want my projects to feel like work, taking me back to square one, except without the income.

    Explaining what I'm doing to others has been an interesting exercise too. Some understand completely, others are completely baffled, with the usual question being, 'don't you get bored not having anything to do all day?' The answer is a resounding no. I can honestly say there hasn't been a period of time during the last 3 months when I can say I was bored. Maybe it is due to the career I've chosen, but I always have a project on my mind that I want to explore. Besides that, I've always got books to read, TV shows to watch, places to visit, travel plans to research and more. In fact I'd say I might even have a problem of trying to do too many things at once and not seeing things through to completion.

    Catching up with people is important though, if only just to keep yourself sane. It can be easy to not converse with anyone face-to-face, but I find that demotivating and makes me less productive. Sometimes meeting with someone is enough to re-motivate and re-focus me to work on whatever or help me look at a problem from another direction. My general rule of thumb is, if I don't know what day of the week it is, I need to head outside. (I also now understand the appeal of working in cafes. Unfortunately, I don't have a MacBook, or even a functional laptop at the moment, so that'll have to wait.)

    There are plenty of positive side-effects as well, besides being able to work on the things I love. It's amazing how much I have improved my diet by simply having my fridge, my pantry and my kitchen around during lunch instead of the same three fast-food places back where I used to work (never got organised enough to bring lunch to work). Not having a strict 9-12-5 schedule also means I can spontaneously meet up with people; I can choose to not have lunch around 12pm (and not be left with the cafe leftovers when I eventually do have lunch), in fact I can choose to eat whenever, however frequently I want; and I can choose to go out for a run at 2pm and return to work after. Being in a much more central location also means I can now visit the bank or other service outlets without having to take a morning off. Little things matter.

    The future

    Just doing the things you love doing isn't as easy as it sounds, even if you take the financial concerns away. In many ways, I'm finding it harder than working for megacorp - the lack of certainty and structure still wrecks havoc with my mind. It is definitely more rewarding though, and should be even more when I get this current project out there.

    I've got a list of ideas that's slowly growing, other languages that I want to learn, plus I'm hoping to squeeze some travel in at some point, so the next few months are going to be pretty exciting.

    I've been pretty lax with my website here for the last few years, but I do intend to keep it up to date with the various things I get up to.

    (Yep, this post was inspired by a similar one here.)

     
  • Run SuSE Studio images without persistence for OpenSuSE 12.3

    A while ago, I wrote an article about booting OpenSuSE images (specifically SuSE Studio images) without persistence, without having to dedicate a device/partition to it. If you follow the OpenSuSE documentation, it will direct you to wipe and replace a device/partition with the ISO image using an ISO writing tool. That isn't ideal because you can't use the device for anything else (without partitioning it at least).

    Unfortunately, changes in the booting process put a stop to the trick described in the previous article. Fortunately however, I have found a new way to do it, and it is cleaner and simpler than the previous method. It will also work with machines that have a CDROM installed as well (this was a flaw in the old method as it tried to act as a CDROM fallback). I have only tested this with OpenSuSE 12.3, but it should work with other versions in the OpenSuSE 12.x series.

    1. Ensure your device is formatted in one of the standard Linux filesystems (btrfs, ext* etc.) or FAT32. I haven't tried NTFS, but I'm pretty sure the boot image can't read that and so this wouldn't work.
    2. Ensure your device has a label assigned to it. On Windows, right-click on the device, click Properties and check the text box at the top of the dialog. On Linux, see this article for a list of tools to use to modify it.
    3. Copy the ISO file to the root of your device.
    4. Mount the ISO file or otherwise extract the boot directory from it on to the root of your device, then unmount the ISO file.
    5. On your device, in the boot/<arch>/loader directory (i.e. if it is a 64-bit image, it will be boot/x86_64/loader; if it is a 32-bit image, it will be boot/i386/loader), make a copy of the isolinux.cfg file and name it syslinux.cfg.
    6. Open up syslinux.cfg (on Windows, you'll have to use an editor that is aware of Unix file endings; try notepad++ if you don't have such an editor), and on the end of both lines beginning with append, add the following (substituting as required) then save and close the file -
      OpenSuSE 12.3: isofrom=/dev/disk/by-label/<the label assigned in step 2>:<the path to the ISO file on your device>
      
      OpenSuSE 13.1: isofrom_device=/dev/disk/by-label/<the label assigned in step 2> isofrom_system=<the path to the ISO file on your device>
      For example, if your device is labelled LINUXLIVE and your OpenSuSE 12.3 ISO is named My_Live_Linux_Desktop_12.3.x86_64-0.0.1.iso, then that line will be -
      isofrom=/dev/disk/by-label/LINUXLIVE:My_Live_Linux_Desktop_12.3.x86_64-0.0.1.iso
    7. If you're on Windows, download SYSLINUX 4.04 from here and extract win32/syslinux.exe to the root of your device. If you're on Linux, make sure you have the syslinux 4.04 package installed.
    8. Open up a terminal with administrative/root privileges and execute the following command (substituting as necessary; <arch> refers to the same value as in step 5) -

      On Windows (note the forward-slashes for the -d argument value; it must be forward-slashes or it will not work) -
      <drive letter of device>\syslinux.exe -f -m -a -d /boot/<arch>/loader <drive letter of device>
      On Linux,
      syslinux -f -m -a -d /boot/<arch>/loader <path to your device, e.g. /dev/sdb1>

    And that's it, reboot, select your device as the device to boot from and your OpenSuSE desktop should load.

    Instead of using explicit device names (e.g. /dev/sdb1), I used the device/partition label instead so if new devices were added there wouldn't be a possibility of it changing the device name and hence causing the boot to fail. You can choose to use the other forms of persistent device naming instead, or even the explicit names, as long as it identifies your device.

     
  • Escaping square brackets in Python's glob

    Widely supported in both Unix and Windows shells, globbing is the magic that expands path specifications (e.g. ls readme*.txt) into a list of files that matches the specification. Besides the wildcard, there are some lesser known characters that act as globbing operators as well. That includes the square brackets, which mean match any of the characters enclosed once. Unfortunately, square brackets are also legal characters for filenames, and certain files often use square brackets in their names by convention. Some glob implementations, Bash and Powershell for example, have escape characters that allow matching the square brackets literally. Others however, do not and when you intend to match square brackets literally, they get interpreted as globbing operators and hence yield incorrect results.

    For these situations, there is a trick to escape globbing operators - enclose the operator you want matched explicitly inside square brackets, like this -

    ls project[[]X[]].*.pdf

    This will match all files starting with project[X]. and ending with .pdf. By enclosing each bracket in square brackets, it tells the glob implementation to match any of the characters inside literally, once.

    Python is one of those implementations that requires this trick (internally, Python converts the glob pattern into a regular expression - see the translate method in python/lib/fnmatch.py - but the converter does not have the ability to handle escape characters).

    Here is some Python code that adds this trick to all square brackets in the given glob_pattern,  causing glob to match all square brackets literally -

    import glob
    import re
    
    # given the following glob_pattern
    glob_pattern = 'project[X].*.pdf'
    
    # replace the left square bracket with [[]
    glob_pattern = re.sub(r'\[', '[[]', glob_pattern)
    # replace the right square bracket with []] but be careful not to replace
    # the right square brackets in the left square bracket's 'escape' sequence.
    glob_pattern = re.sub(r'(?<!\[)\]', '[]]', glob_pattern)
    
    files = glob.glob(glob_pattern)
     
  • Network shares and XBMC on Apple TV 2

    The Apple TV 2 is a great piece of hardware, but by itself it isn't particularly useful unless your life revolves only around the Apple universe. Luckily, once jailbroken and loaded with XBMC (thanks to the instructions here), it makes one of the best media centre devices, and also one of the cheapest (compared to devices like the Boxee Box or Popcorn Hour).

    It isn't perfect (720p max, can't hold and scroll on the remote, occasionally crashes...), and I wish I had Boxee's awesome remote, but its ability to play pretty much any format at that price outweighs all its flaws. Plus as a bonus, it still works fine as an Apple TV, including AirPlay (one of the few useful out-of-the-box features), even while inside XBMC.

    The XBMC port is still a bit green though, but is generally fairly stable and usable. I had issues connecting it to my Linux-based NAS via SAMBA though - XBMC/Apple TV 2's SMB (network/Windows shares) implementation doesn't seem to be able to do authentication properly, at least the secure kind. It just sits there, and eventually comes up with an 'operation not permitted' error.

    After picking through the XBMC logs on the Apple TV 2 and SAMBA logs on my NAS, I had a hunch it was authentication encryption causing the issue, so I disabled it with the following parameters in smb.conf -

    encrypt passwords = no
    null passwords = yes

    Everything worked.. except Windows. Windows 7 (and possibly Vista as well) refuses to authenticate with a network share via unencrypted means, for good reason.

    There isn't a solution yet, but there is a workaround - by setting authentication to occur on a per-share basis on the NAS, instead of a per-user basis, XBMC can access shares with guest permissions as SAMBA does not require authentication and defaults to guest on those shares. (You can still choose to authenticate to those shares, and Windows will automatically try to do so with your current credentials.)

    To do this, simply add or change the following line in your smb.conf -

    security = share

    On the shares that you want to be able to access from XBMC, add the following option -

    guest ok = yes

    Restart SAMBA, and you should be able to connect from XBMC.

    (As an aside, I believe that's how Windows works as well, because I could access the guest-allowed shares on Windows from XBMC fine, but not the secured ones.)

     
  • Changing fonts on boxee box

    Out of the box (pun not intended), the boxee box works does most of what it promises fairly well. It needs a bit of spit and polish, but it's a decent attempt - plus if it was done perfectly, there wouldn't be a way to get around the update screen on startup :)

    Unfortunately, boxee, and therefore the boxee box, isn't very multilingual. In particular, the default skin font, Museo Sans, doesn't contain the glyphs needed to display non-Latin alphabet languages, e.g. Chinese, Japanese or Hebrew. It is a well known problem, but fortunately, there is a workaround - replace the skin font with another font that does contain those glyphs.

    This is easy to do on a standard PC because you have full access to everything. Not quite the same on the boxee box - the main system runs off a read-only ISO (that isn't really an ISO, but close enough). Fortunately, there are some writable areas, including the area that stores user settings and metadata, as well as skins (boxee supports custom skins, but there's no UI to support this).

    To change the fonts, you need to be running firmware version 0.9. On later firmware releases (version 1.0 or later), the box has been locked down a lot more and the hack used to gain command line access no longer works. There are various ways to downgrade back to version 0.9 though. This process is relatively fiddly; you should have some experience with the Linux commands if you're trying this.

    Bypassing the initial update and future updates

    1. When you first turn on your boxee box, it should be running v0.9, but it will automatically try to download the latest update. To circumvent this, either connect to wifi, or disconnect the ethernet cable. If you're connecting via a cable, plug the cable back in once you get to the wireless configuration screen. If you get stuck at the update screen, power off the box and try again.
    2. Complete the rest of the configuration.
    3. Go to Settings -> Appearance -> Screensaver, and change Screensaver mode to Off. Note that Black is not the same as Off, and doing this may lead to potential burn-in on your display. If you leave the screensaver on though, boxee will pop up a dialog at some random point and you'll have to update or power off to get rid of it.

    Getting access to the terminal

    1. Go to Apps -> Repositories (under Extras) .
    2. Click Add Repository, enter the following URL and click Done - http://erikkristensen.com/boxee/
    3. Select Erik's App Repository, then UnBoxed. Click Start to start the app.
    4. Click the Enable telnet button, and take note of your boxee box's IP address.
    5. Using telnet (if you're using Windows, try executing telnet in the command prompt; if it doesn't work, download and use PuTTY) connect to your boxee box at the default telnet port. No credentials are needed to connect.

    Changing the font

    To change the font, you first need to make a copy of boxee's default skin into the userdata area, before changing the skin fonts and finally the config to point to this new skin.

    Make a copy of the default skin

    Simply execute the commands below in order in the telnet session. The trailing dots are very important.

    1. cd /.boxee/UserData/skin
    2. mkdir boxee-customfont
    3. cd boxee-customfont
    4. ln -s /opt/boxee/skin/boxee/720p .
    5. ln -s /opt/boxee/skin/boxee/colors .
    6. ln -s /opt/boxee/skin/boxee/media .
    7. ln -s /opt/boxee/skin/boxee/sounds .
    8. cp -r /opt/boxee/skin/boxee/Fonts .
    9. cp /opt/boxee/skin/boxee/skin.xml .

    Replace the font

    The most complete free font I've found so far is the Droid Sans set of fonts, designed for the Android operating system. You could also use Arial Unicode MS (installed with Windows, located in C:\Windows\Fonts), but that doesn't look very nice in my opinion, especially on the big screen. The Droid Sans set of fonts are freely available here, or directly downloadable here in .tar.gz format. I've re-hosted a recent snapshot of the font though, so you won't need to download it manually.

    Execute the following commands to replace with the Droid Sans font; adjust steps 2, 5, 6, 7 accordingly for other fonts. These steps pick up from the previous steps. You can get fonts on to the device either via the web with wget, a network share, or by plugging in an USB drive.

    1. cd Fonts
    2. wget http://edgylogic.com/media/uploads/droidfonts/DroidSansFallback.ttf .
    3. rm MuseoSans_500.ttf
    4. rm MuseoSans_700.ttf
    5. cp DroidSansFallback.ttf MuseoSans_500.ttf
    6. cp DroidSansFallback.ttf MuseoSans_700.ttf
    7. rm DroidSansFallback.ttf

    Changing the active skin

    This last set of steps activates the derived skin on your boxee box. Because of the limited space on the device, the only interactive text editor available is vi, which can be intimidating to some people. An alternative would be to set up a network share or use a USB stick to transfer the file to a computer, modify it, and transfer it back.

    Either way, to activate this alternative skin, you need to modify /.boxee/UserData/guisettings.xml. Once loaded, locate the following segment -

    <lookandfeel>
        ...
        <skin>boxee</skin>
        ...
    </lookandfeel>

    Now chance the value of the skin element to boxee-customfont, like this,

    <lookandfeel>
        ...
        <skin>boxee-customfont</skin>
        ...
    </lookandfeel>

    Save, transfer the file back to the boxee box if needed, then restart the boxee box. The default font should now have changed to whatever you changed it to, enabling you to see some CJK glyphs, instead of boxes.

    To revert these changes, simply revert the changes under Changing the active skin, then delete the /.boxee/UserData/skin/boxee-customfont directory.

    I haven't worked out a way to suppress the update notifications, so they'll be a bit annoying but bearable. If you upgrade, this hack may break and you won't be able to get terminal access to the box again without downgrading.

    Thumbs up to the info over at boxeeboxwiki.org for the info on how to do this.

     
  • Rendering variable-sized SVGs

    As I tweeted earlier, I've been playing with SVG recently. But not in the way most people use SVG - I've been working on a website that generates SVGs on the fly with variable sets of data. This means coding the SVG manually, as opposed to drawing it visually in something like Inkscape. 

    As the set of data is variable, the dimensions of the resultant SVG is also variable. This means it is easier to just leave off the height and width attributes on the head SVG element, because it can't really be determined until it is rendered (and even then, there could be differences in the renderer or settings that may affect it). You would expect it to be determined implicitly, and it does kind of happen, but not as well as you'd expect.

    The 4 major browsers all offer differing behaviour. Here's an SVG that demonstrates the problems. Try opening it in different SVG-supporting browsers, Opera, Safari, Chrome or IE 9.

    To start off with, for some silly reason, no browser has the ability to pan or zoom SVG out of the box, not even if you are opening the raw SVG in the browser. And SVG without explicit dimensions don't get scrollbars, so anything that doesn't fit on the screen might as well not exist, even though there is obviously an implicit size for it to work with. Who knows why.

    Anyway, everything looks good when you open it. In that SVG, there is a script that scales the image when you scroll your mouse wheel. Try it. Alternatively, you can adjust the zoom level in your browser using the menus or CTRL and - or +.

    Personally, the expected behaviour is that the entire image is scaled and the browser window should be filled with the image. Scrollbars should appear if the image is larger than the window. This means that the default SVG viewport should be the size of the image, and if the dimensions are not explicitly specified, they should be implied based on the rendering's bounding box. The SVG specs aren't very explicit about what should happen in this situation as far as I can see.

    Here's what happens in the browsers I have installed:

    Opera 10.63 - this one isn't bad. The scaling works as expected. And performance is pretty good. The only issue are the grey vertical lines at hourly intervals. Those lines should stretch the height of the viewport (x1=100 y1=0 x2=100 y2=100%), i.e. the image. In this case Opera seems to have decided the viewport size is the browser window size due to the absence of explicit image dimensions. Ok, and looks good at the default zoom level, but as soon as you zoom out, you can see that they end at the original zoom level's edge, instead of extending to the new viewport edge as expected. To be fair, I can't find anything in the SVG specs that explicitly say what should happen, but this behaviour doesn't seem useful.

    Safari 5.0.3 / Chrome 8.0.552.210 beta - the WebKit family of browsers don't cope with this very well. As you zoom, it scales the image's dimensions as well as the SVG elements. This means that when you zoom out, the image appears clipped, making zooming out pointless. You can see the same behaviour when zooming in, as the scrollbars appear. Looking around at the SVG DOM, it seems that the root SVG element is given a value of 100% for both the width and height if they are not specified. On load, this is calculated from the implicit viewport, the window. When scaling though, these values are recalculated using the new scale factor instead, causing the problem.

    Firefox 4 Beta 7 - first off, the performance is horrible compared to every other browser. Stutters and lags, and it isn't even particularly complex, though it is large. When scaling, it exhibits the same issues as Opera. Hope they fix it before 4 goes gold.

    Internet Explorer 9 Beta - surprisingly, this is the only browser that rendered the SVG as expected (although it didn't have scrollbars either). The image scaled as expected, and the grey vertical lines worked as well. Performance was impressive too. Maybe they had the advantage of hindsight.

    The solution

    There are two. The first one is obvious - explicitly specify dimensions on the SVG image. All these problems go away if you do this.

    The second is a bit of a hack - if your SVG is going to be viewed in something that can execute JavaScript, you can use the following snippet to add the missing width and height attributes based on the bounding box of the rendered content, i.e. the width and height as rendered. The problem with this is that it doesn't cause scrollbars to appear in Opera. In Firefox 4 Beta 7, the scrollbars only appear as you start scaling. It works fine in Firefox 3.6.10 though.

    <script type="text/javascript">
        <![CDATA[
            function fixDimensions() {
                // fix bug with Opera, Firefox and Chrome when scaling. Surprisingly,
                // only IE9 works as expected.
                var svgEl = document.documentElement;
                var bBox = svgEl.getBBox();
                if (svgEl.width.baseVal.valueAsString == "100%")
                        svgEl.width.baseVal.valueAsString = bBox.width + "px";
                if (svgEl.height.baseVal.valueAsString == "100%")
                        svgEl.height.baseVal.valueAsString = bBox.height + "px";
            }
        ]]>
    </script>

    You then need to add the following attribute to your root SVG element -

    onload="fixDimensions()"

    That is needed as it doesn't seem to reliably work in all browsers if you add that handler using addEventListener, presumably because it isn't added before the load event fires.

    Here is the same SVG, but with the above script, and without the scaling script. Use the browser menus or keyboard shortcuts to test.

     
  • Downloading large files with VBScript

    Downloading (well doing anything really) is a pain with VBScript, but recently, while writing a script to download, install and update apps from the net, I discovered that the MSXML2.XMLHTTP COM object can't download large files. Well, it isn't necessarily large files that break it, but files that take a while to download. There seems to be an in-built timeout, and once that has been exceeded, it will cut the connection.

    The only evidence of this is the status code - it will return 0.

    Turns out you need to use MSXML2.ServerXMLHTTP in order to set timeouts to a longer period. The relevant sub-routine is setTimeouts. Note this doesn't seem to be a timeout in the traditional sense - this timeout value specifies how long the entire response can take, not how long to wait if there is no response from the server.

    Below is some code that wraps all this in a neat function.

    Function Download ( ByVal strUrl, ByVal strDestPath, ByVal overwrite )
        Dim intStatusCode, objXMLHTTP, objADOStream, objFSO
        Set objFSO = CreateObject("Scripting.FileSystemObject")
    
        ' if the file exists already, and we're not overwriting, quit now
        If Not overwrite And objFSO.FileExists(strDestPath) Then
            WScript.Echo "Already exists - " & strDestPath
            Download = True
            Exit Function
        End If
    
        WScript.Echo "Downloading " & strUrl & " to " & strDestPath
    
        ' Fetch the file
        ' need to use ServerXMLHTTP so can set timeouts for downloading large files
        Set objXMLHTTP = CreateObject("MSXML2.ServerXMLHTTP")
        objXMLHTTP.open "GET", strUrl, false
        objXMLHTTP.setTimeouts 1000 * 60 * 1, 1000 * 60 * 1, 1000 * 60 * 1, 1000 * 60 * 7
        objXMLHTTP.send()
    
        intStatusCode = objXMLHTTP.Status
    
        If intStatusCode = 200 Then
            Set objADOStream = CreateObject("ADODB.Stream")
            objADOStream.Open
            objADOStream.Type = 1 'adTypeBinary
            objADOStream.Write objXMLHTTP.ResponseBody
            objADOStream.Position = 0    'Set the stream position to the start
    
            'If the file already exists, delete it.
            'Otherwise, place the file in the specified location
            If objFSO.FileExists(strDestPath) Then objFSO.DeleteFile strDestPath
    
            objADOStream.SaveToFile strDestPath
            objADOStream.Close
    
            Set objADOStream = Nothing
        End If
    
        Set objXMLHTTP = Nothing
        Set objFSO = Nothing
    
        WScript.Echo "Status code: " & intStatusCode & VBNewLine 
    
        If intStatusCode = 200 Then
            Download = True
        Else
            Download = False
        End If
    End Function