Converting files to UTF-8 from the command line

Converting to and from charset encoding is boring. Life would be much easier if everything was saved in proper UTF-8.

I just coded a couple of scripts that will help be read file encoding from files and convert them to UTF-8.

First, not all encoding can be easily read. Some uses markers that are easy to check, other requires heuristic and guessing. I won't deal with the second part. Most of the files I'm currently handling are either Latin1 (ISO-8859-1) or Latin2 (ISO-8859-2), so I'll limit the scope of my scripts to those two sets.

Fortunatly, the default file command can do that.

$ file -bi file.txt
text/plain; charset=us-ascii
$ file -bi bad.html
text/html; charset=iso-8859-1
$ file -bi good.html
text/html; charset=utf-8

Now that we have a way to know the input encoding, we can use recode to convert to UTF-8. You can install recode easily with sudo apt-get install recode.

Once done, it is just a matter of convert latin1..utf8 bad.html

I wrapped those methods in two scripts : encoding and utf8 that respectively output the file encoding and convert the file to utf8.

Downloading from Youtube for offline watching

Call me old fashioned but I do not really like the "always connected" motto that the industry is following right now. I like to have physical music and video files on my hard drive, so I can access them anywhere, anytime, without relying on me being connected to a network.

And yes, I get the irony of talking about physical _files._

Anyway, I now download movies directly from Youtube, Dailymotion, Vimeo and consorts, copy them to my smartphone, and watch them in the train, waiting queues or bed.

To achieve that, here are the command-line tools I'm using. First is youtube- dl, and as its name does not imply at all, it can download from more than Youtube. The second one is vimeo_downloader which is specialised in downloading from Vimeo. And the last one is the more generic getflashvideos that can download from a variety of smaller websites.

All those scripts accepts the video url as parameter.

Happy scripting and happy viewing !

Installing mitmproxy

I spend the last hour trying to install mitmproxy. This is supposed to be a commandline proxy to intercept, read, edit and replay HTTP(S) requests. I say "supposed" because I haven't had the chance to try it so far as I was blocked on a compilation error.

According to the official documentation, installing mitmproxy should be as easy as pip install mitmproxy. Being new to using pip, I first had to install it (sudo apt-get install python-pip) and figure out that one need to install stuff with pip using sudo.

But sudo pip install mitmproxy resulted in charming compilation errors in my case. Something along the lines of :

src/lxml/lxml.etree.c:16:20: fatal error: Python.h: No such file or directory

After a bit of googling, I understood that the issue was some missing headers on my part. After running sudo apt-get install python-dev I was good to run a sudo pip install mitmproxy again.

Next step will be to configure the beast and learn how to use it.

Doxie + Dropbox + Ikea = Love

Thanks to doxie and dropbox, I'm going the paperless way. Every important paper I get in my physical mailbox, I now scan and put on the cloud. I have been doing it for bills, medical papers, tickets, warranty and am now even doing it for Ikea instruction booklets.

This way, when I'll be moving to a new appartement, I'll still be able to dismantle and rebuild my furniture, even if they have gone out of production.

As a gift, here are the links :

Back in Paris !

After one year in New Zealand, I'm now back in Paris. The trip to New Zealand was one of the best years of my life, I discovered so many things and places, met so many people, that it changed me in more ways that I can really count. It also greatly influenced the way I want to work.

Being back in Paris, I now need to find a new job. I could go back to freelancing, but I don't really want to. Freelancing had its advantages, but also some important downsides. The biggest one being that you're mostly working alone, and jumping from one project to the next.

I now want to work in a team, to teach, and to learn. Sharing with others what I know and learning from them what I don't is one of the most important things I'm now looking in my job.

I also want to be working on one big project instead of countless small ones. When you're freelancing, you're working on a project for weeks or months, creating a website from scratch and making it take a life of its own, line after line. But in the end, you hand it over to the guy who paid you to do it. And you never really get to see it growing. I now want to work on a project that I'll be able to follow, fix, and improve, days after days once it is launched in the world.

In New Zealand, I have been a WWOOFer (Willing Worker On Organic Farms) for over 8 months. All this was possible through the wwoofing website. Now, back in France I extensively use services such as couchsurfing, airbnb or blablacar. I love the way those websites allow people to meet through their services. I've met very interesting people thanks to that and I'd love to work for a company that can help linking people together.

On a more technical-oriented side, I also want to change my tools. I've been making websites for the past 10 years, coding the server side part in PHP. I now want to try something else. I used my time in New Zealand to learn a new language on my own. I first tried Python, but couldn't really go anywhere with it. I then tried Ruby and this was like discovering a new world. Writing and reading Ruby seems like writing and reading prose. I haven't yet build any website with Ruby, but coming from a cakePHP background, I guess Ruby on Rails will seem natural to me. I now use Ruby extensively for my own scripting, be it for changing a subtitle .srt fps, automatically synchronizing my cellphone files with my laptop, batch updating my mp3 tag collection and mostly any tedious task that involves scripting on my day-to-day life. I haven't got the chance to try node.js yet, but that's definitely next on my todolist.

Following the same idea, I also want to get to better use git and its collaborative features. I've only used subversion in a team, and the experience wasn't that great. Or I've used git, but only for myself as I was the only developper and I'm sure I only used it to barely 10% of what it could do. I've also tried to follow the TDD principle of writing tests first and then the code to make the test pass. I managed to do it once or twice, but the whole process seemed so long and difficult that I'm pretty sure I've missed something here, and would like to have someone to show me how they do it.

On a related note, I now try to free as much code as I can through open- source, but I'd like to do even more. I currently only share small scripts without tests and with scarce documentation. I know I could do better than that, but I'll need someone to show me the way.

I've managed to switch from Windows to Ubuntu in the past years and can't imagine going back. Working with your filesystem from inside a terminal and writing scripts to make your life easier is so powerful that I cannot go back to a GUI OS now. I've also ditched a classical IDE, namely Komodo Edit, for something different, namely vim. The switch has been hard at first, but now all the vim keys are binded to my muscle memory and I love vim for all the exact same reasons I love a terminal : you can script your tasks !

This is only a small subset of the technical changes I've already made or am currently making. I'd like to find a job where I'm hired, not for my past skills, but for the skills I want to develop. I currently have a resume where I can attest extensive experience with PHP, but that's not what I want to do in my future. What I want to do is, obviously, not on my resume. That is currently the tricky part I have to face when looking for a job.

As you can see, I have a pretty good picture of the kind of place I want for my next job. Luckily, I'm living in Paris, with a lot of job opportunities and I'm pretty sure I'll manage to find a place that match most of it. If you're currently hiring, and my expectations meet yours, don't hesitate to shoot me an email.