Use the system trash with ZSH terminal

When you start using the terminal as your main file explorer instead of a GUI one, you soon discover two important things.

First, it is much faster to browse accross your filesystem, copying and moving files in the terminal that it is with your mouse. This is hard to believe at first (how can writing text on a dull black screen be faster than drag'n'dropping ?) but it is nonetheless true (after a bit of practice, sure).

The second thing is that it is also much much easier to *permanently delete very important files *as deleting a file through the terminal has no trash bin nor any other safeguard mechanism.

Scripting a rm replacement

At first, I scripted my own rm rm replacement that was manually moving files to ~/.local/share/Trash/files (the common Trash directory) instead of deleting them. But it was a bit naive and couldn't really work on removable drives nor provide a "restore" mechanism.

Fortunatly, the trash-cli package on Ubuntu provides a set of methods to deal with the trash from the command line. They have very explicit names such as trash, list-trash, restore-trash or empty-trash.

ZSH aliases

I had to resort to quite a bit of ZSH tweaking to make it a perfect rm replacement. First, I added a simple alias for the rm command.

alias rm='trash'

Then I also wanted to change the default rmdir command. I could have used the same type of alias (alias rmdir='trash') but I would have lost the builtin ZSH autocompletion of directories zsh provides with rmdir.

When you define aliases with ZSH, you can choose if you want it to autocomplete based on the right hand side of your alias (NO_COMPLETE_ALIASES) or the left hand side (COMPLETE_ALIASES). Yes, the name of the options seem wrong to me too, but this is actually how it works.

I prefer setting NO_COMPLETE_ALIASES so I can use the correct autocompletion on my commands with my aliases, but for the rmdir case this was proving to be an issue.

rmdir autocompletion

So, I started writing my own rmdir implementation in a custom script. This was merely a wrapper to rmdir but putting it in its own script allowed me to change its name and thus changing its autocomplete method.

I named it better-rmdir, and put it in my $PATH. Here is the code

#!/usr/bin/zsh
trash $@

As you can see, this is just a wrapper, taking the initial arguments and passing them to trash.

But I also created a file named _better-rmdir and put it in my $FPATH (this is where ZSH goes looking to autocomplete methods). I just copied the code of the original _directories method (that you can probably find in /usr/share/zsh/functions/Completion/Unix/), and adapted it to fit my newly created better-rmdir

#compdef better-rmdir
local expl
_wanted directories expl directory _files -/ "$@" -

And finally, I added an alias (alias rmdir='better-rmdir') and everytime I ask for an autocomplete on rmdir it actually looks for the autocomplete of better-rmdir, which is the code contained in _better-rmdir and which in turn return only directories.

Now I have complete rm and rmdir commands in my terminal that move files to the trash.

A small note about video encoding and formats

Following my previous note on audio files, I made some research on video files. I have a huge amount of video on my hard drive, so much that I actually needed to do a bit of cleaning.

That's when I discovered that I had a plethora of different file formats (avi, mp4, mkv) and in various resolution, quality and filesize. Also, filesize was not always dependent on quality. I had some really heavy files that were not better looking that smaller one.

So I started some research on the Wikipedia and Google to understand more about all those fileformats, and here are my results.

Containers

First of all, the avi, mp4 or mkv file extension denote a container format. This container is just a box that contain video and audio stream. Not all boxes are created equal, though.

avi is a Microsoft container but gained (in)famous popularity in the first days of p2p filesharing. This is a very simple container, able to hold one video and one audio stream. Then, was created the mp4 container. This one is an ISO box, so already much better than the Microsoft one. This actually is a good container, allowing the use of several audio and videos streams as well as subtitles.

Then, came the mkv container. It is as good as mp4, and even allow more customization, but the important point is that it is patent free. The only downside to mkv is that it is not as widely supported as the other two format, but the specs being public and open, this is slowly changing.

I also had some other extensions, like wmv, mov or divx. divx is only an avi in disguise, because of legal reasons. wmv is another Microsoft container, this one being an improved avi with DRM. mov is an Apple container, aimed to media creation. A mov is actually quite a powerful container, that could be used for much more than simply media playback but also media creation.

If I had to choose, I'll pick mkv.

Video codecs

Now, what do we put in those boxes ? First, a video stream. We could put an uncompressed video stream in it, but this will result in huge filesize, so we actually never do that.

Instead, we use codec (short for coder/decoder) to compress this stream in something with a more manageable size. The most famous is DivX. DivX started as an encoder hacked from Microsoft first avi files and was heavily used to encode movies before sharing them online. Because of legal dispute with Microsoft, the guys that created DivX had to recode it from scratch. In the meantime, an opensource fork of DivX, named Xvid was created and after a few version became even better than the original Divx.

MPEG, MPEG2 and MPEG4 are succesive iteration of another codec that compress a video stream by looking at its pixels and checking if they changed from one frame to another, to avoid redrawing them. With each succesive versions it then started to track movement, color and lightning of these pixels. In the end, it gave birth to the h264 codec which is used on Blueray (while MPEG2 was used on DVD). h264 is the de facto standard of HD video of today. It requires more processing power than its MPEG counterparts but deliver a much better quality for an equal filesize. Recent hardware is today optimized to handle h264 natively.

As for mkv and mp4, there also is the same difference between H264 and Theora. Theora got equivalent quality than H264, but is patentfree. On the other end, H264 is very widely supported while Theora is not (as can be seen in HTML5).

Microsoft also created a closed source wmv codec to go with its wmv container. It is supposed to be based on MPEG4, but there is little known to it, so it of absolutly no use to me.

So, as you can see, there is almost no relation between a codec and its container. You can have a divx encoded file in an mkv, or a Theora in an avi. But, in the real world, some codec are most often found with some container, like divx in avi or h264 in mp4.

Here, I'll pick h264 because of the hardware support.

Audio codec

As for the video stream, the audio stream is also compressed using an audio codec. The most common audio codec is the mp3. This is a lossy codec, meaning it discards information to get the filesize lower. It is based on a method that will discard sounds that the human hear will not be able to hear anyway.

mp3 has its drawbacks, like being bloated from succesive mpeg versions and still being patented.

A new codec, the AAC succeed to mp3. It will also discard sounds the human hear cannot get, but will also encode redundancy in a better way. Even if mp3 was the de facto standard for a long time, AAC is gaining a huge popularity because it is backed up by all the big companies.

Then, the story gœs on and on. There is an open source, patent free equivalent to AAC, being ogg. Once again, its as good as its opponent (even better on low bitrates), but there isn't as many devices able to read ogg as they're are able to read AAC. But once, again, things are slowly changing.

Here, I'll choose ogg over the alternatives.

Conclusion

I have a bunch of files, in different containers, encoded with different encoders and I'm going to try to clean a bit all this. First, I'll get rid of all the "bad" containers (mov and wmv) and use mkv instead.

For files that are too big, I'll try to convert them to a better format. h264 video with an ogg audio stream in an mkv container.

Well, there also is the matter of the video resolution, data rate, fps and audio frequency but this might be the subject of another post.

How I want my music files encoded

I have a lot of music files on my computer. Some comes from CDs I encoded, other comes from the magic of file sharing. Most of them are in MP3 format, but I have a few ogg and wma tracks, mixed with even a bit of wav or flac files.

Even the mp3 files are encoded in a bunch of different settings, with various bitrates and frequencies. The only thing that is quite consistent in all that is that my tagging of metadatas.

Other than that it's quite a mess and has my collection grow larger, it starts to bother me. No two albums, even for the same music duration, are the same filesize.

I decided to clean a bit this whole mess. Let me explain my goal, and how I plan on doing that.

First, I want all my files encoded with the same settings, so the music I'm listening to and its file size is consistent accross different albums.

Second, I want all the files correctly taggued with track number, artist, album and song name. I don't really care about the year nor the genre. Genre is highly subjective, and year can be misleading in mixed albums like soundtracks, are we talking about the year of the release of the cd, or the year of the release of the individual song?

Third, I want all my files in ogg format. First, because I want to get rid of as many proprietary format as I can, and mp3 is proprietary. But mainly because mp3 is an encoding format bloated with the previous layers of MPEG layer 2 and MPEG layer 1, resulting in larger file size and more encoding complexity.

And finally, I want to have the best sound quality that I can afford, without going to extreme like listening to FLAC directly as my ears won't be able to catch the difference anyway.

So, I'm gonna buy a big hard drive, something around 1To I think, and put on it all my music files in FLAC format, encoded directly from CD. This FLAC files will have all the required metadata saved in them.

Then, I'll create an ogg subdir in each of my album dirs where I'll encode my FLAC in ogg with a bitrate of 128kbds (as the human hear won't be able to really get the difference with anything higher than that), and using the metadata from the FLAC files.

Finally, I'll put the ogg files on my portable media player and listen to them on the go.

I do not need to put the FLAC on the PMP, has my ears won't be able to hear the difference. I only need to put a compressed version. Maybe in a few years, when PMP hard drive size will have increase I'll be able to increase the quality of my ogg and use better quality files. If I ever need to do that, I won't have to rip the CDs again as I'll already have the FLAC to create the encoding.

This will be hard work, but I guess it will be worth it in the long run. At least, now I know a bit more about encoding formats and settings.

This blog got hacked

Today, at about 9pm (GMT+12), I found my own blog hacked. When I wanted to access it, I was redirected to a malware site.

Half an hour later, it is back online again, and here is what I did.

First, I downloaded the webpage on my computer using curl http://www.pixelastic.com/to analyse it in search of a clue on the attack vector.

Unfortunatly, I couldn't easily find the culprit. No img tag loading a php file, no XSS injection that I could find. My guess is that the attacker tampered the js files loaded to add its own custom evil script. But as my js files are concatened and compressed in a file with a md5 name, it wasn't obvious that they had been compromised.

After that, I sshed to the server and tried to find what files where modified since my last commit. As I'm using Mercurial for that, this was a simple matter of hg status.

And I got a shitload of result. Actually, all my php files had been modified (and as I'm using cakePHP, that means, a lot of file). Running hg diff, I found out that all the php code of each file had been replaced with an evil code (enclosed in several layers of eval+base64).

I updated my working directory to the latest commit with hg update --clean to get all those files as they were before the attack. Running hg status once more still showed a bunch of new php files added. Running hg purge finally get rid of them.

I finally deleted all the compressed css and js files, to force them to be created again, and that's it, the website is online again.

I still don't have a clue on how this happened. How did someone access my files ? Is that an XSS attack ? Is my password cracked ? Is there another security weakness I'm not aware of ?

EDIT : Got hacked again. Seeing that the cakePHP Cache files were deleted, I guess it is a known attack on a cake vulnerability. Got the website up again, but will fix it as soon as my holidays allow.

Downloading all the TEDTalk podcasts

As I'm about to move for a couple of long months without an internet connection and a lot of travel hours, I thought of downloading the TEDTalk podcasts.

Being a linux user for about a year now, I'm taking the habit of using the linux tools to do the boring and repetitive work. Downloading more than a hundred files seems like a good candidate for a script.

The complete list of audio podcasts is available here. I simply selected all text and copied it into a file.txt.

Then, it was simply a matter of extracting only the urls, and feeding them to wget

cat file.txt | tr ' ' '\n' | tr '\t' '\n' | grep http > list.txt
wget --no-clobber -i list.txt

This took a while to download as there are more than 400 files, but that way I'll have some interesting talks to listen to during my trip.