How to decode URL-encoded string in shell?

142

I have a file with a list of user-agents which are encoded. E.g.:

Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en

I want a shell script which can read this file and write to a new file with decoded strings.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en

I have been trying to use this example to get it going but it is not working so far.

$ echo -e "$(echo "%31+%32%0A%33+%34" | sed 'y/+/ /; s/%/\\x/g')"

My script looks like:

#!/bin/bash
for f in *.log; do
  echo -e "$(cat $f | sed 'y/+/ /; s/%/\x/g')" > y.log
done

Edit Report

1 Answer

As @barti_ddu said in the comments, \x "should be [double-]escaped".

% echo -e "$(echo "Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en" | sed 'y/+/ /; s/%/\\x/g')"
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en

Rather than mixing up Bash and sed, I would do this all in Python. Here's a rough cut of how:

#!/usr/bin/env python

import glob
import os
import urllib

for logfile in glob.glob(os.path.join('.', '*.log')):
    with open(logfile) as current:
        new_log_filename = logfile + '.new'
        with open(new_log_filename, 'w') as new_log_file:
            for url in current:
                unquoted = urllib.unquote(url.strip())
                new_log_file.write(unquoted + '\n')

answered 2011-06-06T11:39:59.960

How to decode URL-encoded string in shell?

1 Answer

Your Answer