Alex Rivera | Logout

Storing a large number of images

Asked 2009-01-15T10:57:43.097
53

I'm thinking about developing my own PHP based gallery for storing lots of pictures, maybe in the tens of thousands.

At the database I'll point to the url of the image, but here's the problem: I know is impractical to have all of them sitting at the same directory in the server as it would slow access to a crawl, so, how would you store all of them? Some kind of tree based on the name of the jpeg/png?

What rules to partition the images would you recommend me?

(It will focused for using in cheapo dot coms, so no mangling with the server is possible)

image tree filesystems
Edit
Report

3 Answers

10

I usually just use the numerical database id (auto_increment) and then use the modulu (%) operator to figure out where to put the file. Simple and scalable. For instance the path to image with id 12345 could be created like this:

12345 % 100 = 45
12345 % 1000 = 345

Ends up in:

/home/joe/images/345/45/12345.png

Or something like that.

If you're using Linux and ext3 and the filesystem, you must be aware that there are limits to the number of directories and files you can have in a directory. The limit is 32000 for dirs, so you should always strive to keep number of dirs low.

answered 2009-01-15T12:38:02.737
6

When saving files associated with an auto_increment ids, I use something like the following, which creates three directory levels, each comprised of 1000 dirs, and 100 files in each third-level directory. This supports ~ 100 billion files.

if $id = 99532455444 then the following returns /995/324/554/44

function getFileDirectory($id) {
    $level1 = ($id / 100000000) % 100000000;
    $level2 = (($id - $level1 * 100000000) / 100000) % 100000;
    $level3 = (($id - ($level1 * 100000000) - ($level2 * 100000)) / 100) % 1000;
    $file   = $id - (($level1 * 100000000) + ($level2 * 100000) + ($level3 * 100));

    return '/' . sprintf("%03d", $level1)
         . '/' . sprintf("%03d", $level2)
         . '/' . sprintf("%03d", $level3)
         . '/' . $file;
}
answered 2010-07-28T19:50:57.493
1

I am currently facing this problem, and what Isaac wrote got me interested in the idea. Tho my function differs a little.

function _getFilePath($id) {
    $id = sprintf("%06d", $id);
    $level = array();
    for($lvl = 3; $lvl >= 1; $lvl--)
        $level[$lvl] = substr($id, (($lvl*2)-2), 2);
    return implode('/', array_reverse($level)).'.jpg';
}

My images are only in thousands so i only have this up to 999999 limit and so it would split that into 99/99/99.jpg or 43524 into 04/35/24.jpg

answered 2011-03-05T00:39:59.073

Your Answer