Amazon S3

Want to store DOCman files on Amazon S3? This is easily done by using s3fs.

s3fs allows you to mount an Amazon S3 bucket as a file system so you can access it like any other file.

Installation

Because there are no official packages available, you will need to compile s3fs yourself. Luckily this is very easy.

Installing the build dependencies

If you are on Ubuntu:

sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

If you are running CentOS:

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

If you are running another operating system, please refer to the s3fs wiki.

Build the s3fs source code and install it

cd ~
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

Set up user permissions

If you need a read-only set up, you can skip this step. If you need write permission for PHP, you have to add the correct user to the fuse user group. On most systems, this is either www-data or apache. If you are unsure which user to assign, consult with your system administrator.

To add the www-data user to the group, simply run:

sudo addgroup www-data fuse

Mounting your S3 bucket

Store credentials

Store your login credentials where s3fs can find them (make sure to replace with your correct ID and access key):

echo "<AWS Access Key ID>:<AWS Secret Access Key>" | sudo tee /etc/passwd-s3fs
sudo chmod 600 /etc/passwd-s3fs

Mount S3 bucket

Now mount the bucket:

sudo mkdir /mnt/mys3bucket /var/cache/s3cache
sudo s3fs -o use_cache=/var/cache/s3cache,allow_other,nonempty joomlatools-s3fuse-test /mnt/mys3bucket

Test it by creating a file:

echo "Test" | sudo tee /mnt/mys3bucket/test.txt

Verify if the file shows up on your Amazon S3 bucket through the Amazon web console.

If you get errors or the file is not being synced, take a look at your system log (on Ubuntu: /var/log/syslog) to see what s3fuse is saying.

Mount S3 bucket after reboot

Add an entry to the /etc/fstab file:

joomlatools-s3fuse-test /mnt/mys3bucket fuse.s3fs _netdev,uid=33,gid=33,umask=007,use_cache=/var/cache/s3cache,allow_other,nonempty,retries=5 0 0

Move your files to the S3 bucket

Create directory

Create the joomlatools-files directory on S3 and give it the necessary permissions:

mkdir /mnt/mys3bucket/joomlatools-files
sudo chown --reference=/var/www/yourwebsite.com/joomlatools-files/ /mnt/mys3bucket/joomlatools-files
sudo chmod --reference=/var/www/yourwebsite.com/joomlatools-files/ /mnt/mys3bucket/joomlatools-files

Copy files

Copy the joomlatools-files directory contents to S3:

sudo rsync -a /var/www/yourwebsite.com/joomlatools-files/ /mnt/mys3bucket/joomlatools-files/

Add symlink

Backup the original directory and replace it with a symlink to your mounted S3 bucket:

cd /var/www/yourwebsite.com/
mv joomlatools-files joomlatools-files-backup
ln -s /mnt/mys3bucket/joomlatools-files joomlatools-files

Test your DOCman installation. Remove the joomlatools-files-backup directory if everything works.

Your files are now being stored on Amazon S3!

Frequently asked questions

Will the mounted S3 bucket take up much disk space?

No. Because s3fs is a virtual file system, it will only download the files when they are accessed. These will be stored in a temporary directory which s3fs cleans up automatically. 

If you want to reduce the number of downloads, you can use a local cache to keep files around on your disk. For more information, please refer to the  s3fs docs.