think tank forum

can't log in?

· register an account
· recover your account
· search the forums

15 years ago
r1, link

maple
i like large datasets

anyone using s3 (i think i've seen lucas using it a few places)?

a company approached me about taking over their site. their current guy is horrible but the site is starting to max out its bandwidth limit per month. the current host has a max of 3TB per month they offer, so i can't just upgrade that. there are about 100 immediate things i'd like to do to this site (from design to infrastructure) but the first one is keeping their site from going down all the time since their advertiser and started to wonder wtf. the site is running joomla (plan on replacing that with something less complex) and i've seen some one-off plugins for it to integrate with s3.

also, i've seen that for that amount of bandwidth it would cost ~500 a month with s3. not really anything for a site that has quality advertisers like they do but, still a big bump over their current hosting plan. well, obviously their current hosting plan doesnt work for them so it kinda hard to compare. hah.

just wanted some opinions. i need to read up on all amazon services i guess. i only know s3. dont know about cloudfront, aws etc

ahh. all my recent projects have been php. back to the php world i guess. i've been loving python though.

15 years ago
link

lucas
i ❤ demo

yeah, i use it for all of my binary storage needs (except ttf avatars).

afaik, s3 is very reasonably priced. so even if they have to pay $500/month, hopefully it's easily justified because of their revenue.

s3+cloudfront is an amazing service. you should definitely read more about it. if you have any questions, let me know. :)

15 years ago
link

maple
i like large datasets

you use it for sites or also for personal stuff? just curious.

15 years ago
link

bluet
⎈

he was using it for backups

15 years ago
r4, link

lucas
i ❤ demo

15 years ago
r1, link

lucas
i ❤ demo

i'm currently using around 20GB of S3 storage, which costs about $3 USD per month.

all of the requests and transfers only cost a handful of pennies.

i'm pretty happy about getting 20GB of storage for $3/month!

15 years ago
link

maple
i like large datasets

yeah i switch all videos (large majority of their bandwidth) over to s3 after posting this thread. helped tremendously. super easy setup, and migration was very easy as well. sites running faster than ever.

i've been thinking about putting all my photos and videos from my camera up s3 to keep them safe and off my computer (~40GB), but now im thinking of restarting my flickr pro account and just making everything private. then i can access them in a viewable way (not just data like s3) on any machine.

i dont like storing anything just on the internet (run a mail, backup, web, etc on my own servers) so ill think this over.

i guess i could build a web front end on my machine that uses s3 as the backend for videos and photos.

15 years ago
link

lucas
i ❤ demo

flickr has not proven itself to be a reliable service in my eyes.

> i guess i could build a web front end on my machine that uses s3 as the backend for videos and photos.

it's easy and fun. i'd recommend mine, but you prefer making your own stuff in django. :p

15 years ago
link

maple
i like large datasets

> it's easy and fun. i'd recommend mine, but you prefer making your own stuff in django. :p

hah. very true. but on the other hand i am lazy.

> flickr has not proven itself to be a reliable service in my eyes.

how so? security, stability, privacy or what?

15 years ago
r1, link

maple
i like large datasets

actually. i just looked at your software again. i really like it. i don't take pictures for the photographs themselves like you do, though. it's not art for me (i like art, just not creative). its strictly a memory supplement. i have like ~10,000 digital pictures over the last 10 years (majority in last 4). and while your software is cool (even though its php :) ), its not an efficient way to peruse that many photos i dont think. no photo = no memory.

sidenote: iphoto is a great way to browse local images. it has some stupid features that i dont use, but for browsing images, its amazing. also i just started playing with its facial recognition bits and it's awesome.

15 years ago
r1, link

lucas
i ❤ demo

> i really like it.

thanks. :)

> its not an efficient way to peruse that many photos

so maybe you'd want to ditch the two-tier organization, and simply sort by date. you could add a labeling system and have a search feature.

> how so? security, stability, privacy or what?

stability--i've witnessed too many "hiccups." flickr does not have an sla. flickr does not have a good way to import or export photos en masse (in my opinion). i doubt flickr has a way to export data associated with photos (metadata, labels, titles, descriptions, notes, comments).

how does one go about exporting photos en masse from flickr, anyway? if you have 10,000 photos stored, can you download them all with a few clicks?

15 years ago
r2, link

lucas
i ❤ demo

i guess there are a few flickr export apps out there. flump is made by the adobe people.

i guess you could also be unlazy to get stuff off of flickr using the api.

15 years ago
link

maple
i like large datasets

how are you dealing with thumbnails with your software (havent looked at the code)?

you generate them on first few or during import? do you upload them up to s3 after you create them or keep them just locally on the webserver as they are small and you dont need to keep them?

15 years ago
link

lucas
i ❤ demo

i generate them all by hand using the batch mode of irfanview. they all go into s3.

15 years ago
link

maple
i like large datasets

gotcha - also i meant first *view* not first few.

15 years ago
link

lucas
i ❤ demo

if i were using aws's elastic compute cloud (ec2) anyway, i'd have ec2 grab the high res photos from s3, make various sizes, and put them in s3.

because i'm using my own server, i thought it would be ridiculous for my server to download them from s3, resize, and upload to s3.

i'd rather have a local script make the necessary sizes, then i'd upload them all together. i have to batch rename them all locally anyway, so why not make some sizes too?

15 years ago
link

asemisldkfj
the law is no protection

do you store your stuff in an iphoto library or do any editing in iphoto, maple? I've tried it before but I really despise how it trashes your filesystem hierarchy, doesn't really create a very logical one of its own, and creates duplicates any time you make an edit to a photo. saving the original makes a lot of sense sometimes, but being able to turn this feature off would make sense too, I think.

15 years ago
link

maple
i like large datasets

@asemi
i dont really do much editing other than rotating images. the folder structure is exactly the same as the events i created in iphoto. i dig that. but since i dont change them that much, when i move boxes i just copy the originals folder over to the new system.

i think what i will do is backup to s3 for proper backup and then use iphoto or flickr for viewing. that way even if flickr blows up i still have a copy. also that will give me a chance to play with ruby and the flickr api for that mass uploading/downloading.

15 years ago
link

asemisldkfj
the law is no protection

I thought I remembered the folder structure being more incomprehensible than that. hmm.

15 years ago
link

maple
i like large datasets

just so you can see; this is iphoto 09

http://fakeempire.com/media/img/iphoto_folders.jpg

15 years ago
link

asemisldkfj
the law is no protection

not bad. I still hate the original/modified thing. if I was dealing solely with photos I took it would make sense, but I have a ton of images that are just graphics. now that Preview has resizing ability I'm good with that usually.

15 years ago
link

maple
i like large datasets

yeah i only use it for photos, i have separate folders for images/wallpapers/graphics etc

15 years ago
link

lucas
i ❤ demo

i'm currently storing 20 GiB of data in s3 across 20 buckets.

i store:
* incremental backups of mysql databases
* incremental backups of svn repositories
* mission critical project data
* mission critical personal data
* website content for delivery via cloudfront

life in s3!

<3

technology » s3