The Joyent Community

that doesn't feel well

geography wrote:

I'm resizing it too. But that is not where it dies.
It dies right before I write it to disk (With Magick::Image.write from RMagick)

I meant "resizing" as anything to do with RMagick. There've been a couple threads on this, search around. Resizing you image before upload or not using RMagick and instead using ImageMagick via the commandline might or might not work.... basically you can't do much, unfortunately.

Using open(fname, 'w'){|f|f.write img.to_blob} has the same problem... I guess I'll try and run it from the command line but that seems like a drastic step. Why is the memory so limited, shouldn't the server be able to save 500k files, or does imageMagick have a very high overhead?

So what i've learned so far...

TextDrive has a memory limit.
-When using RMagick in a rails app running with fcgi and lighttpd this memory limit is exceeded.
-When the memory limit is exceeded the fcgi process is killed (No expcetions caught or anything in Ruby... the process ends)
-The problem occurs when writing images to disk (With either File.open or Image.write)
-The problem still occurs if I call GC.starts after RMagick calls

Here is what I don't get...
-Why is the memory limit so low (Or why does a simple RMagick call take up so much memory? My dispatch procs take up around 50 megs, I can't imagine that opening a 600k file would come close to using that much memory)
-Would putting my RMagick code in a different thread allow me to have more memory at my disposal?

So after some investigation it seems like RMagick is a total memory hog.

I load a 750k image with this tiny standalone ruby program...

image = nil
open(filepath) do |f|

Monitoring the mem usage (using top) I get this 8th column is mem usuage

#File being downloaded
7198 ruby 7.6% 0:04.83 2 15 60 3.37M+ 2.65M 4.88M+ 44.5M+
7198 ruby 14.4% 0:05.01 2 15 68 18.1M+ 2.73M+ 20.0M+ 80.9M+
7198 ruby 35.0% 0:05.49 2 15 67 41.8M+ 2.73M 43.7M+ 80.8M-

#File being loading into Magick::Image
7198 ruby 0.1% 0:05.49 2 15 67 41.8M 2.73M 43.7M 80.8M-
7198 ruby 0.1% 0:05.49 2 15 67 41.8M 2.73M 43.7M 80.8M-

#File being written to disk
7198 ruby 3.4% 0:05.53 2 15 72 43.4M+ 2.73M 45.3M+ 122M+
7198 ruby 55.9% 0:06.29 2 15 72 61.1M+ 2.73M 63.0M+ 122M
7198 ruby 48.4% 0:06.93 2 15 72 76.1M+ 2.73M 78.1M+ 122M
7198 ruby 48.2% 0:07.55 2 15 67 42.5M- 2.73M 44.4M- 80.8M-

So I guess the moral is RMagick eats up a ton of memory. I can't really see anyway around this problem though, maybe using the command line would eat up less memory (which is what I'll try next)

Last edited by geography (2005-11-20 21:29:17)

ImageMagick isn't a memory hog (or not as much of one, at least). RMagick is. GD might take more, definitely not a solution. The lightest-weight out of all of the image processing libraries is probably Imlib2 (has mature Ruby bindings, too, I think) but I'm not sure it's installed at TxD because it does require some X libs unfortunately.

Your best bet is probably to manipulate the files via ImageMagik's commandline tools.

So we can't have a website that lets a user uploads a image and create a thumbnail out of it?

Jan wrote:

I see the problem as a very simple one. IMO, things like image processesing should be done with local resources.

Baking cached thumbnails from admin uploads is an essential feature requirement of 9 out of 10 CMS orders a developer might recieve. People need this and there is nothing abnormal about resource use. It has never been a problem with PHP (i have been doing this on 4 hosts for 3 years).

rainbowsheep wrote:

Well, the obvious thing to do here is notify the RMagick developers, which I've done. Perhaps it's a configuration issue.

I doubt it. It's just RMagick gobbles up memory needlessly (it has an awesome API, though!). I do hope they fix it, because it's a joy to use.

rmagick wrote:

I don't normally visit this forum but Joe mentioned this thread to me.

Background: ImageMagick requires a little over 4 bytes/pixel to represent an image if it (ImageMagick) has been configured with QuantumDepth=8, or 8 bytes/pixel if it has been configured with QuantumDepth=16. So an image from my 4megapixel camera needs 16 or 32Mb.

Two things:

1. Is ImageMagick built with QuantumDepth=16 or QuantumDepth=8? If 16, and you don't need 16-bit pixels, then ImageMagick is using twice as much memory as necessary and almost twice as much CPU. Unless you need 16-bit pixels, configure ImageMagick using --with-quantum-depth=8. (--with-quantum-depth=16 is the default.)

2. See my note "Help, my script runs out of memory!" at http://rubyforge.org/forum/forum.php?th … um_id=1618

RMagick is a very thin layer over ImageMagick and as such should NOT be using very much memory itself. If you think differently, please email me at rmagick AT rubyforge DOT com and I'll be glad to investigatge.

Perhaps the FreeBSD port for ImageMagick is using --with-quantum-depth=a_billion? :-)

btw, does calling in-place resize! consume less memory than calling resize or does it still bulids two images at a point?

Does this:

irb(main):001:0> require 'RMagick'

=> true

irb(main):002:0> Magick::QuantumDepth

=> 16

means ImageMagick was compiled with --with-quantum-depth=16 or do I misunderstand the meaning of that constant?

If so, I feel it's a more than reasonable compromise to have it compiled with 8, if it means we can deal with larger images. how about yous?

So I wrote an super small wrapper for imageMagick that works well on textdrive.
http://journal.gleepglop.com/articles/2 … e-ruby-way

It basically wraps the command line with a clean interface. If you've been having trouble with ImageMagick it may be of some help.

$ mogrify -list Resource
File Area Memory Map Disk
------------------------------------------------

i.e. ImageMagick on cardero has 500mb memory limit. Just fyi