Memory Size of xxx Bytes Exhausted – It’s Really Annoying


Since the release of version 3.0 of WordPress more people are seeking for help because of this problem: My memory limit is exhausted! I also had some problems while testing my Plugin Codestyling Localization with the new version when it was editing a large Plugins or WordPress itself.

So I started to research the problem, read other articles about this topic, made some tests and found out some surprising results.

Research and Incredible things in PHP itself

PHP is as pure and untyped interpreted language designed with a focus on massive text manipulation. Therefore, I expect no deal with memory, as in high optimized programs in C or C++ . However, what I was reading in an error report of PHP Trac made my jaw dropped. That they deal with so much memory just to get more speed is ridiculous. Here is the link to the entry PHP Trac #41053. Let me try to explain.

Let’s start with the example code of the PHP Trac entry:

\n");
}
printMemoryUsage();
 
$end = 1000000;
$array = array();
printMemoryUsage();
for($i = 0; $i < $end; $i++)
{
    $array[$i] = 0;
}
printMemoryUsage();
?>

What does this code do exactly? Not much, except:

  • Can output Memory usage
  • To fill an array in a loop with keys as a serial number and value 0
  • The loop generated 1000000 (one million) pairs, the number (integer) to 0
  • It has been tested under a 32bit system

Naively, I would make the following statement: I have 2 integer numbers 32bit with 4 bytes and multiply 1000000 times = 2 * 4 * 1000000 = 8000000 (eight million) bytes net amount of memory. For this I reckon about 10% overhead for the hash management, makes about 9 million bytes or 8.5 MB.

Now the great results with PHP 5.1.1:

Memory = 0 MB
Memory = 0 MB
Memory = 57.5 MB

… and as icing on the cake PHP 5.2.1:

Memory = 0.1 MB
Memory = 0.1 MB
Memory = 99.4 MB

Apart from the fact that in PHP 5.2.2 then again it looks like in PHP 5.1.2, I think 58 megabytes is a lot of meat which PHP needs just to store the data.

It also received an immediate explanation from the PHP team, why this is so. The internal structure of the variables of PHP can be anything at any time and must be convertible. Therefore, a simple integer value assigns in this way, according to data and my own PHP C code analysis, actually 68 bytes.

Thus, if each variable has an overhead of 64 bytes + load (string is usually more than just 4 bytes), then you can easily figure out how many variables you get in memory before the memory limit is exceeded. And we have not even calculated the program code itself yet!

Even worse is it in 64 bit systems, because the C structures of PHP are not correctly aligned in memory, as the IA64 architectures would like to have. There is thus additionally memory “padded” and an entry can have 80 bytes or larger. So that’s why PHP consumes on 64 bit systems significantly more memory than 32bit. How the 64 bit programming looks like and where the pitfalls are is explained here: Data Alignment when Migrating to 64-Bit Intel® Architecture.

The lack of understanding of hosting for 64 bit systems

Lovely Hoster, when you offer customers a 64bit server with 64bit PHP, then 32M or 40M aren’t enough. The internal architecture of PHP is not yet fully adapted to 64-bit systems, so it would be entirely memory-saving. In addition, the pointer in C code are actually twice as large, so executable code in memory and generated structures also need more space.
So either give the customer a choice or provide them adequately storage, everything else is self-deception and foolish.

WordPress Core Team and the carelessness in dealing with memory

I have a couple of words for the WordPress Core Team. You do make a good job. Everybody makes errors and bugs and the least think a new feature throughout. But you should not close your eyes and ears to the needs of the community that has made you so big.

function wp_initial_constants( ) {
    global $blog_id;
 
    // set memory limits
    if ( !defined('WP_MEMORY_LIMIT') ) {
        if( is_multisite() ) {
            define('WP_MEMORY_LIMIT', '64M');
        } else {
            define('WP_MEMORY_LIMIT', '32M');
        }
    }

(from the core file: default-constants.php)

It is finally time to admit what BS you have put in the code, and finally realize that WordPress 3.0 in a default installation will barely able to run on 32M and WordPress Multisite under 64MB is an illusion.

Might also mean that it is more difficult for users abroad with another not built-in language, go ahead and test it on your systems! Then you will notice that your implementation of the language files, processed under the above Memory usage of variables from a 360 kByte WordPress language file costs easily 4MB memory. With the Theme and each translatable Plugin more memory usage will be added and you probably see the end of the memory very soon.

What does it mean for me personally?

I will strive to continue its efforts to identify the memory hog and examine my own code to see if I can reduce the memory usage.
Therefore, there will be an update of my Plugin, which can then handle even the most adverse conditions with WordPress language files.
In addition I have limited a multisite installation artificially to 32M. Without a language file I’m already at 22M, with loaded language file it is already more than 26M and got another 4 – 6M space for the language file processing. Normally this isn’t sufficient at all, as you already read above.

That is why there is this new feature in my next update and my Plugin will scan under these conditions WordPress and large Plugins. You can even edit them. I can not guarantee anything, because it also depends on the amount of Plugins you have, but as long as its not just less than 4 – 6MB you have left, it should work.

Conclusion and Outlook

I will analyze the memory hunger of some components and when I notice that there is something we can do about, I’m going to tackle it sooner or later.

If someone has interest to participate in the hunt for memory flaws, let me know. I hope you understand that I want to help but perhaps can not respond always in a short manner.


Posted

in

by

Tags:

Comments

14 responses to “Memory Size of xxx Bytes Exhausted – It’s Really Annoying”

  1. Ovidiu Avatar

    I can’t say I understand every little bit of your explanation but this seems incredible. I really do hope you won’t be flamed for the open words!

  2. Rares Avatar

    Whoever wrote the wp_initial_constants() function needs to be crucified.

  3. Frank Avatar

    We also hope – we want only write to understand the problem in PHP and WordPress. We want to provide a better understanding not criticizm!

  4. Joseph Scott Avatar

    There are profiling tools for PHP (in addition to the memory usage functions) that you can use to track down usage details. Figuring out how to reduce the memory needs of WordPress would be more productive than exchanging insults.

  5. Frank Avatar

    Heiko intends to show, no offense. Moreover, the problem is illustrated in PHP, not only the application is the problem.

  6. Andrew Nacin Avatar

    It is finally time to admit what BS you have put in the code, and finally realize that WordPress 3.0 in a default installation will barely able to run on 32M and WordPress Multisite under 64MB is an illusion.

    That’s not very fair, nor accurate. A default installation of WordPress hums along nicely at about 14 MB on my computer. I throw in a language pack and a few “heavy” plugins, and I’m up to 23 MB. Any plugin that takes a serious toll on the front-end memory limit needs to be profiled, because it is likely loading a lot of stuff it doesn’t need to.

    For administrators in the admin area, we increase the memory limit to 256 MB — an absurd number that ensures that plugins which are heavy in the admin, core and plugin updates, heavy image crunching, and the like, will all work.

    Furthermore, WordPress 3.0 was carefully profiled when we realized we would have an issue with the upgrade process — and thus this plugin came about — and 3.1 has seen a lot of performance improvements across the board.

    I’ll further echo what Joseph has said above. You’re welcome to profile WordPress and see where things can be improved. Hurling things around won’t improve the product, though.

    Regards,
    Nacin

  7. Alex M. (Viper007Bond) Avatar

    If you think WordPress uses too much memory, then why not actually do something about it instead of bitching and moaning? As the saying goes, patches welcome. 😉

  8. Heiko Avatar

    The intention for this article was not to lay the blame on somebody. As you may have read, I did investigate first the memory utilization by PHP itself at the C code level. Because PHP allocates too much memory for internal structs, any application running on top at script level (like WordPress or others) will face with the fact, that any script code executed can massively increase the memory usage.
    WordPress internally tends to store a huge ammount of data in global arrays (vars) which are additionally associative like as example $GLOABAL is. So the example taken from PHP developer trac matches this behavior perfectly.
    Many user with limited PHP memory are hitting the limits after upgrade the older versions to WP 3 and above. I didn’t name here the Suhosin extension of Apache directly, but if it is active and configured, the programmatical increase of memory limit will immediately lead to the exhausted messages because Suhosin doesn’t permit any changes of that value.

    What I have written in this articel does not only affect WordPress but any other CMS/Blog system too based on PHP. I know, that the core developers do the best to provide a professional system, perform tests and profiling. But if software grows and have enlarged needs, they should be clearly stated. We can’t blame WordPress for the memory utilization because the main reason is the PHP C level code! But the script code can be analysed and modified to conserve memory as much as possible.
    As example for core devs: using text keys at the filter/hook storage like “accepted_args” or “function” will consume more C level space in PHP because the C struct also allocates additional memory for the string. If it would be only a number like 0 and 1 it need not to use additional string allocation. This may not sum up to a huge ammount of saved memory but leads in normal cases to 48k upto 200K less memory utilization. If this single example would be consequently keept in mind, there would be enough potential to reduce the utilization of memory.

  9. Michel Avatar

    Very interesting article. What is my experience ? As WP plugins developer, I recently observe this memory message when preparing a new version of xili-language on my localhost server (MAMP).
    After a long step by step tests (because the message is currently not circonstantial), I discover a recursive call of a hook present since previous version but revealed by add od new features for this new version. The lesson : before increase memory, test deeply the code and plugins interactions… It can be time consuming but efficient..

    Michel
    dev.xiligroup.com

  10. […] WPEngineer started a discussion about memory use in WordPress. […]

  11. […] WPEngineer nous démontre comment résoudre le problème qui engendre les erreurs de type « Memory size of… » […]

  12. […] how much memory your WordPress website is using? The introduction of WordPress 3.0 means that more and more people are finding that their memory limit is exhausted. The WP Overview WordPress Plugin is for the real […]

  13. Dawesi Avatar
    Dawesi

    Is this issue isolated to linux? I’ve seen no such issues running the hundreds of WP installs on Win2k8….

  14. Drew Avatar

    It worries me that WP has not improved upon its failing performance issues. I really hope they are working towards that in the 4.0 release. It’s what really needs working on rather than adding more interface features.