EE 3 model grab and refactoring for update script
Development and Programming
I have a really big site that I’m rehearsing upgrading from 2.9.2 to 4. In rehearsing, I’ve found that the installer scripts get stuck and suck up around 3gb in memory (I had to allow PHP to consume that much memory to get through the upgrade) twice, and both pain points are the same situation.
Note that this rehearsal involves disabling all extensions and removing all modules, so it is a framework only upgrade dealing with the database, no third-party addons at all.
The issue is that this site has 63,000+ rows in the channel layout table (exp_layout_publish). In the installer scripts, specifically in ud_3_01_00.php and (I think) ud_3_05_03.php, there is an upgrade function that starts with this:
$layouts = ee('Model')->get('ChannelLayout')->all();
and then iterates over the $layouts var and does some operations of some type. In both install script points where it does this, it takes around 35-40 minutes to get past this point, and like I said, 3gb of memory, because all 63K rows are loaded into memory.
The first time it happens is in the function synchronize_layouts() in ud_3_01_00.php, starting at line 393.
I need to figure out what ee('Model')->get('ChannelLayout') functions are available, and if I can find a way to modify these scripts to chunk the layouts table data instead of doing it all at once. It would be much more memory manageable to iterate 63 times over 1000 records, instead of loading all 63K records into memory at once.
I’m hoping an EE SE can chime in as to ways I can recode those functions to chunk the upgrade process on a database table of that size, using the ee('Model') construct, instead of just getting it all at once.
Do you have any idea why you have 63k channel layouts? In your working 2.9.2 installation, did you really create that many? What add-ons are installed in v2?
Hey Derek,
I actually solved this problem the way I intended to, doing this:
// This loops 64 times and covers all 63k records
$layout_count = ee('Model')->get('ChannelLayout')->count();
for ($n = 0; $n < (ceil($layout_count/1000)) * 1000; $n+=1000){
$layouts = ee('Model')->get('ChannelLayout')->limit(1000)->offset(($n))->order('layout_id', 'ASC')->all();
foreach ($layouts as $layout) {
// Do layout stuff here
}
unset($layouts);
}By chunking those two function, I got the upgrade from about 80 minutes to 8 minutes (niiiice!).
I can guess why we have so many; we have hundreds of user groups and hundreds of channels. Apply the issue across 6+ years, with often adding more channels and user groups over time, and applying a channel layouts all through out this time, it must of have grown in time along with number of channels and member groups.
We definitely aren’t going to claim we have an efficient set up for this client, but it is what it is. And yeah, there are too many addons to list, many of which we are factoring out for this upgrade.
You have got to email [email protected] and let us take a look at the scale that you’re using ExpressionEngine with. That’s amazing! Glad you worked around the timeout.