Better handling of 64KB+ content fields
Feature Requests
Yesterday, after 10+ years of using EE, I fell foul of the maximum content field size, which I think is 64KB, and undocumented. This was in 2.11.3, in which fields larger than 64KB seem to be silently truncated, potentially losing data. In my case, the field had been edited enough times that data was actually lost on the server, and had to be recovered from my test set-up.
Losing data silently like this isn’t good. I’d have expected at least a warning. But since EE2 is on its way out, I’m not going to whine too much about that.
I just deliberately pushed EE3 (3.5.8) in the same way, though, and it doesn’t fail much less clumsily, giving a low-level SQL error and dump of the field:
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column ‘field_id_5’ at row 1:
That sort of works as a warning that something has gone wrong, but it’s not a tidy way to catch or handle the problem.
In addition to gentler error-handling in this situation, it might be nice to have a display in the editing screen which gives a sense of how close a field is to its practical limit. Something like: “2KB used of 64KB”.
Thanks for considering.
I agree with you that we should trap content that would exceed the schema’s maximum size instead of letting MySQL barf an error. But can I ask what type of content you are storing that you are placing 64k+ into a single field? That seems very counter to the purpose of custom fields and using a CMS.
Derek:
But can I ask what type of content you are storing that you are placing 64k+ into a single field? That seems very counter to the purpose of custom fields and using a CMS.
It’s a directory of people that’s grown very quickly over the past couple of years. It definitely should be reworked somehow, but it’s one of those situations where there are also advantages to having the content in a single place, so there hasn’t been any strong reason until now.
I wasn’t necessarily thinking of just my situation, though. In the general case, it’s not hard to imagine writing that easily fills 64KB — long-form journalism or short fiction, for example. These aren’t pieces that would break naturally into smaller chunks. With even simple markup, 64KB is about 8000-10000 words, which isn’t unusual in a Medium-like environment. I have no idea how rigid the 64KB limit is, but whatever the limit is, it should be made more apparent to the user.
(Aside! The box I’m typing this reply in tells me how many characters I’m allowed, and how many are left. That’s exactly the sort of thing that I’m thinking would help the EE back-end.)
I can confirm that this is still an issue in 5.3.0. I have a blog that uses a “Body” field for the main content of each post. Tonight I was editing a draft that was fairly long, but not epic, and suddenly I discovered that text from the bottom of the post was disappearing with each save. I lost a lot of content, and then I lost hours trying to figure out what was going on. As Paul says, as much as the data limit is a problem, the silent loss of data is worse.
https://github.com/ExpressionEngine/ExpressionEngine/issues/464