(04/28 UPDATE): I’m now running my first-pass patch right here on my own site. ūüöÄ

WordPress¬†is an extremely flexible piece of software, and it comes with many different settings. Some are made visible to users via Admin > Settings¬†and others are stored invisibly so users aren’t bothered by them, but all of them¬†are saved in a single database table named wp_options. Today, it looks something like this:

The wp_options database table schema

This database table actually has a few interesting qualities to it. Conceptually, it’s a very simple key/value approach¬†to storing any kind of arbitrary information. It’s a distant cousin to¬†all of¬†the meta¬†database tables WordPress comes with (for posts, comments, terms, and users) and I’m a big fan of¬†the entire meta-data¬†API ‚Äď it’s¬†now fully implemented across all major object types (except blogmeta and term_relationshipmeta)¬†and, honestly, it’s one of the few “complete”¬†APIs you’ll¬†interact with inside of WordPress today, aside from probably roles & rewrite rules.

The options¬†API, however, is actually quite a bit different from¬†meta, enough to warrant this blog post, and enough for me to have spent the past 4 days studying it, researching it, and generally trying to find ways to improve how it performs under stress and load. (I’ve come to appreciate meta even more now, though I have¬†some concerns about it that I’ll share near the end of this post.)

And for all the flexibility that WordPress allows us, once we start taking advantage by adding plugins, themes, multisite, and so on, we¬†are likely to reach a point where the wp_options¬†database table becomes a graveyard of abandoned options that never get used again.¬†“Full” is a funny (but purposeful) word avoidance choice here. A quick search of the web will tell you that the maximum number of rows a database table like wp_options¬†can have is around¬†1¬†billion rows, but because of how WordPress interacts with¬†them, “full” could actually mean only a few hundred rows, or 1 huge row if you’re being terrible.


Name, Value, Autoload

This table has 2 unique keys ‚Äď option_id and option_name¬†‚Äď though WordPress only ever concerns itself with option_name. ¬†A unique key means no two rows can contain the same value in that column, which is like saying “no two switches control the same light-bulb” or “no two drivers licenses identify the same one person.” In WordPress, this means that every setting only has 1 unique entry to look for, making it very easy to know that it’s doing what you’ve asked it to.

The option_value column is a¬†LONGTEXT, which means under today’s typical MySQL configurations, one row could contain around 4 billion bytes (or about 4 gigabytes.) You probably don’t want to store that much information there, but I think it helps to understand the theoretical maximum.

Lastly, the autoload column exists to minimize the number of peeks into the database WordPress needs to load all of the settings, and will only ever have a value of¬†yes¬†or no.¬†(This is pretty¬†silly today, and¬†it wouldn’t be an unusual old MySQL performance hack if the autoload¬†column were indexed.) The reason this column is important, is because¬†every question takes time to answer, and in this case, asking WordPress 100 separate times to look in the database for 100 options would take 99¬†times longer than asking it to look 1 time for all of them ‚Äď if you know exactly what options to load automatically, you can get all of them at once ‚Äď at least, that’s the intended logic anyways.


Load All of the Options!

I won’t bore you with all of the details, but the number of relatively complex decisions¬†that¬†WordPress makes¬†just to load the correct page on a website is both impressive and dumbfounding. Before any of that can happen, though, it needs to peek into the wp_options database table to understand how to make those decisions in the ways you’ve configured it to. Here’s a screenshot of some of the oldest and most common entries:

Because WordPress is a friendly piece of software, it doesn’t obfuscate anything in the database by default, and it uses a generally concise and simple naming scheme for almost all things ‚Äď an option like use_smilies¬†determines whether or not to, use smilies.

Since¬†WordPress 2.2.0, a function named wp_load_alloptions()¬†(previously named¬†get_alloptions()) is used to ask one time for all options where autoload¬†is equal to yes. Inside this function is where the first option optimizations occur, including our dramatic introduction to the object cache that comes with WordPress by default. It’s another rudimentary key/value storage system, only instead of in a database table like options or meta, it’s in memory in PHP, so it’s locally “hot” and very, very fast (much faster than querying a database, for example.)

Every single time you load a page that’s powered by WordPress, it loads all those options so they’re “primed” and ready to be referred back to at anytime until the page has completely finished loading. The nice thing about this, is you can call the same get_option( 'use_smilies' )¬†function a hundred times, and it will return the same result without ever needing to ask the database whether or not that option exists.

At least, that’s the case with autoloaded options. Everything else… well… that’s another story.


Once, upon a time…

It was a pretty clever idea to load all of the options in one fell swoop, and more-so to allow certain options to be excluded from that. Some¬†options aren’t necessary on every single page-load, so if it’s only used¬†conditionally or periodically, it would be wasteful to always ask for it, keep it cached, and never actually refer to it.

But¬†the autoload¬†column actually comes with a bunch of added complexity. What happens if you need an option that doesn’t exist in the database yet? Or if an option needs to switch from being autoloaded, or to not be? That means each option you need that isn’t autoloaded will cost 1 additional database query to go out and get. And it means switching an option’s autoload property requires updating the database and the object cache at the same time to reflect those changes in the live environment.

If the autoloaded options all live under the¬†alloptions¬†cache key, not-autoloaded and nonexistent options naturally live under the¬†notoptions¬†cache key. I’m not kidding ‚Äď WordPress also caches the “other” options requests anyways, because if it didn’t, calling¬†get_options( 'bedtime_story' )¬†100 times would hit the database 100 times looking for an option that doesn’t exist in the database.

So if all options are cached, and not-options are cached, what’s the point of having an autoload¬†column at all? There’s an argument to be made for abandoning its usage, but backwards compatibility isn’t a feature, it’s an ethos ‚Äď it can’t just be ignored ‚Ästand this code goes all the way back to the very first versions of WordPress, so we can’t easily unplug it after all these years, can we?


Oh yeah…

All of that explanation up there leads us to¬†a hidden limitation of the options API deep down in its core. While WordPress comes with a simple local object cache, some persistent object¬†caching drop-ins (Memcache, Redis, etc…) have limitations on how much memory¬†1 cache key can occupy, and those limitations are much smaller than the 4GB¬†LONGTEXT¬†theoretical maximum of 1 row, let alone 1 billion theoretical maximum rows all in one alloptions¬†cache key.

For example, Memcached has a 1MB (default) upper limit for the entire¬†alloptions¬†array of autoloaded option names & values, and another 1MB upper limit for the notoptions¬†names and values array. That’s not per-option, it’s per array, for the¬†cumulative size of all items in each individual cache-key, so autoloading¬†them together into 1 primary key becomes a huge problem very quickly.

It’s actually one of those problems that any good¬†web hosting company has been silently solving for you forever, and subsequently a problem that I’m working on solving for them, and for you, and for WordPress going forward.


What’s Next?

13 years ago, WordPress was obviously a very different piece of software than it is today ‚Äď and we were very different people, so with any luck, our approaches to solving problems back then should¬†be different than how we’d do it today, too. I’ve spent the last 4 full work days doing the deep dive into why the current approach was good then, what seems good now, and what problems might we have going forward.

This post represents part 1 of those 3 steps.

I’ve written and rewritten hundreds of lines of code already, and determined that abandoning the alloptions¬†and notoptions¬†cache keys completely is the most desirable option, and using the¬†option_name¬†as the cache key for each individual option will provide the most headway going forward. I have a few other hypotheses to test, around storing entire objects vs. only the option_value, around having a separate cache group for notoptions¬†keys to reside in, and about adding a separate¬†wp_optionmeta¬†database table to allow for each 1 option to have its own¬†arbitrary attributes.

For me, I get to spend a few more days trying to implement my ideas and testing that they actually perform as good as (or better than) what WordPress is already capable of, and that it adequately spreads the pain around when drop-ins like Memcache or Redis are in use. I’ll report back my findings here, and share them on the relative WordPress core Trac tickets once there’s something worthy of a RFC.


You forgot something

Oh yeah! About the meta-data API, right? Interestingly, when trying to get a piece of meta data about an object in WordPress, the update_meta_cache()¬†function works similarly to wp_load_alloptions()¬†in that it queries it’s database table for all rows relative to the object’s ID. One¬†key difference is¬†that all meta is always loaded for the object (there is no autoload¬†option.) Another is that¬†any changes¬†to meta-data will delete the relevant cache, but options modify the cache¬†directly without deleting it so subsequent calls don’t typically generate¬†additional database queries.

As¬†custom post types start to store more and more things in wp_postmeta, cache keys may start to reach the same theoretical upper limit that wp_options¬†has been plagued with because they are not unique. For example, if you store a thousand matched meta keys for a single post, and each of those keys is of any significant size, it could totally hit the same Memcache 1MB default boundary. I don’t know of this happening to anyone yet, but it seems plausible if not inevitable.

If you’ve made it this far in this post, you care about a silly dumb problem as much as I do, so we are instantly friends. If you’re also independently working in this area of WordPress, I’m excited to compare notes and Voltron up and blazing sword our way through this together. ūüíú


Relevant reading

For most of my life, whenever someone would ask what kind of music I was into, I struggled to come up with a clear and obvious response. I’ve always enjoyed many different artists, songs, and genres, each for a myriad of reasons, and the one style of music that I consider “my favorite” isn’t one that most people can relate to.

And that’s really what the “what kind of music do you listen to” question is designed to do ‚Äď relate. It’s like when someone asks “what is your favorite sports team” and you enjoy sports but aren’t passionate about any specific type or group.

Then you tell them you love jai-alai, and either have to explain why, or skip over the conversation entirely.

It’s really hard to relate to people when the universe has gifted you with a heart that craves the unrelatable.

I love hip-hop, old-school, funk, edm, d&b, electro, Miami-bass, and a bunch of other genres that most people would go “oh… cool.” without actually thinking it was cool at all, because they don’t know anything about them.

But late last year, I came to realize it wasn’t really any “kind” of music that I liked as much as it was the “instrument” used to produce individual songs. There was one specific sound that hooked me when I was 11 years old, and it came from the Roland TR-808 drum machine.

WordCamp Miami had an 80’s theme this year, and that’s really when the 808 found its momentum. During my presentation for BuddyCamp 2017, I mentioned 808 The Movie, which is as adequate of a history lesson about my favorite type of music as will likely exist in my lifetime, though the west-coast omissions of The Egyptian Lover, Arabian Prince, and a few others still leaves me a bit disappointed in how such a huge part of what influenced the 808 movement ended up totally missing.

Anyways…

It’s with a heavy heart I share the following post from Hip-Hop DX, which talks a bit about the life (and recent passing) of the inventor of the instrument that I hear in my imagination everyday when I play back my favorite songs in my head.

EXCLUSIVE: Egyptian Lover speaks on Kakehashi’s legacy.

Source: Music World Mourns Loss Of Roland TR-808 Drum Machine Inventor Ikutaro Kakehashi

Ikutaro Kakehashi, like Gunpei Yokoi, influenced my life in ways that they’ll never know and that I’ll continue to subtly share in the code that I write and the WordPress plugins that I release and help build. They are under-appreciated craftsmen in their respective fields of study, both talented and lucky, who were able to help invent entirely new cultures and movements around their creative visions, and I hope that I’m able to leave something similar behind someday. ūüíú

January &¬†February were very busy and eventful months for me, as I’ve been working full-time¬†on WordPress Core & Community related endeavors thanks to two companies, and largely 2 individuals, who decided to take a bit of a risk and try something relatively new with me.

The scene is¬†WordCamp US 2016 in Philadelphia, PA ‚Äď the weather is surprisingly warm and the event is abuzz with close to 1800 WordPress fans in attendance. For me, it’s a reunion of sorts, because it’s only a few times per year I’m able to have in-person time with many of my ex-coworkers, current open-source colleagues, and friends I’ve made thanks to WordPress but maybe haven’t collaborated with yet.

I had several really productive conversations, mostly me pitching the idea of asking for financial support to “work on WordPress” in a way that does not place the burden on any one specific company, and Josh from Pagely and Josh from Pantheon both welcomed the experiment with open minds and were excited to get started.

January

I refer to the WordPress 4.7.2 release as “January” as I spent the majority of it¬†reviewing & hardening several important security improvements to the core codebase (namely, the REST API and Press This.) In addition, I upgraded bbPress across all of WordCamp.org, and laid the groundwork for upgrading bbPress across all of WordPress.org in the coming months.

The security team also paired up with liaisons from almost all major hosting companies and CDNs to bring awareness to and monitor for any public exploitation attempts, and I was actively involved in on-boarding and expectation setting while we introduced an entirely new workflow for all of us.

Thanks to Josh & Sally Strebel at¬†Pagely for supporting me and WordPress during this critical month. ūüíú

February

With WordPress 4.7.2 “January” out, things “settled down” and I switched to LoopConf mode where I gave a talk about how Unix Philosophies apply to WordPress development, and was interviewed by Brian Krogsgard of Post Status fame about the longevity of open-source software (I’ll embed that video below.) LoopConf was great, Salt Lake City was fun, and I’ll do a separate write-up about my experience soon.

I’ve also focused more the backlog of issues and nice-to-haves for the bb’s, largely related to how the WordPress.org Support team works with it as a tool. Working with other volunteer contributors like Boone Gorges, Stephen Edgar, Brandon Allen, and Sergey Biryukov again, has been a treat.

Thanks to Josh Koenig from Pantheon for an awesome groundhog’s day, happily looping for 28 days working on WordPress. ūüíú

March & April

I have 2 weeks “off” planned in March for WordCamp Miami and my wedding anniversary, and I’ll be speaking at CypherCon 2.0 in Milwaukee and I think a few other events that I can’t remember right now that I’ll come back and update later. Development wise, I suspect will be more of the same: security, on-boarding, WordCamp.org, and bbPress upgrades across WordPress.org.

I’m also going to help Pantheon with some marketing, which is something I’m excited to help with and learn more about, since I’ve spent most of my career helping build the things other people get to pitch to the world.

And here’s that Post Status video I mentioned earlier.

On December 4, at WordCamp US, an idea was formed. Over beers and snacks, a small group of friends lamented the lack of appreciation for Tweetstorming.

“Get a WordPress” people say, as if the thought never crossed their minds that 37 tweets in a row about the same thing starts to defeat the purpose of Twitter.

Myself, being the most stubborn, independent, and likely drunk of our group, volunteered to solve this problem the only way I knew how ‚Äď a WordPress plugin.

Having helped build Jetpack’s Publicize feature all those years ago, it wouldn’t be my first rodeo, but now that it’s working, it just might be my last.

Introducing Publishiza, the shittiest way to Tweet your thoughts while also blogging about them too. Check out https://publishiza.com to learn more.

WordPress, to me, is an independent publishing platform. It grants me the right to complete &¬†total autonomy when¬†it’s desireable, but also allows me to group up with others when that makes sense too.

An article from The New York Times puts into a nice perspective why I think WordPress is really important to the future of the web.

Every pirated music video or song posted on YouTube or Facebook robs the creators of income, and YouTube in particular is dominated by unlicensed content. Google’s YouTube has an over 55 percent market share in the streaming audio business and yet provides less than 11 percent of the streaming audio revenues to the content owners and creators. But Facebook, which refuses to enter into any licensing agreement on music or video, is challenging YouTube in the free online video and music world.

And this…

‚ÄúThey are taking all the money,‚ÄĚ he noted. ‚ÄúThey have algorithms we don‚Äôt understand, which are a filter between what we do and how people receive it.‚ÄĚ

I don’t think this is anything new ‚Äď tides will shift, and new technologies will emerge to try and help with distribution of content ‚Äď but it’s scary to me now that so much of what’s being published funnels out into our enormous world through only a few hoses.

There’s more money in the world changing hands than every in recorded history, and I suppose¬†it’s always been this way ‚Äď content creators are starving artists and content distributors are benefactors ‚Äď but¬†the trickle-down distribution of wealth continues to run perpetually dryer vs. wetter.

I think in the WordPress space,¬†companies like Envato are undervalued. Their operating costs are surely not as low as one may think, yet they continue to pay out millions of dollars to digital artists & creators. I think there may be room for more Envato’s to carve out their own niches, and WordPress plugins like Easy Digital Downloads and WooCommerce are the long-term solutions for people hoping to have a sustainable independent lifestyle.

In 2014, Matt Mullenweg challenged the WordPress community to volunteer 5% of their time towards open-source and WordPress.org.

A few months later, I ran a successful fundraising compaign that allowed me to donate 6 months worth of time towards BuddyPress & bbPress.

For 2017, and hopefully with your help, I’m going to try something a little different that I’ve nicknamed: ūüíĮ‚ąě.

My goal is be a fully funded independent ambassador for WordPress & the surrounding initiatives, backed by many of the best companies who continue to push WordPress beyond its limits on a daily basis.

Practically speaking, I imagine this to work like a monthly retainer to work on WordPress core and Dotorg. Someone pays, say, $7500, and I get to say January 2017’s progress was brought to you by Pagely, February by GoDaddy, March by CrowdFavorite, April by Jetpack, May by WebDevStudios, June by GiveWP, and so on, forever.

(The structure is a bit TBD. Maybe it’s weekly rather than monthly, or quarterly, or something else entirely. Hopefully you get the idea.)

It’s like a podcast, but instead of airtime ad placement, it’s coretime leadership, contributor relations, and issue management.

Some of y’all who think this is somewhat unconventional may say ‚Äď Why not just be a full-time employee?

That may end up happening ‚Äď it may actually be the smarter move in the long run here ‚Äď but I think there’s high value in being an independent, nonpartisan thought leader that *wants* to help everyone¬†navigate the ins & outs of contributing to open-source, more specifically WordPress, et al. Since that’s where I’ve seen the most success, I’d like to put maximum effort into continuing that trend; if it’s not sustainable, then we learned something together and try something different.

This weekend at WordCamp US, I’ve had the pleasure of meeting with several folks who love the idea and are‚Ķ deeply‚Ķ interested in making this work however they can, up to and including growth beyond myself for other individuals who are able to provide more value as an independent voice.

If you’d like to help, please email me. If you have questions, concerns, criticisms, or feedback, please get in touch somehow.

Thanks for having an open mind, and for considering investing in the future of something that really, truly means a lot. To the folks who are already onboard, I’m so excited you are willing to¬†give this a try with me.

P.S. Have I mentioned how much better WordPress is on mobile? I wrote all of this from my iPhone without any issues, which is still so incredible to me. Great job, mobile team!

P.P.S. If you’re unfamiliar with why I think I can do this, I’ve contributed to every major WordPress release since 2.5, am a member of the security team, meta team, and have helped lead BuddyPress & bbPress since 2009. I love WordPress and want to help it continue to grow in whatever direction it needs to take. I’m going to do it anyways, and with your help we can do it together.

Normally I’d start with a history lesson, but for this I’ll cut to the chase since most of you reading this already know what I’m talking about‚Ķ

Projects like Django function & thrive without their REST API in every core installation. Not being in core means the component maintainers are able to develop independently of any core release cycle ‚Äď both faster & slower if need be.

Externally facing API’s are a mouse on a wheel. Just as XML-RPC is ready, REST is here to take its place. Once we are all comfortable with REST 10 years from now, something else will replace it to, and so on.

WordPress is of the size and popularity now where building wp-admin into a hybrid of REST & AJAX calls will be confusing to future contributors.

Going back and rebuilding wp-admin to be fully REST is a thankless job, and would take ten years to complete at the current pace. It’s more likely wp-admin would be replaced, which would break backwards compatibility with existing plugins. It’s akin to hiding Windows 3.1 under the 95 skin, and abandoning all previous applications. No way.

Many contributors see a core-merge as a natural graduation path to their efforts, for lots of things ‚Äď even small things like ideas and recommendations. We get attached to them, and define success & ultimate acceptance as the end result being gifted to the world via WordPress core.

That path is unrealistic for most, and unhealthy for WordPress in the long run. This is why plugins exist, and why plugins are awesome. The REST plugin is one of the most awesome ones, and it should stay that way so that the team of passionate people who love it, can continue to love it by totally avoiding core conflict.

Maybe it gets bundled with WordPress like Akismet does, but it needs to be activated to work. Maybe it comes active by default, and can be disabled or deleted for those who will never need it.

A merge into core, right now, I think is still not the right call.

For years I wanted, pleaded, planned, and architects the concept of user profiles and activity streams on WordPress.com. I still think it would have been enormously popular (and still can be) but looking back, I understand now how my ideas didn’t match Matt’s overall vision and needs for a product that he designed originally.

I think the REST API is a similar thing.

This project is so huge and the media attention so great, it is now bigger than itself. It’s almost less about the code/team/effort, and more about the spiritual support and endorsement of those efforts. Maybe that’s what I’m turned off by right now ‚Äď that it’s been so long, it’s impolite to suggest a core-merge of this code isn’t the right thing to do.

FWIW, even if it had 100% API coverage, I still don’t think I’d want it on in core WordPress by default, because spiritually I’m really vibing on features like this being optional plugins right now, and I’m afraid that a core-merge will complicate the momentum and progress of the REST API core code once it’s in.

My $0.02. Don’t mean to offend. And I’m down to help anyone no matter what the end results ever are. ūüíú
P.S. I wrote this totally from the iOS app, which was a pretty pleasant experience. Great work, from a team of developers outside of WordPress core.

This weekend I’ll be driving down to WordCamp Chicago in my BRZ. It’s about 2 hours total from home to hotel, and I could use a playlist for the drive there and back.

I’ll be in Chicago chatting about the differences between BuddyPress and bbPress, and I’ll probably talk a bit about Prince, too. I’m on stage at 3pm, and I’d love it if you stopped by to say hi. ūüíú