Possibilities of a CMS in the Spatial Computing Future

In my role at WP Engine, I have the opportunity to travel the world serving as an ambassador for new and interesting ways of using WordPress. I was recently in Raleigh, North Carolina, at All Things Open talking about the possibilities of a CMS in the spatial computing future. This was a lofty topic for 45-minute talk, so I thought I’d share a more in-depth look at a new way to build Extended Reality (XR) experiences with WordPress using a headless approach.

The Headless Craze

I think many of us see headless primarily as a way to make our sites faster and decouple them from the CMS that controls our content. While true in some cases, this is probably not the most compelling lens to view this concept in when we consider what this means for native apps and Augmented, Virtual and Mixed Reality, more broadly defined as Extended Reality.

In order to understand how WordPress can power XR experiences we must take a step back and understand what key features developers and designers desire in interactive applications.

What is the core appeal of a CMS?

Zooming out and looking at what a CMS is at its core highlights a few key features that make my future-looking imagination race. A CMS can be broadly defined as:

  • post/page data
  • meta associated with data like users and posts
  • a secure user system
  • a rich content editor
  • extendability through plugins/themes/etc. 

WordPress shines brightly across this entire spectrum and, pairing that with open-source flexibility, you can build whatever you can dream. User systems and content are key requirements in interactive and media-rich applications such as XR. An open-source solution like WordPress gives developers greater freedom and more control over the source code and the software meaning they can tweak it to perform how they want, unrestricted by licensing. Another level of freedom was achieved when the new Gutenberg editor was merged into WordPress 5.0, further democratizing the power of the world’s most popular CMS.

Possibilities with the New WordPress Editor

We’ve spent a lot of time thinking through the front-end benefits in a unified way to control markup using blocks. I think one aspect of the new WordPress editor that has largely been overlooked in front-end development is the data structure of the blocks themselves. 

Pulling data via the REST API typically returns the final markup of a post. This makes it hard for non-html based applications to parse or use the data. You can certainly sanitize the final output, but it doesn’t provide confidence going forward and often removes important context.

Before a post renders markup in the front end, there are valuable attributes used to determine that final markup. These options are often just as useful to a non-browser application as they would be for a browser-based experience; it’s just a matter of making sure the content is sanitized for your purposes.

Let’s imagine we have an audio block with the attributes of src, loop, autoplay, etc. Those attributes are just as important to a non-browser application that expects the same decisions to be made at runtime. By querying a Gutenberg post, one can pull the audio file and also the attributes of how it should behave, leading to the application being unified with browser behavior.

In the not too distant future, the physical and digital worlds will continue to blend together. As younger generations increasingly interact in ways other than through screens, keyboards, and mice, we should probably begin to create content with that future in mind. Do we really want all that markup saved? Is it even important? If it is, maybe that attribute should be saved, keyed, and exposed via JSON. Markup is really only useful to the front-end rendering.

Sound-Only AR Blog Reader Concept

Bose is pioneering an interesting approach to augmented reality. They have quickly been updating their product offerings to include 6 Degrees of Freedom (6DoF) sensors inside their headphones and sunglasses to power sound-only AR experiences. This gives us three important mechanics for XR; 6DoF head tracking, private sound experiences, and gesture recognition. I decided to use this technology to illustrate the potential in the data structure of Gutenberg blocks. 

photo of Bose AR sunglasses

Using the Gutenberg Object Plugin open-sourced and built by Roy Sivan and contributors, I was able to expose the attribute data in a REST friendly way. I was then able to shift my focus on building mechanics and an MVP application to read a blog without sight. The application puts buttons in 3D space around the visitor. To the right, a “Next Post” button. Directly in front of them, a post body button that plays back a reading of the post data. The voice-to-text technology was made possible by using the Amazon AI Plugin for WordPress

When double-tapping on the frame of the glasses while facing the post body, a voice prompts the visitor to use a nod gesture to playback the post audio. Within 24 hours of sketching the idea, I had a working proof of concept and I attribute that largely to not having to figure out how to prepare my data. Blocks and JSON did it all for me!

The following image is a sketch of the concept along with the final product. 

sketch of Bose AR blog reader.
final design of Bose AR blog reader prototype.

Blocks in Traditional Augmented Reality

Spatializing blocks was interesting to me, so I built another proof of concept that demonstrates how 3D prefab objects in a game engine can be associated with 2D blocks on a webpage. Below is a screenshot of a Gutenberg post rendered around my hotel room.

Example showing relation of website and AR experience.

While a literal 1:1 representation of a webpage in AR illustrates that we can couple our design decisions in and out of a browser, I think we can go a step further. A block can represent many things; so why not use custom blocks to abstract 3D ideas?

Non-Standard File Types and Abstracting 3D Objects

HTML has a finite list of file types allowed out of the box. That is not to say we are limited to only those options. WordPress is flexible enough to extend to others by simply adding allowed types. 

I’ve been building a music game called Broken Place that allows players to interact with real-time music in AR. The sound engine is running an open-source programming language called Pure Data (PD) that has a very specific file type that only it can read. In WordPress, I am managing post content in a custom post type called “songs” where players can upload their PD patch. Players simply log into the website as a contributor user role and upload their patch. The song post is queried from the native app and the patch is loaded. 

An Augmented reality synthesizer on a beach

The content submission options open the potential of allowing users to design their own layouts for music interfaces. Imagine in the graphic below, that each of the 3D knobs pictured were associated with a Gutenberg block called “Knob” and each of those knobs have an attribute associated with the sound message being sent to the audio engine. Content creators could determine how the sound messages are mapped in a grid using the column block. When you nest blocks within a column block, the REST response also nests the objects as children of the block and logic can be applied using the JSON structure.

A synthesizer interface in VR

Narrative Design Potential

In post data, we are telling stories; some informational, some personal, but they serve as thought capsules. I dream of a future where posts are much more than just text and images on a screen. Spatializing content provides amazing opportunities for narrative design. I’ve seen this potential represented in the AMP Stories and hope that structure could help inspire stories told within 3D physical spaces. 

image of AMP Stories block

Imagine the same post structure of AMP stories, but for 3D spaces that you can drop in your home or office. Mozilla Hubs is leading the charge in collaborative XR spaces open to all devices and I see interesting potential in combining a WordPress narrative design for Mozilla Hubs to allow shared storytelling experiences. 

I’m excited to continue exploring new ways of using headless WordPress to power XR experiences and help usher in an entirely new era in how we interact with and create digital content.

Would you like to write for WP Tavern? We are always accepting guest posts from the community and are looking for new contributors. Get in touch with us and let's discuss your ideas.

7 Comments


  1. Some interesting insights but also somewhat flawed as:

    1. This content will always be dear delivered over the web.
    2. The only standardised way if navigating that will be a browser (which are are improving rapidly)
    3. That browser will consume HTML and successor HTML like markups (eg Svg, Vrml)
    4. The only reliable way of delivering this is through server side rendering, javascript + api frontend are inherently brittle, non accessible, and hard to index (robots are important too)

    This all means headless cms are a dead end.

    However the technologies mentioned in the articles are a glimpse of the future and WordPress needs to enhanced to deliver a broader range of content and functionality. But the changes needed are back end things like the taxonomy architecture, post status api, object relationships, notification queues, and database schema and support.

    Report

    Reply

    1. Hi Peter! Thanks for reading. I’m trying to understand where the flaw is in regards to the prototypes and ideas above. I always love discussing this topic, so I’ll address your points there relative to XR and native apps. :)

      1. This content will always be dear delivered over the web.

      Absolutely no argument from me on that. No matter the tech, we will always need some way reach the web, but currently we focus a lot on end content being saved as HTML. That limits the extendability of the content. More on that below…

      2. The only standardised way if navigating that will be a browser (which are are improving rapidly)

      3. That browser will consume HTML and successor HTML like markups (eg Svg, Vrml)

      I wholeheartedly believe browsers will remain THE dominant tool for displaying web content. I don’t think many folks can argue against that. In my illustrations above, I’m simply showing how markup is an obstacle currently, and how defining context, much like how Gutenberg does with attributes, is important. This is not to say we shouldn’t render the content down to HTML in the front end any differently. The idea here is to unify both your content and design decisions.

      The one thing developing still are standards between web and XR tech. I encourage you to check out OpenXR ( https://www.khronos.org/openxr/ ) and the standards they are attempting to set on a hardware level. This all eventually goes back to the browser model however, context for content is still needed and different in every environment.
      XR is throwing so many new environments into the mix (MR/AR/VR/etc) and it’s hard to stay flexible with the tech advancements every few months. This notion of using Gutenberg blocks to expose only context and content is important to stay nimble with the changes.

      I hope my points above do not support an idea that we will one day be exclusively fully headless on the browser level. My illustration here is only that we are already primed to unify our content to native app experiences; and currently, the XR market is heavily positioned in native apps. I think Aframe and Mozilla Hubs are doing an excellent job of showing the potential and need for browser support, which is in turn influencing those OpenXR standards.

      Report

      Reply

      1. I don´t disagree with anythingin your comment but headless WP is generally defined as pluginless with a javascript front end.

        I think this is flawed, server rendering the markup (html or it successors) is the future. With improving bandwidth, server speeds, service workers, smart caching etc the trends are wordking against javascript front ends, the cool kids are not so smart ;-)

        And for server rendered markup the traditional wordpress plus plugin architecture is perfect. Even if wordpress itself could do with some backend improvements to handle new types of data and experiences.

        Report


  2. VR, XR or whatever term you want to describe it will actually “date out” the current internet browsers and platform similar to the the introduction of web-browser based internet did to the traditional books, cds, dvds and even to some extent movie theaters.

    WordPress is nothing extraordinary. Actually, laravel cms is much better in comparison to wordpress. Introduction of “Gutenberg” plugin (within ” because its clearly a double-speak. the plugin has nothing to do with gutenberg) can’t change the fact that wordpress is a php blogging platform at its core.

    The backend software such as mariadb, mysql will go through some kind of mutation and survive the VR future. But current web-browser based front-end software will be part of the “history”.

    Pulling data via the REST API typically returns the final markup of a post. This makes it hard for non-html based applications to parse or use the data. You can certainly sanitize the final output, but it doesn’t provide confidence going forward and often removes important context.

    The keyword of the whole text beneath the Possibilities with the New WordPress Editor is data storage and retrieval.

    The thing is WordPress stores data in a third party database software such as mysql or mariadb. What’s the point of logging into a website through a traditional web-browser, upload a video, text or an audio then to watch, listen or read it via a VR headset?

    I believe the future of VR would be similar to a current internet only in 2 ways.
    1. an OS which will run on VR set of the user
    2. Number of Nodes or Virtual Reality Networks that would be connection end-points to the interested users.

    You might just log in through a one-click app installed on ur VR OS and get into the VR Node. Similar to cloud gaming platforms popping up in the last few years. Even google got into this business. You don’t need to download any huge games (data) to play the game which is perfect for current vr headsets as there is no need to go berserk and invest $ to r&d for petabyte ssds size of a fingernail. You won’t need to install anything on user’s end unless the user wants to. (no browser caches in current terms)

    For example, Unity, Unreal, Lumberyard (Amazon) or even cocos-2d has much more potential in VR, XR future than wordpress or laravel or any other php-based platform.

    Sorry for the long rant.

    Report

    Reply

    1. Ali,

      several of your foundational points are questionable or wrong

      1. Why will VR date out the browser? If anything browsers are killing apps atm and browsed are expanding into other areas. Browsers already are the predominant place to build software. Write once use anywhere.

      2. WordPress is both extraordinary? Well yes and no. It has somewhat flawed but has an unmatched community and plugin architecture. Seems as good a place as any to build the future. People already are…

      3. Any VR system will need a standardised way of navigating and a HTML like language delivered via the browser seems as good a way as any it will need to be organised. Path dependence is a thing and single vendor solutions will never become a standard.

      Report

      Reply

      1. Pete,

        With all due respect, I have just stated my understanding/vision of VR future that has yet to come. Those points weren’t prophecy of any kind. They might be totally wrong or could picture the future in perfect sense.

        Nevertheless, I would like to reply to your statements hoping it would lead to a constructive discussion. Moreover, the last comment that I had made were written in a fast manner and reading it again I see that it had lots of mistakes in sentence structure. I will try my best to express my thoughts in proper English. Just to let you know, I’m not a native speaker.

        1. Why will VR date out the browser? If anything browsers are killing apps atm and browsed are expanding into other areas. Browsers already are the predominant place to build software. Write once use anywhere.

        Let me put it this way. World of Warcraft is played via keyboard and mouse. However, an advanced VR headset with remote controllers or even a futuristic VR cabin could/will eliminate the keyboard and mouse while the player(user) will have much more realistic experience of the game. Such games, movies or any other form of content can’t be visualized in 2d screen which browsers are meant to be used in. The whole point of VR headset is getting rid of traditional monitors in favor of the headset.

        Few days I played a racing game on a vr headset with proper wheel and moving car platform. It felt very real to me. The sound, visuals and even the platform I was sitting on moved according to the movement of the car in the game. It just loses the point to run that game on a browser instead of its own app because it would be inefficient, unproductive and unreliable. Because VR headset doesn’t need the browser. You can directly access the OS.

        As for the apps vs browser. The first search result on google tells more than I could write down to explain in few pages.
        https://jmango360.com/wiki-pages-trends/mobile-app-vs-mobile-website-statistics/

        Moreover, I’m managing few top alexa 10k websites for customers. Almost all of them have 70-85% percent of daily visitors either using smart phones or tablets. The number of visitors using desktops/laptops have 13-15% share of daily visits.

        2. WordPress is both extraordinary? Well yes and no. It has somewhat flawed but has an unmatched community and plugin architecture. Seems as good a place as any to build the future. People already are…

        In comparison to wordpress, there are quite a few databaseless CMS (less cpu, memory usage, no php engine, no database engine, pure html etc..) which are much better for ordinary day-to-day blogger’s needs. The only thing that makes wordpress much better option to ordinary bloggers/website owners is having a huge theme and plugin developer community (paid or free). Themeforest or Codecanyon are as influential in wordpress’s popularity rise as Matt who had coded the wordpress itself.

        WordPress is very ordinary. It’s not extraordinary. I agree that it has an unmatched community and plugin architecture. However, with the forced introduction of Gutenberg Plugin to the core I think its no different than Saddam’s Iraq or Putin’s Russia. There are a handful of people deciding the future of the whole platform for millions of users. According to the Venturebeat’s 2018 statement, WordPress powers 30% of the web. The introduction of Gutenberg Plugin to the core was decided by less than 100 developers/contributors under the vision of “democratizing the web”. Half of them are one way or another affiliated with automattic or 2 other companies (i don’t remember the exact names of the companies). There is a clear conflict of interest. Kim Jong-Un’s North Korea is more democratic in comparison the way wordpress.org is run if you take those statistics into account (100 or let’s say 1000 contributors deciding what the 30% of the websites on internet should use).

        Even the motto of “democratizing” the web is conflicting with the actual introduction of Gutenberg plugin to the wordpress core. People would be installing the Gutenberg plugin on their websites if it was useful to them. The best option to democratize the web is to share the simple core where people could add or remove functions by using plug-ins. I have yet to hear anyone talking about the number of downloads the Classic Editor has on wordpress.org and discuss why the stats are so high (5million+ and counting). It has to be noted that the Classic Editor was introduced after the addition of gutenberg to the core. At the moment, it’s on par with the most popular plugins in the repository if you check the number of downloads.

        What I’m trying to say is that => “Seems as good a place as any to build the future. People already are…” is an overstatement.

        I hope I’m wrong but I’ve grown up in a “third world country” and I’ve seen my share of “visionaries”, “leaders” and “prophets”. These days the wordpress is shaped by Matt’s “vision” of the future and I’m quite disturbed by this. I don’t know him personally and I haven’t had a chance to talk to him. However, I believe that he is a good fella. But as a good man he should stop forcing his vision to shape the future of whole community. I think the wordpress community owes to honor him but nobody deserves constant “thanks” for their contribution. We don’t thank, revere, congratulate or mention Indians/Arabs for the current numerical system we use each time we type 1,0,4 or 5. We don’t thank, revere, or even know the identities of the first Englishmen and Englishwomen who came up with the English language that enables you and me to have conversation today. Likewise, the same logic applies to Bitcoin, p2p, linux and so on so forth. The founders/creators of paid or open source software deserve credit but nothing more or nothing less. Let the people who use them to shape the software. This is my critique (from social point of view) of the unhealthy environment and state that the wordpress community has come to.

        From the software perspective. I believe, the wordpress developers and contributors should focus mainly on database structure than anything else. It is flawed and you can go through the critiques of this issue by well-known dba’s after a simple google search.

        3. Any VR system will need a standardised way of navigating and a HTML like language delivered via the browser seems as good a way as any it will need to be organised. Path dependence is a thing and single vendor solutions will never become a standard.

        Can you imagine something similar to Google Play/Apple Store to install apps on ur VR headset? Can you imagine having a much bigger, single repository for every public app ever made for VR being accessible to you after a Virtual Click away? If so then browser will be outdated.

        Report


  3. Ali,

    Re your points again some thing I agree with but mixed inwith things that dont make sense

    1. Yes VR is great but I’ve no doubt that browsers will support a common language for it. That language will be html like and wordpress would have no problem serving it.

    The browser IS killing apps I suggest you familiarise yourself with PWAs. Unless you need specific features you would be mad to build a native app in a couple of years.

    2. I largely agree, I don’t hate gutenberg but is a distraction from fixing some of the flaws in wordpress. I agree 100% with your point about the db structure btw.

    However despite all it’s flaws people are building the futures because of its plugin architecture and huge codebase. It’s used for far more that blogging.

    3. I certainly can imagine that future. However standalone apps don’t cut it now and won’t in the future. The web delivered through the browser is standardised, a website (Or a pwa), can act as an app but work similarly to otger sitees and urls make everything addressable and indexable. This is much more powerful than standalone apps and I think if anything 3D will make the need for web like features and standards even more important.

    I agree wordpress is not ready for this now and gutenberg is not getting us there. But wordpress has plenty going for it and “is a good a place as any” to start.

    Report

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.