Create 2022.12 O3DE Rev-the-Engine Report.md

Signed-off-by: amzn-yuyi <[email protected]>

user-research/2022.12 O3DE Rev-the-Engine Report.md

@@ -0,0 +1,214 @@
+# 2022.12 O3DE Rev-the-Engine Report
+
+## Summary
+
+This document outlines the successful completion of the AWS Rev-the-Engine (RTE) Kingpin Goal of delivering passable UX scores for 5 out of 9 workflows in Open 3D Engine (O3DE). All workflows, aside from Multiplayer and Packing & Deployment, had Satisfaction Scores over “3” on a 5 point scale (i.e., passable). 
+
+## Introduction
+
+Improving O3DE’s user experience is a critical step in driving customer acquisition, retention, and advocacy. The user experience of O3DE, an open source and real-time 3D engine, was assessed through attitudinal benchmarking surveys in June 2022 and December 2022. By tracking self-reported satisfaction over time and across major workflows within O3DE, we’ve identified opportunity areas and measured the impact of improvements made to the product throughout 2022.
+
+To achieve the goal, our team made a series of improvements to O3DE, including October’s 22.10.0 release. Some improvements highlighted in the 22.10.0 release are:
+
+* Onboarding and Collaboration: Remote projects, templates, and networked asset cache have made it easier to onboard and collaborate with other team members.
+* Multiplayer: With improvements to the client-server connection, debugging, and network spawning, setting up multiplayer is more straightforward than before.
+* Artist workflow: Improvements for artists include animation import, root motion extraction, motion matching, and DCC studio toolkit.
+* Usability: We’ve made usability improvements to various features including Viewport Interaction, Gem Creation Wizard, Asset Browser, and hot reloading of assets.
+* Terrain performance: The new O3DE Terrain system enables developers to start working on larger-scale worlds with significant performance improvements for both editing and runtime/rendering. 
+* New features: Sky Atmosphere and Stars components.
+
+## Methods
+
+Survey participants were members of the O3DE community who reported having recent experience using O3DE. Participants were recruited from O3DE’s Discord, Twitter, LinkedIn, email, and AWS’s Slack from December 9-16, 2022. One participant was selected with a random number generator to receive the $100 Amazon.com gift card as a thank you for their time. A total of 76 participants were included in the analysis.
+
+An online survey created on Alida was administered to assess O3DE users' attitudes towards the O3DE. The survey asked participants to respond to demographic questions (e.g., years experience with game engines), specific workflow questions (i.e., ease of use, meets needs, satisfaction), and holistic O3DE questions (e.g., how likely they would be to recommend O3DE to a friend). The survey was designed to measure participants' overall satisfaction with O3DE, as well as their perception of various aspects of O3DE's usability, functionality, and design. 
+
+We structured O3DE’s 40+ features into 9 common game development workflows, and users were asked to provide their sentiments and feedback only to the workflows they have used in O3DE. These 9 common game development workflows include:
+
+*  Onboarding (e.g., downloading O3DE, installing O3DE, creating a project, customizing gems) 
+* Experience Design (e.g., prefabs, components, game mechanics, physics, camera, input system, script canvas, viewport, play mode) 
+* Actor Development (e.g., setting up actor assets, animation system, motion events, ragdoll, hit detection) 
+* World Building (e.g., dynamic vegetation system, landscape canvas, prototyping with White Box Tool) 
+* Look Development (e.g., materials, textures, lighting, PostFX) 
+* Multiplayer Development (e.g., multiplayer content and networking for a game) 
+*  Team collaboration (e.g., versioning control tools, sharing, collaborating in the same file, remote project) 
+* Engine Extension (e.g., building new gems that contain new tool UIs, components, script canvas nodes) 
+* Packaging & Deployment (e.g., packaging project files, deploying to other platforms, sharing with others for play test) 
+
+For each workflow, users were asked to rate their agreement with the following statements.
+
+* Ease of Use:  O3DE’s [workflow] met my needs. 
+* Needs Met:  O3DE’s [workflow] was easy for me to use. 
+* Satisfaction:  I am satisfied with O3DE’s [workflow]. 
+
+Descriptive statistics were used to summarize the results of the survey. Open-ended responses were coded and analyzed using thematic analysis to identify common themes and patterns in participants' responses. Inferential statistics were used to determine if there were significant differences in user experience scores based on various demographic factors, such as years of experience with 3D engines or amount of experience with O3DE. To compare the results of this study to previous research, we reviewed data from the July 2022 benchmarking study. Participants’ responses are kept confidential and only shared in aggregate.
+
+## Results
+
+The effect of AWS Affilitation on O3DE Satisfaction was examined because 30% of participants were AWS affiliates (i.e., employees and contractors) and it was hypothesized that they might respond more favorably toward O3DE. The results of the analysis revealed no significant relationships between AWS Affiliation and Satisfaction on Any Workflows. Furthermore, the results revealed no significant relationships between AWS Affiliation and other factors (i.e., O3DE Usage, likelihood to recommend O3DE, team size, O3DE trustworthiness, likelihood of using O3DE again, O3DE’s website helpfulness, learning new skills from O3DE usage, ability to resolve issues with O3DE). As a result, AWS affiliates’ responses are included for the following analyses. This decision was made because the results suggested these two groups were not from different populations.
+
+The survey results suggest that on average O3DE meets users needs slightly better than it is easy or satisfying to use. This suggest that users feel slightly more favorable about O3DE having the right features, compared to how easy and enjoyable it is to use those features. However, all of the scores are hovering around 3 on a 5-point scale which indicates room for continued improvement.
+
+<img width="449" alt="image (12)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/14be4d5c-4bc7-4d6d-95fc-f0a911aad58a">
+
+
+Satisfaction remained relatively similar to scores collected in the June ’22 Survey. There were statistically significant increases in satisfaction scores for World Building, Multiplayer, and Engine Extension. These indicate improvements made to the engine improved how participants felt about the workflows and/or that over the last 6 months participants using those workflows became more comfortable with them. One participant wrote in, “It’s getting better, better, and better.”
+
+<img width="438" alt="image (13)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/bf1352a7-a93b-44a8-b312-a98db67f60bf">
+
+Five workflows had Satisfaction scores above 3 in June 2022 and this was raised to seven in December 2022. These results suggest that Multiplayer and Packaging & Deployment are candidates to focus on in the future.
+
+| **WF** | **June Satisfaction**	| **December Satisfaction**	| **% Change**
+|--|--|--|--|
+| Onboarding | 3.4	| 3.4	| 0
+| Gameplay | 3.1 | 3.3	| 6%
+| Actor Dev	| 3.2	| 3.1	| -3%
+| World Building | 2.9	| 3.3 |	13%
+| Look Dev |	3.3	| 3.5	| 6%
+| Multiplayer |	2.4 |	2.9	| 20%
+| Team Collab. | 3.3	| 3.4 |	3%
+| Engine Ext.	| 3	| 3.4 |	13%
+| Packaging & Deploy.	| 3 |	3 |	0
+|	**Final Score** | **3.1**	| **3.3**	| **6%**
+
+Notably, Multiplayer and Actor Development both had ~.05 point difference between Meeting Needs and Ease of Use Scores. This could be interpreted as participants believing that the features they need exist, but need to be made easier to use. Otherwise, the 3 scores corresponded to each other closely.
+
+<img width="448" alt="image (14)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/b2d2999a-6824-4228-81cc-59427b8f5b2d">
+
+On average, scores for trustworthiness, likelihood of using O3DE again, website helpfulness, and learning new skills through O3DE were above 3 out of 5. However, Ease of Resolving Issues had a mean score of 2.76 putting it below the others. This could be an opportunity to dig deeper into what is causing their perception and to design some solutions to it.
+
+<img width="449" alt="image (15)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/1bc28907-76fb-41d3-b0de-7c303422deec">
+
+
+Despite meeting passable scores on 7 out of 9 workflows, O3DE performed worse than Qualtrics’ industry average in Net-Promoter Score (i.e., the likelihood that a participant would recommend O3DE to a friend or colleague). The -15 score can be interpreted as saying: There are 15% more people spreading negative word of mouth than there are spreading positive word of mouth about O3DE. This highlights the need to continue focusing on improving user experience for O3DE in order to drive referrals and adoption.
+
+<img width="443" alt="image (16)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/83440ed3-4586-4bff-982c-5289a5393a46">
+
+Participants raised some specific pain points and suggestions for improvement in the write-in comments. For example one participant said, “It's ultimately a complex piece of software with a lot of features and functionality. However, the first time experience from zero to hero should be shorter than it is.” Other suggestions include:
+
+
+* Stability (e.g., engine crashing, changing APIs, bugs, less vulnerability to errors regarding Editor / Game mode components, script canvas so that don’t need to run game for changes to take effect)
+* Ability to recover from errors (e.g., documentation, live support)
+* Improved installation and setup (e.g., Mac, Linux, offline environments, lower system requirements, remove build project before opening it)
+* Ability to ship games with release packaging  (e.g., avoiding Terminal/CMD usage)
+* Faster time to iterate and be creative through better learning and support (e.g., demos, tutorials, templates, less flexible version of the features, bigger community, updated and complete documentation, less steep learning curve, less complexity, Assets, examples and tutorials for robotics, educational material is too advanced)
+* Performance (e.g., slow build times when using source engine, long time to open material editor, slow start-up, frames per second, visual fidelity)
+* Usability (e.g., unify tools and workflows, explanation of icons, confusing menus, lack of visual tools, grabbing borders of windows difficult, moving building project process to be done within editor rather than through command prompt, navigation in editor, Especially for actions outside of the engine, converting a project to a gem for example, Project organization, in-engine file manager to select thing)
+* Scripting (e.g., improve Lua, Lua doesn’t scale, C++ is a pain, more C++ functions exposed to Script Canvas, Class references from the Lua editor only show properties and functions/methods, for C++ components could be more granular, similar to Unreal Engine's, support c#)
+* Improved regression testing
+* Platform support (e.g., mobile, Easier way to deploy projects on Android. Current setup is prone to errors due to how many commands and third party software is needed.)
+* Modularity (e.g., Improved ability to add new systems or tools to the engine, understand the more advanced details of O3DE Bus System & how entities tie into the engine, the wanted customizability is interesting but always little bit frustrating,)
+* Asset handling (e.g., importing scenes, FBX import pipeline to be able to preview what to import)
+* Continued improvements of existing systems (e.g, Dynamic Vegetation/Landscape Canvas/Terrain, Easier way to shape the terrain, multiplayer capabilities and network optimization options, Physics for simulation, Gradient system is not very fast nor easy to grasp, animation editor creating extra actor IDs which prevent animations from playing in the animation editor, Realistic terrain generation in planetary size as openWorld not small levels, modifying the Rendering-Pipeline easier. An Imperative method like e.g. nvidia falcor uses would be way easier to use than having to manually edit several json files.)
+* Lack of essential features (e.g., like render to texture from cameras, particle systems, and a shader node system, Camera pipeline, image acquisition. Needs to be more suitable for simulating camera sensor, particle editor, A fps/3pp character as compatible as metahuman from u5)
+
+
+Onboarding and Gameplay Design workflows were used by the highest number of participants, while Multiplayer and Team Collaboration were used by the lowest number of participants. Onboarding being one of the most common workflows might be counter-intuitive because we surveyed existing users (67% of participants used O3DE 20+ times), but Onboarding includes tasks like project creation or installing a new version of the engine which need to be done throughout a users’ tenure using O3DE. About, ~30% of participants used 1-2 workflows, ~30% 3-4 workflows, and ~30% used 5 or more workflows recently. This suggests that a third of O3DE users are using a wide variety of features and, consequentially, may be less experienced in any specific features. This supports the need for workflows to be well-integrated with each other and for O3DE’s 40 features to be intuitive to use (albeit difficult to master).
+
+<img width="446" alt="image (17)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/c2681d76-2eea-4c0f-a467-82b158fe0039">
+
+![image (18)](https://github.com/o3de/sig-ui-ux/assets/75449675/d06ff67d-07b7-401a-8f4f-6b7714ffa4cc)
+
+
+The data suggested that AWS affiliates trended toward having more experience with 3D engines compared to non-AWS affiliates using O3DE. This highlights the importance of designing the engine to be used by users who are less experienced with engines than the engine creators themselves.
+
+<img width="443" alt="image (19)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/ee09dab8-5111-4ba6-b37d-18e342c9eb23">
+
+
+O3DE aside, Unity and Unreal were the most popular engines used by participants; and 77% of participants used another engine recently in addition to O3DE. This could indicate that O3DE is not fully meeting users’ needs so they need to use another engine, that they are comparing engines to find the best fit for their needs, or something else.
+
+<img width="445" alt="image (20)" src="https://github.com/o3de/sig-ui-ux/assets/75449675/53397dec-72fa-4232-a947-50021381eb03">
+
+![image (21)](https://github.com/o3de/sig-ui-ux/assets/75449675/1584f32a-475a-4d73-88d4-ebe15ee29e13)
+
+
+Also, it is important to continue tracking these attitudinal scores over time to identify any potential areas for improvement. By focusing on areas where scores are lower, we can prioritize efforts to address any issues or pain points that users may be experiencing and ensure that O3DE continues to meet the high standards of our customers. We will be taking steps to address these issues in order to further improve the user experience with O3DE. In conclusion, the results of this UX benchmarking survey show that O3DE has made significant improvements in several key attitudinal metrics over the last year and there is still room for improvement.
+
+
+## Recommendations
+
+Based on the suggestions received from the open-ended responses of the survey, the O3DE team should consider the following improvements.
+
+Stability:
+
+* Invest in quality assurance efforts to identify and fix bugs in the engine.
+* Work to ensure that changes to APIs do not negatively impact stability.
+* Develop more robust regression testing tools and processes to ensure that changes to the engine do not negatively impact existing features.
+
+Error Handling & Learning:
+
+* Provide more robust documentation and live support resources to help users recover from errors.
+    * Recommendation:
+        * Messaging guideline initiative to improve the overall quality and sensitivity of the error messages. (need to find ticket)
+            * https://github.com/o3de/o3de/issues/3498
+            * https://github.com/o3de/o3de/issues/3509
+* Invest in updating and improving documentation to make it more comprehensive and accessible. 
+    * [Epic] Improving the IA and search functionality of the doc site
+    * [Epic] Context-based help: Provide help content where user needs them.
+* Consider creating educational material that is more suited to beginners.
+    * [Epic] Project begin
+* Develop more comprehensive demos, tutorials, and templates to help users get up to speed more quickly.
+    * MKT SIG has been doing some analysis on the tutorial/video coverage.
+    * [Epic]: Improve the discoverability of the existing content and game jam projects
+        * Create a page with all the demos and game jam projects on GHI to improve SEO
+        * Consider showcasing the demos and game jam projects on O3DE.org for visibility
+* Expand the O3DE community through outreach and engagement efforts.
+    * @MKT SIG
+
+Installation and Setup (Bhanuja Sanghavi):
+
+* Streamline the installation process to make it more user-friendly and accessible, particularly on Mac and Linux systems.
+* Reduce system requirements to make O3DE more accessible to users with lower-end hardware.
+* Develop more intuitive tools for building and packaging projects, to minimize the need for Terminal/CMD usage.
+
+Performance:
+
+* Invest in optimizing the engine to improve performance in areas such as build times, start-up time, and frame rate.
+    * @sig-core
+* Develop tools and techniques for improving visual fidelity.
+    * @SIG-graphic-audio
+
+Usability:
+
+* Work to unify and streamline tools and workflows within the engine.
+    * Improve AP experience to streamline with the editor Joshua Rainbolt
+    * Easy access and discoverability for plug-ins and 3rd-party tools
+        * DCC tools integration Jonny Galloway
+    * BJDS
+    * 
+* Improve the clarity and consistency of menus, icons, and other user interface elements.
+* Develop more visual tools to help users understand and interact with the engine.
+* Consider moving more processes into the editor to minimize the need for command line usage.
+
+Scripting:
+
+* Work to improve the performance and scalability of Lua within the engine.
+* Consider adding support for additional scripting languages, such as C#.
+
+Platform Support:
+
+* Work to improve support for deploying projects to mobile platforms, including Android.
+
+Modularity:
+
+* Develop more intuitive and flexible tools for adding new systems and tools to the engine.
+* Provide more resources and documentation to help users understand the O3DE Bus System and how to work with entities within the engine.
+
+Existing Systems:
+
+* Streamline the process for importing scenes and assets, including the FBX import pipeline.
+* Continue to improve and optimize existing systems such as the Dynamic Vegetation and Landscape Canvas.
+* Develop more intuitive tools for shaping terrain and optimizing multiplayer performance.
+* Invest in improving the animation system and other areas of the engine.
+
+Essential Features:
+
+* Develop essential features that are currently lacking, such as a particle system and a shader node system.
+* Improve the camera pipeline and image acquisition tools to make them more suitable for simulating camera sensors.
+
+
+
+
+
+
+