Galibzon/cherry picks p24092 (#18668)

* Fixed Shader reload bug. (#18555)

* Fixed Shader reload bug.

There are three parts to this Pull Request:
1- Bug fix (Shader::OnAssetReloaded)
2- A new level called `ShaderReloadTest`, which contains assets + python
   script to validate the fix.
3- A new feature for O3DE that allows Gems and Game Projects to load custom
   PassTemplate assets without writing C++ code.

To understand the Bug Fix let's review the structure of ShaderAssets:
The `AZ::RPI::Shader`` class owns `AZ::RPI::ShaderAsset m_asset`.
This was the only asset that the Shader class was expecting to be notified
during Shader::OnAssetReloaded.
But, the ShaderAsset references several Root AZ::RPI::ShaderVariantAsset(s),
one for each Supervariant. The structure looks like:
```
AZ::RPI::ShaderAsset m_asset
  |
  |---> AZ::RPI::ShaderVariantAsset (Supervariant 0 Root variant)
  |---> AZ::RPI::ShaderVariantAsset (Supervariant 1 Root variant)
  | ...
  |---> AZ::RPI::ShaderVariantAsset (Supervariant N-1 Root variant)
```
When the user modified the Shader code, the AssetManager would dispatch
an `OnAssetReloaded` event for each asset in the chain shown above... But
the `Shader` class was only listening to `m_asset`. To make matters worst
the AssetManager wouldn't update the internal references to the Root
 ShaderVariantAssets.

The solution is two folds:
1. Register for OnAssetReloaded events for m_asset and for each of its Root
   ShaderVariantAsset(s).
2. When OnAssetReloaded() is called, keep track of all the new asset references
   in a dictionary. And only when the size of the Dictionary matches the total
   number of expected notifications then the Shader calls Init() again, and
   the dictionary is cleared and ready for the next time the user modifies the
   shader code.

In order to validate the bug fix, a new Level called `ShaderReloadTest` was
 added, which is a self contained set of assets and a Python script called
 `shader_reload_test.py`. To learn more about this script, please read the
 attached `README.md`. REMARK: This level is NOT a typical Python Automation
 Test, instead it is expected to be ran manually by engineers whenever they
 suspect that there are issues with Shader reloading.

Finally a new System Component called AZ::RPI::PassTemplatesAutoLoader was
added. It provides a very convenient opt-in service for Gems and Game Projects
to load custom PassTemplates. See `PassTemplatesAutoLoader.h` for details.
The O3DE wiki will be updated with description on how to use this service.

Signed-off-by: galibzon <[email protected]>

* Fixed race condition related with Material::m_shaderVariantReadyEvent. (#18595)

AZ::Event is not thread safe.
MeshDrawPacket::Update was calling
m_material->ConnectEvent(m_shaderVariantHandler);
from multiple threads.

The fix was to add a Mutex in Material class
that protects usage of Material::m_shaderVariantReadyEvent

Signed-off-by: galibzon <[email protected]>

* Fixes race conditions and deadlocks when loading levels. (#18609)

* Fixes race conditions and deadlocks when loading levels.

Fixes #18597

The most important change in this PR is that InstanceDatabase::EmplaceInstance
has been changed so it manages when to lock `m_databaseMutex`. In particular,
m_databaseMutex is not locked anymore during `InstanceData<Type>` creation
because creating resources like StreamingImage(s) is a multi-threaded operation
coordinated by `AsyncUploadQueue` where secondary threads (like `Secondary Copy Queue`) sometimes release
other StreamingImage(s) and that operation also locks m_databaseMutex. This is a clear
deadlock case if m_databaseMutex was already locked on the Main Thread.

The second most important change pertains to some components that instantiate StreamingImage(s)
objects in the same thread when the AssetBus calls one of the OnAssetReady() or OnAssetReloaded()
functions. The AssetBus locks its own mutex. This can also cause deadlocks,
because when  AsyncUploadQueue() is called, and an StreamingImage
is released during the `Secondary Copy Queue` thread, the StreamingImage class would call:
```cpp
Data::AssetBus::MultiHandler::BusDisconnect(GetAssetId());
```
Which will deadlock when it tries to lock the AssetBus mutex.
The fix is to defer the OnAssetXXX functions on the TickBus queue.
REMARK:
This PR fixes:
1. ImageBasedLightComponentController.cpp OnAssetXXXX
2. MaterialComponentController.cpp OnAssetXXXX
3. TerrainSurfaceMaterialsListComponent.cpp OnAssetXXXX
BUT, any other component that instantiates StreamingImage(s) exactly during OnAssetXXX functions
would need to apply a similar fix to avoid deadlocks.

The 3rd kind of bugs that have been fixed is related with race condition crashes
when some `InstanceData<Type>` classes are created during multi-threaded scenarios
like MeshDrawPacket::DoUpdate(), which runs from multiple concurrent jobs. To avoid
the crashes a new mutex was added to `InstanceDatabase<Type>` called `m_newInstanceMutex`
which is only locked when a new `InstanceData<Type>` is being created and constructed.

Finally, a major improvement was done to `CryMT::queue` class by changing its underlying container
from `std::vector` to `std::queue`. I discovered the innefficiency when I added all kinds of verbose
AZ_Printf statements when debugging all the bugs mentioned above. And I noticed that the main reason
for slow debugging was because the old implementation using `std::vector` was calling:
```cpp
v.erase(v.begin());
```
Which, for a vector, is a big no-no because erasing from the begginning of a vector causes long copy operations.
Hard to believe we've been dealing with this lingering performance threat for so long.

Added a python script:
Gems/EditorPythonBindings/Editor/Scripts/level_load_unload_stress.py
that helps automate and iterate across a list of levels.
Opens a level, enters/exits game mode, and can wait an arbitrary amount
of frames in between those operations.

Signed-off-by: galibzon <[email protected]>

* Fixed two race conditions in InstanceDatabase. (#18632)

These two race conditions were exposed by unit tests.

The fix was to move decrementing m_useCount inside
InstanceDatabase::ReleaseInstance(), which locks
m_databaseMutex first.

------------  CASE 1 (Fixed) -----------------------
First race condition (Caused a crash):
Thread 1                                  Thread2
   │                                         │
   │                                         │
   │                                         │
RefCount==1                                  │
RefCount--                                   │
RefCount==0                                  │
InstanceDatabase::ReleaseInstance            │
Thread Yields Just before LOCK MUTEX         │
                                          LOCK MUTEX
                                          FindsTheInstance
                                          RefCount==0
                                          RefCount++
                                          RefCount==1
                                          UNLOCK(Mutex)
                                           ... some stuff
                                          RefCount --
                                          RefCount == 0
                                          InstanceDatabase::ReleaseInstance
                                          LOCK MUTEX
                                           RefCount .compare_exchange_strong(0,
                                           RefCount == -1
                                           Instance Destroyed
                                           UNLOCK MUTEX
Thread 1 Resumes
LOCK MUTEX
Instance Is Now Garbage! (Because of Thread2)

------------- CASE 2 (Fixed) ------------------------------
Second race condition leaked memory.
Related with Orphaned instances:
Thread 1                                 Thread 2
   │                                         │
   │                                         │
   │                                         │
RefCount == 1                                │
Refcount(--)                                 │
RefCount == 0                                │
isOrphaned = m_isOrphaned(FALSE);            │
InstanceDatabase<Type>::ReleaseInstance      │
Thread Yields Just before LOCK MUTEX         │
                                         LOCK MUTEX
                                          FindsTheInstance
                                         m_isOrphaned = TRUE;
                                         database.erase(instance);
                                         // instance is NOT destroyed.
                                         UNLOCK MUTEX
Thread 1 Resumes.
LOCK MUTEX
Instance Is Not In The Database.
isOrphaned == FALSE.
Instance is Never Deleted.

Signed-off-by: galibzon <[email protected]>

* Fixed PreviewRenderer material update crash. (#18643)

Fixes #18629
Fixes #18630

The crash was originated by `AtomToolsFramework::PreviewRenderer`. And
the easiest way to reproduce was by making changes to an active material
asset using the MaterialEditor. Order of events for the crash:
The MaterialComponentController owned by AtomToolsFramework::PreviewRenderer,
detects a change in the material asset and it enqueues
MaterialComponentController::InitializeMaterialInstancePostTick in the TickBus
which captures the `this` address. Then immediately `MaterialComponentController::Deactivate()` is called.
Later, in the next TickBus event, `MaterialComponentController::InitializeMaterialInstancePostTick`
is executed with the same `this` pointer that was destroyed in the previous frame.

Solution, added a std:queue<Asset> to MaterialComponentController where
all notified assets are stored. The lambda function was removed so
there's no storage of the `this` pointer anymore. Then, when the
MaterialComponentController is Deactivated, all queued assets are safely
released and there's no lambda function to execute in the next Tick.

Signed-off-by: galibzon <[email protected]>

* Fixes DX12::StreamingImagePool related deadlock. (#18644)

Fixes #18623

Unlike Vulkan, DX12::StreamingImagePool uses a TiledAllocator,
which is protected by `m_tileMutex`. The bug:
On the Main thread `m_tileMutex` was being locked before calling `AllocateImageTilesInternal`,
and `AllocateImageTilesInternal` enqueues Work with AsyncUploadQueue and
waits for it (While keeping `m_tileMutex` locked). This work is
eventually processed by `Secondary Copy Queue`.

Due to unrelated work enqueued on The `Secondary Copy Queue`,
the function `StreamingImagePool::ShutdownResourceInternal()` was being called
which also locks `m_tileMutex`. The deadlock would occurred if the Main
Thread already had `m_tileMutex` locked.

The Fix: Both `AllocateImageTilesInternal` and `DeAllocateImageTilesInternal`
only use `m_tileMutex` before or after Async work is completed.

Signed-off-by: galibzon <[email protected]>

* Fixes deadlocks that occurred on some ASV tests. (#18665)

* Fixes deadlocks that occurred on some ASV tests.

Some AtomSampleViewer tests use `AZ::RPI::AssetUtils::AsyncAssetLoader`
to load StreamingImages, which always deadlocks if StreamingImage resources
are instantiated under the scope of AssetBus::OnAssetXXX events:
```
** Main Thread                | ** Secondary Copy Queue Thread
AssetBus::lock(mutex)         |
AssetBus::OnAssetReady        |
StreamingImage::FindOrCreate  |
AsyncUploadQueue::queueWork   |
Wait For Work Complete        |
                              |
                              | workQueue signaled
                              | Pop Work
                              | StreaminImage::Destructor()
                              | AssetBus::Disconnect()
                              | AssetBus::lock(mutex) <- Deadlocked
```

The solution is that AsyncAssetLoader simply saves the asset reference
upon OnAssetXXX events, and connects to the SystemTickBus to dispatch
the callback on the next System Tick.

Signed-off-by: galibzon <[email protected]>

---------

Signed-off-by: galibzon <[email protected]>

AutomatedTesting/Assets/Passes/AutomatedTesting/AutoLoadPassTemplates.azasset

@@ -0,0 +1,20 @@
+{
+    "Type": "JsonSerialization",
+    "Version": 1,
+    "ClassName": "AssetAliasesSourceData",
+    "ClassData": {
+        "AssetPaths": [
+            //////////////////////////////////////////////////////////////
+            // Passes required by ShaderReloadTest Level 
+            {
+                "Name": "SimpleMeshPassTemplate",
+                "Path": "Levels/ShaderReloadTest/SimpleMeshPass.pass"
+            },
+            {
+                "Name": "SimpleMeshPipelineTemplate",
+                "Path": "Levels/ShaderReloadTest/SimpleMeshPipeline.pass"
+            }
+            //////////////////////////////////////////////////////////////
+        ]
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/README.md

@@ -0,0 +1,79 @@
+# About ShaderReloadTest
+This level, along with the companion python script named `shader_reload_test.py` were created with the only purpose of validating that the engine can properly react to changes to Shader Assets.  
+  
+Historically, OnAssetReloaded() event was not being properly processed by the AZ::RPI::Shader class. Developers would find themselves in the situation of modifying and saving a shader file several times to make sure the rendered viewport would represent the correct state of the Shader files.  
+  
+The script `shader_reload_test.py` is expected to be executed manually. This is not part of the Unit Tests, nor should be executed
+as part of the Automation Tests.  
+  
+# How to use
+1- In the Editor, open this level: `ShaderReloadTest`.  
+2- After the level is loaded, execute the following command in the console:  
+```
+pyRunFile C:\GIT\o3de\AutomatedTesting\Levels\ShaderReloadTest\shader_reload_test.py
+```
+Depending on where the engine was installed, you may change the begginning of the absolute path shown above: `C:\GIT\o3de\AutomatedTesting\...`
+  
+By default, the previous command will run only ONE Shader modification cycle, if you want to run 10 test cycles you can try the `-i`(`--iterations`) argument:
+```
+pyRunFile C:\GIT\o3de\AutomatedTesting\Levels\ShaderReloadTest\shader_reload_test.py -i 10
+```
+  
+# For Power Users
+To get a list of all the options this script supports use the `-h`(`--help`) argument:  
+```
+pyRunFile C:\GIT\o3de\AutomatedTesting\Levels\ShaderReloadTest\shader_reload_test.py --help
+[CONSOLE] Executing console command 'pyRunFile C:\GIT\o3de\AutomatedTesting\Levels\ShaderReloadTest\shader_reload_test.py --help'
+(python_test) - usage: shader_reload_test.py [-h] [-i ITERATIONS]
+                             [--screen_update_wait_time SCREEN_UPDATE_WAIT_TIME]
+                             [--capture_count_on_failure CAPTURE_COUNT_ON_FAILURE]
+                             [-p SCREENSHOT_IMAGE_PATH]
+                             [-q QUICK_SHADER_OVERWRITE_COUNT]
+                             [-w QUICK_SHADER_OVERWRITE_WAIT]
+                             [-c CAMERA_ENTITY_NAME]
+
+Records several frames of pass attachments as image files.
+
+options:
+  -h, --help            show this help message and exit
+  -i ITERATIONS, --iterations ITERATIONS
+                        How many times the Shader should be modified and the
+                        screen pixel validated.
+  --screen_update_wait_time SCREEN_UPDATE_WAIT_TIME
+                        Minimum time to wait after modifying the shader and
+                        taking the screen snapshot to validate color output.
+  --capture_count_on_failure CAPTURE_COUNT_ON_FAILURE
+                        How many times the screen should be recaptured if the
+                        pixel output failes.
+  -p SCREENSHOT_IMAGE_PATH, --screenshot_image_path SCREENSHOT_IMAGE_PATH
+                        Absolute path of the file where the screenshot will be
+                        written to. Must include image extensions 'ppm',
+                        'png', 'bmp', 'tif'. By default a temporary png path
+                        will be created
+  -q QUICK_SHADER_OVERWRITE_COUNT, --quick_shader_overwrite_count QUICK_SHADER_OVERWRITE_COUNT
+                        How many times the shader should be overwritten before
+                        capturing the screenshot. This simulates real life
+                        cases where a shader file is updated and saved to the
+                        file system several times consecutively
+  -w QUICK_SHADER_OVERWRITE_WAIT, --quick_shader_overwrite_wait QUICK_SHADER_OVERWRITE_WAIT
+                        Minimum time to wait in between quick shader
+                        overwrites.
+  -c CAMERA_ENTITY_NAME, --camera_entity_name CAMERA_ENTITY_NAME
+                        Name of the entity that contains a Camera Component.
+                        If found, the Editor camera will be set to it before
+                        starting the test.
+```  
+  
+# Technical Details
+The level named `ShaderReloadTest` looks like the `Default Atom` level, but contains Three aditional entities, all of them working together using the Render To Texture technique. The Texture asset is called `billboard_visualize_rtt.attimage`:  
+
+1- `Billboard` Entity: This entity presents in a flat Mesh the rendering result of a custom Render Pipeline named `SimpleMeshPipeline`. The `Base Color` Texture in its material (`billboard_visualize_rtt.material`) points to `billboard_visualize_rtt.attimage`.  
+2- `RTT Camera` Entity: This entity holds a Camera that Renders To Texture, it uses the ustom Render Pipeline named `SimpleMeshPipeline` to render into the Texture Asset named `billboard_visualize_rtt.attimage`.  
+3- `Shader Ball Simple Pipeline` Entity: This entity has the `Shader Ball` mesh with a custom material named `simple_mesh.material`. Any object that uses this Material will be visible under the  `SimpleMeshPipeline`, with a simple color Red, Green or Blue.  
+  
+## Why A Custom Render Pipeline For This Test?
+The idea is to be able to change Shader code and wait the least amount of time for the Asset Processor to complete the compilation. When you modify a Shader file related to the Main Render Pipeline it may trigger a chain reaction of around 500 files to be reprocessed. On the other hand, by having a custom Render Pipeline, along with a custom material type (`simple_mesh.materialtype`) which references only one Shader (`SimpleMesh.shader`) it only take a less than 3 seconds to compile the Shader and see color updates in the screen.  
+  
+## Why All Assets Are Placed under `Levels\ShaderReloadTest`?
+Because it makes the whole test suite self contained and easy to port to other Game Projects.  
+The only asset related to this Level, that was added outside of this folder is `Assets/Passes/AutomatedTesting/AutoLoadPassTemplates.azsset`. In the future, if more custom passes and/or pipelines need to be added by other Test Suites, they can be added to this file.  

AutomatedTesting/Levels/ShaderReloadTest/ShaderReloadTest.prefab

@@ -0,0 +1,809 @@
+{
+    "ContainerEntity": {
+        "Id": "Entity_[1146574390643]",
+        "Name": "Level",
+        "Components": {
+            "Component_[10641544592923449938]": {
+                "$type": "EditorInspectorComponent",
+                "Id": 10641544592923449938
+            },
+            "Component_[12039882709170782873]": {
+                "$type": "EditorOnlyEntityComponent",
+                "Id": 12039882709170782873
+            },
+            "Component_[12265484671603697631]": {
+                "$type": "EditorPendingCompositionComponent",
+                "Id": 12265484671603697631
+            },
+            "Component_[14126657869720434043]": {
+                "$type": "EditorEntitySortComponent",
+                "Id": 14126657869720434043,
+                "Child Entity Order": [
+                    "Entity_[1176639161715]",
+                    "Entity_[697889136307]",
+                    "Entity_[499622636480]",
+                    "Entity_[762313645747]"
+                ]
+            },
+            "Component_[15230859088967841193]": {
+                "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                "Id": 15230859088967841193,
+                "Parent Entity": ""
+            },
+            "Component_[16239496886950819870]": {
+                "$type": "EditorDisabledCompositionComponent",
+                "Id": 16239496886950819870
+            },
+            "Component_[5688118765544765547]": {
+                "$type": "EditorEntityIconComponent",
+                "Id": 5688118765544765547
+            },
+            "Component_[7247035804068349658]": {
+                "$type": "EditorPrefabComponent",
+                "Id": 7247035804068349658
+            },
+            "Component_[9307224322037797205]": {
+                "$type": "EditorLockComponent",
+                "Id": 9307224322037797205
+            },
+            "Component_[9562516168917670048]": {
+                "$type": "EditorVisibilityComponent",
+                "Id": 9562516168917670048
+            },
+            "LocalViewBookmarkComponent": {
+                "$type": "LocalViewBookmarkComponent",
+                "Id": 8201633998829772483,
+                "LocalBookmarkFileName": "ShaderReloadTest_17338497290764052.setreg"
+            }
+        }
+    },
+    "Entities": {
+        "Entity_[1155164325235]": {
+            "Id": "Entity_[1155164325235]",
+            "Name": "Sun",
+            "Components": {
+                "Component_[13620450453324765907]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 13620450453324765907
+                },
+                "Component_[2134313378593666258]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 2134313378593666258
+                },
+                "Component_[234010807770404186]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 234010807770404186
+                },
+                "Component_[2970359110423865725]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 2970359110423865725
+                },
+                "Component_[3722854130373041803]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 3722854130373041803
+                },
+                "Component_[5992533738676323195]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 5992533738676323195
+                },
+                "Component_[7378860763541895402]": {
+                    "$type": "AZ::Render::EditorDirectionalLightComponent",
+                    "Id": 7378860763541895402,
+                    "Controller": {
+                        "Configuration": {
+                            "Intensity": 1.0,
+                            "CameraEntityId": "",
+                            "ShadowFilterMethod": 1
+                        }
+                    }
+                },
+                "Component_[7892834440890947578]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 7892834440890947578,
+                    "Parent Entity": "Entity_[1176639161715]",
+                    "Transform Data": {
+                        "Translate": [
+                            0.0,
+                            0.0,
+                            13.487043380737305
+                        ],
+                        "Rotate": [
+                            -76.13099670410156,
+                            -0.847000002861023,
+                            -15.8100004196167
+                        ]
+                    }
+                },
+                "Component_[8599729549570828259]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 8599729549570828259
+                },
+                "Component_[952797371922080273]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 952797371922080273
+                }
+            }
+        },
+        "Entity_[1159459292531]": {
+            "Id": "Entity_[1159459292531]",
+            "Name": "Ground",
+            "Components": {
+                "Component_[12260880513256986252]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 12260880513256986252
+                },
+                "Component_[13711420870643673468]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 13711420870643673468
+                },
+                "Component_[138002849734991713]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 138002849734991713
+                },
+                "Component_[16578565737331764849]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 16578565737331764849,
+                    "Parent Entity": "Entity_[1176639161715]"
+                },
+                "Component_[16919232076966545697]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 16919232076966545697
+                },
+                "Component_[4228479570410194639]": {
+                    "$type": "EditorStaticRigidBodyComponent",
+                    "Id": 4228479570410194639
+                },
+                "Component_[5182430712893438093]": {
+                    "$type": "EditorMaterialComponent",
+                    "Id": 5182430712893438093
+                },
+                "Component_[5245524694917323904]": {
+                    "$type": "EditorColliderComponent",
+                    "Id": 5245524694917323904,
+                    "ColliderConfiguration": {
+                        "Position": [
+                            0.0,
+                            0.0,
+                            -0.5
+                        ],
+                        "MaterialSlots": {
+                            "Slots": [
+                                {
+                                    "Name": "Entire object"
+                                }
+                            ]
+                        }
+                    },
+                    "ShapeConfiguration": {
+                        "ShapeType": 1,
+                        "Box": {
+                            "Configuration": [
+                                512.0,
+                                512.0,
+                                1.0
+                            ]
+                        }
+                    },
+                    "DebugDrawSettings": {
+                        "LocallyEnabled": false
+                    }
+                },
+                "Component_[5675108321710651991]": {
+                    "$type": "AZ::Render::EditorMeshComponent",
+                    "Id": 5675108321710651991,
+                    "Controller": {
+                        "Configuration": {
+                            "ModelAsset": {
+                                "assetId": {
+                                    "guid": "{0CD745C0-6AA8-569A-A68A-73A3270986C4}",
+                                    "subId": 277889906
+                                },
+                                "assetHint": "objects/groudplane/groundplane_512x512m.fbx.azmodel"
+                            }
+                        }
+                    }
+                },
+                "Component_[5681893399601237518]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 5681893399601237518
+                },
+                "Component_[592692962543397545]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 592692962543397545
+                },
+                "Component_[7090012899106946164]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 7090012899106946164
+                },
+                "Component_[9410832619875640998]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 9410832619875640998
+                }
+            }
+        },
+        "Entity_[1163754259827]": {
+            "Id": "Entity_[1163754259827]",
+            "Name": "Camera",
+            "Components": {
+                "Component_[11895140916889160460]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 11895140916889160460
+                },
+                "Component_[16880285896855930892]": {
+                    "$type": "{CA11DA46-29FF-4083-B5F6-E02C3A8C3A3D} EditorCameraComponent",
+                    "Id": 16880285896855930892,
+                    "Controller": {
+                        "Configuration": {
+                            "Field of View": 55.0
+                        }
+                    }
+                },
+                "Component_[17187464423780271193]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 17187464423780271193
+                },
+                "Component_[17495696818315413311]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 17495696818315413311
+                },
+                "Component_[18086214374043522055]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 18086214374043522055,
+                    "Parent Entity": "Entity_[1176639161715]",
+                    "Transform Data": {
+                        "Translate": [
+                            0.0,
+                            -1.0,
+                            2.0
+                        ]
+                    }
+                },
+                "Component_[2654521436129313160]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 2654521436129313160
+                },
+                "Component_[5265045084611556958]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 5265045084611556958
+                },
+                "Component_[7169798125182238623]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 7169798125182238623
+                },
+                "Component_[7255796294953281766]": {
+                    "$type": "GenericComponentWrapper",
+                    "Id": 7255796294953281766,
+                    "m_template": {
+                        "$type": "FlyCameraInputComponent"
+                    }
+                },
+                "Component_[8866210352157164042]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 8866210352157164042
+                },
+                "Component_[9129253381063760879]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 9129253381063760879
+                }
+            }
+        },
+        "Entity_[1168049227123]": {
+            "Id": "Entity_[1168049227123]",
+            "Name": "Grid",
+            "Components": {
+                "Component_[11443347433215807130]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 11443347433215807130
+                },
+                "Component_[14249419413039427459]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 14249419413039427459
+                },
+                "Component_[15448581635946161318]": {
+                    "$type": "AZ::Render::EditorGridComponent",
+                    "Id": 15448581635946161318,
+                    "Controller": {
+                        "Configuration": {
+                            "primarySpacing": 4.0,
+                            "primaryColor": [
+                                0.501960813999176,
+                                0.501960813999176,
+                                0.501960813999176
+                            ],
+                            "secondarySpacing": 0.5,
+                            "secondaryColor": [
+                                0.250980406999588,
+                                0.250980406999588,
+                                0.250980406999588
+                            ]
+                        }
+                    }
+                },
+                "Component_[1843303322527297409]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 1843303322527297409
+                },
+                "Component_[380249072065273654]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 380249072065273654,
+                    "Parent Entity": "Entity_[1176639161715]"
+                },
+                "Component_[7476660583684339787]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 7476660583684339787
+                },
+                "Component_[7557626501215118375]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 7557626501215118375
+                },
+                "Component_[7984048488947365511]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 7984048488947365511
+                },
+                "Component_[8118181039276487398]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 8118181039276487398
+                },
+                "Component_[9189909764215270515]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 9189909764215270515
+                }
+            }
+        },
+        "Entity_[1172344194419]": {
+            "Id": "Entity_[1172344194419]",
+            "Name": "Shader Ball",
+            "Components": {
+                "Component_[10789351944715265527]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 10789351944715265527
+                },
+                "Component_[12037033284781049225]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 12037033284781049225
+                },
+                "Component_[13759153306105970079]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 13759153306105970079
+                },
+                "Component_[14135560884830586279]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 14135560884830586279
+                },
+                "Component_[16247165675903986673]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 16247165675903986673
+                },
+                "Component_[18082433625958885247]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 18082433625958885247
+                },
+                "Component_[6472623349872972660]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 6472623349872972660,
+                    "Parent Entity": "Entity_[1176639161715]",
+                    "Transform Data": {
+                        "Rotate": [
+                            0.0,
+                            0.10000000149011612,
+                            180.0
+                        ]
+                    }
+                },
+                "Component_[6495255223970673916]": {
+                    "$type": "AZ::Render::EditorMeshComponent",
+                    "Id": 6495255223970673916,
+                    "Controller": {
+                        "Configuration": {
+                            "ModelAsset": {
+                                "assetId": {
+                                    "guid": "{FD340C30-755C-5911-92A3-19A3F7A77931}",
+                                    "subId": 281415304
+                                },
+                                "assetHint": "objects/shaderball/shaderball_default_1m.azmodel"
+                            }
+                        }
+                    }
+                },
+                "Component_[8550141614185782969]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 8550141614185782969
+                },
+                "Component_[9439770997198325425]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 9439770997198325425
+                }
+            }
+        },
+        "Entity_[1176639161715]": {
+            "Id": "Entity_[1176639161715]",
+            "Name": "Atom Default Environment",
+            "Components": {
+                "Component_[10757302973393310045]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 10757302973393310045,
+                    "Parent Entity": "Entity_[1146574390643]"
+                },
+                "Component_[14505817420424255464]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 14505817420424255464,
+                    "ComponentOrderEntryArray": [
+                        {
+                            "ComponentId": 10757302973393310045
+                        }
+                    ]
+                },
+                "Component_[14988041764659020032]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 14988041764659020032
+                },
+                "Component_[15900837685796817138]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 15900837685796817138
+                },
+                "Component_[3298767348226484884]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 3298767348226484884
+                },
+                "Component_[4076975109609220594]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 4076975109609220594
+                },
+                "Component_[5679760548946028854]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 5679760548946028854
+                },
+                "Component_[5855590796136709437]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 5855590796136709437,
+                    "Child Entity Order": [
+                        "Entity_[1155164325235]",
+                        "Entity_[1180934129011]",
+                        "Entity_[1172344194419]",
+                        "Entity_[1168049227123]",
+                        "Entity_[1163754259827]",
+                        "Entity_[1159459292531]"
+                    ]
+                },
+                "Component_[9277695270015777859]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 9277695270015777859
+                }
+            }
+        },
+        "Entity_[1180934129011]": {
+            "Id": "Entity_[1180934129011]",
+            "Name": "Global Sky",
+            "Components": {
+                "Component_[11231930600558681245]": {
+                    "$type": "AZ::Render::EditorHDRiSkyboxComponent",
+                    "Id": 11231930600558681245,
+                    "Controller": {
+                        "Configuration": {
+                            "CubemapAsset": {
+                                "assetId": {
+                                    "guid": "{215E47FD-D181-5832-B1AB-91673ABF6399}",
+                                    "subId": 1000
+                                },
+                                "assetHint": "lightingpresets/highcontrast/goegap_4k_skyboxcm.exr.streamingimage"
+                            }
+                        }
+                    }
+                },
+                "Component_[1428633914413949476]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 1428633914413949476
+                },
+                "Component_[14936200426671614999]": {
+                    "$type": "AZ::Render::EditorImageBasedLightComponent",
+                    "Id": 14936200426671614999,
+                    "Controller": {
+                        "Configuration": {
+                            "diffuseImageAsset": {
+                                "assetId": {
+                                    "guid": "{3FD09945-D0F2-55C8-B9AF-B2FD421FE3BE}",
+                                    "subId": 3000
+                                },
+                                "assetHint": "lightingpresets/highcontrast/goegap_4k_iblglobalcm_ibldiffuse.exr.streamingimage"
+                            },
+                            "specularImageAsset": {
+                                "assetId": {
+                                    "guid": "{3FD09945-D0F2-55C8-B9AF-B2FD421FE3BE}",
+                                    "subId": 2000
+                                },
+                                "assetHint": "lightingpresets/highcontrast/goegap_4k_iblglobalcm_iblspecular.exr.streamingimage"
+                            }
+                        }
+                    }
+                },
+                "Component_[14994774102579326069]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 14994774102579326069
+                },
+                "Component_[15417479889044493340]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 15417479889044493340
+                },
+                "Component_[15826613364991382688]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 15826613364991382688
+                },
+                "Component_[1665003113283562343]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 1665003113283562343
+                },
+                "Component_[3704934735944502280]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 3704934735944502280
+                },
+                "Component_[5698542331457326479]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 5698542331457326479
+                },
+                "Component_[6644513399057217122]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 6644513399057217122,
+                    "Parent Entity": "Entity_[1176639161715]"
+                },
+                "Component_[931091830724002070]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 931091830724002070
+                }
+            }
+        },
+        "Entity_[499622636480]": {
+            "Id": "Entity_[499622636480]",
+            "Name": "Shader Ball Simple Pipeline",
+            "Components": {
+                "Component_[10789351944715265527]": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 10789351944715265527
+                },
+                "Component_[12037033284781049225]": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 12037033284781049225
+                },
+                "Component_[13759153306105970079]": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 13759153306105970079
+                },
+                "Component_[14135560884830586279]": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 14135560884830586279
+                },
+                "Component_[16247165675903986673]": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 16247165675903986673
+                },
+                "Component_[18082433625958885247]": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 18082433625958885247
+                },
+                "Component_[6472623349872972660]": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 6472623349872972660,
+                    "Parent Entity": "Entity_[1146574390643]",
+                    "Transform Data": {
+                        "Rotate": [
+                            0.0,
+                            0.10000000149011612,
+                            180.0
+                        ]
+                    }
+                },
+                "Component_[6495255223970673916]": {
+                    "$type": "AZ::Render::EditorMeshComponent",
+                    "Id": 6495255223970673916,
+                    "Controller": {
+                        "Configuration": {
+                            "ModelAsset": {
+                                "assetId": {
+                                    "guid": "{FD340C30-755C-5911-92A3-19A3F7A77931}",
+                                    "subId": 281415304
+                                },
+                                "assetHint": "objects/shaderball/shaderball_default_1m.fbx.azmodel"
+                            }
+                        }
+                    }
+                },
+                "Component_[8550141614185782969]": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 8550141614185782969
+                },
+                "Component_[9439770997198325425]": {
+                    "$type": "EditorLockComponent",
+                    "Id": 9439770997198325425
+                },
+                "EditorMaterialComponent": {
+                    "$type": "EditorMaterialComponent",
+                    "Id": 8792075311785516254,
+                    "Controller": {
+                        "Configuration": {
+                            "materials": [
+                                {
+                                    "Key": {
+                                        "materialSlotStableId": 3716062507
+                                    },
+                                    "Value": {
+                                        "MaterialAsset": {
+                                            "assetId": {
+                                                "guid": "{A2BEE861-DE26-539E-9BF7-2734D59BC051}"
+                                            },
+                                            "assetHint": "levels/shaderreloadtest/simple_mesh.azmaterial"
+                                        }
+                                    }
+                                }
+                            ]
+                        }
+                    }
+                }
+            }
+        },
+        "Entity_[697889136307]": {
+            "Id": "Entity_[697889136307]",
+            "Name": "Billboard",
+            "Components": {
+                "AZ::Render::EditorMeshComponent": {
+                    "$type": "AZ::Render::EditorMeshComponent",
+                    "Id": 12572874725164207156,
+                    "Controller": {
+                        "Configuration": {
+                            "ModelAsset": {
+                                "assetId": {
+                                    "guid": "{E65E9ED3-3E38-5ABA-9E22-95E34DA4C3AE}",
+                                    "subId": 280178048
+                                },
+                                "assetHint": "objects/plane.fbx.azmodel"
+                            }
+                        }
+                    }
+                },
+                "EditorDisabledCompositionComponent": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 2049210613040809400
+                },
+                "EditorEntityIconComponent": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 8767708722047716714
+                },
+                "EditorEntitySortComponent": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 10433889388491226237
+                },
+                "EditorInspectorComponent": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 15374788991867137193
+                },
+                "EditorLockComponent": {
+                    "$type": "EditorLockComponent",
+                    "Id": 9708938003199330964
+                },
+                "EditorMaterialComponent": {
+                    "$type": "EditorMaterialComponent",
+                    "Id": 3978890222431028414,
+                    "Controller": {
+                        "Configuration": {
+                            "materials": [
+                                {
+                                    "Key": {
+                                        "materialSlotStableId": 902256226
+                                    },
+                                    "Value": {
+                                        "MaterialAsset": {
+                                            "assetId": {
+                                                "guid": "{21D3FAD5-B12A-5209-9B4E-B5F8542FB906}"
+                                            },
+                                            "assetHint": "levels/shaderreloadtest/billboard_visualize_rtt.azmaterial"
+                                        },
+                                        "PropertyOverrides": {
+                                            "baseColor.textureMap": {
+                                                "$type": "AssetId",
+                                                "Value": {
+                                                    "guid": "{9662B3E7-8AEA-5C28-8A48-D659A68929DC}"
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            ]
+                        }
+                    }
+                },
+                "EditorOnlyEntityComponent": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 2774212051242505881
+                },
+                "EditorPendingCompositionComponent": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 3145368490915592884
+                },
+                "EditorVisibilityComponent": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 12731054138024196490
+                },
+                "TransformComponent": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 17527923803740433782,
+                    "Parent Entity": "Entity_[1146574390643]",
+                    "Transform Data": {
+                        "Translate": [
+                            0.0,
+                            0.0,
+                            2.0
+                        ],
+                        "Rotate": [
+                            89.99999237060547,
+                            0.0,
+                            -180.0
+                        ]
+                    }
+                }
+            }
+        },
+        "Entity_[762313645747]": {
+            "Id": "Entity_[762313645747]",
+            "Name": "RTT Camera",
+            "Components": {
+                "EditorCameraComponent": {
+                    "$type": "{CA11DA46-29FF-4083-B5F6-E02C3A8C3A3D} EditorCameraComponent",
+                    "Id": 15407199020367077062,
+                    "Controller": {
+                        "Configuration": {
+                            "RenderToTexture": {
+                                "assetId": {
+                                    "guid": "{9662B3E7-8AEA-5C28-8A48-D659A68929DC}"
+                                },
+                                "assetHint": "levels/shaderreloadtest/billboard_visualize_rtt.attimage"
+                            },
+                            "PipelineTemplate": "SimpleMeshPipelineTemplate"
+                        }
+                    }
+                },
+                "EditorDisabledCompositionComponent": {
+                    "$type": "EditorDisabledCompositionComponent",
+                    "Id": 7240730447703738116
+                },
+                "EditorEntityIconComponent": {
+                    "$type": "EditorEntityIconComponent",
+                    "Id": 2159960220969934884
+                },
+                "EditorEntitySortComponent": {
+                    "$type": "EditorEntitySortComponent",
+                    "Id": 4374852188315041882
+                },
+                "EditorInspectorComponent": {
+                    "$type": "EditorInspectorComponent",
+                    "Id": 9738900035991710312
+                },
+                "EditorLockComponent": {
+                    "$type": "EditorLockComponent",
+                    "Id": 1372682394445022290
+                },
+                "EditorOnlyEntityComponent": {
+                    "$type": "EditorOnlyEntityComponent",
+                    "Id": 66580470534316952
+                },
+                "EditorPendingCompositionComponent": {
+                    "$type": "EditorPendingCompositionComponent",
+                    "Id": 9257311930242844718
+                },
+                "EditorVisibilityComponent": {
+                    "$type": "EditorVisibilityComponent",
+                    "Id": 3242155584068297903
+                },
+                "TransformComponent": {
+                    "$type": "{27F1E1A1-8D9D-4C3B-BD3A-AFB9762449C0} TransformComponent",
+                    "Id": 4381083404231325115,
+                    "Parent Entity": "Entity_[1146574390643]",
+                    "Transform Data": {
+                        "Translate": [
+                            0.0,
+                            -1.25,
+                            0.5299999713897705
+                        ]
+                    }
+                }
+            }
+        }
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/SimpleMesh.azsl

@@ -0,0 +1,54 @@
+/*
+ * Copyright (c) Contributors to the Open 3D Engine Project.
+ * For complete copyright and license terms please see the LICENSE at the root of this distribution.
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ *
+ */
+
+#include <scenesrg.srgi>
+#include <viewsrg.srgi>
+// To get the world matrix shader constant for the current object.
+#include <Atom/Features/PBR/DefaultObjectSrg.azsli>
+
+struct VSInput
+{
+    float3 m_position : POSITION;
+    float3 m_normal : NORMAL;
+    // For the complete list of supported input stream semantics see ModelAssetBuilderComponent::CreateMesh()
+};
+
+struct VSOutput
+{
+    float4 m_position : SV_Position;
+    float3 m_normal: NORMAL;
+};
+
+VSOutput MainVS(VSInput IN)
+{
+    VSOutput OUT;
+
+    float3 worldPosition = mul(ObjectSrg::GetWorldMatrix(), float4(IN.m_position, 1.0)).xyz;
+    OUT.m_position = mul(ViewSrg::m_viewProjectionMatrix, float4(worldPosition, 1.0));
+    OUT.m_normal = IN.m_normal;
+
+    return OUT;
+}
+
+struct PSOutput
+{
+    float4 m_color : SV_Target0;
+};
+
+PSOutput MainPS(VSOutput IN)
+{
+    PSOutput OUT;
+    const float3 RED_COLOR = float3(1, 0, 0);
+    const float3 GREEN_COLOR = float3(0, 1, 0);
+    const float3 BLUE_COLOR = float3(0, 0, 1);
+    // ShaderReloadTest START
+    const float3 TEST_COLOR = BLUE_COLOR;
+    // ShaderReloadTest END
+    OUT.m_color = float4(TEST_COLOR, 1.0);
+    return OUT;
+}

AutomatedTesting/Levels/ShaderReloadTest/SimpleMesh.shader

@@ -0,0 +1,24 @@
+{
+    "Source" : "SimpleMesh.azsl",
+
+    "DepthStencilState" : { 
+        "Depth" : { "Enable" : true, "CompareFunc" : "GreaterEqual" }
+    },
+
+    "DrawList" : "simplepass",
+
+    "ProgramSettings":
+    {
+      "EntryPoints":
+      [
+        {
+          "name": "MainVS",
+          "type": "Vertex"
+        },
+        {
+          "name": "MainPS",
+          "type": "Fragment"
+        }
+      ]
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/SimpleMeshPass.pass

@@ -0,0 +1,70 @@
+{
+    "Type": "JsonSerialization",
+    "Version": 1,
+    "ClassName": "PassAsset",
+    "ClassData": {
+        "PassTemplate": {
+            "Name": "SimpleMeshPassTemplate",
+            "PassClass": "RasterPass",
+            "Slots": [
+                {
+                    "Name": "DepthOutput",
+                    "SlotType": "Output",
+                    "ScopeAttachmentUsage": "DepthStencil",
+                    "LoadStoreAction": {
+                        "ClearValue": {
+                            "Type": "DepthStencil"
+                        },
+                        "LoadAction": "Clear",
+                        "LoadActionStencil": "Clear"
+                    }
+                },
+                {
+                    "Name": "LightingOutput",
+                    "SlotType": "Output",
+                    "ScopeAttachmentUsage": "RenderTarget",
+                    "LoadStoreAction": {
+                        "ClearValue": {
+                            "Value": [
+                                0.7,
+                                0.0,
+                                0.0,
+                                0.0
+                            ]
+                        },
+                        "LoadAction": "Clear"
+                    }
+                }
+            ],
+            "ImageAttachments": [
+                {
+                    "Name": "DepthAttachment",
+                    "SizeSource": {
+                        "Source": {
+                            "Pass": "Parent",
+                            "Attachment": "PipelineOutput"
+                        }
+                    },
+                    "ImageDescriptor": {
+                        "Format": "D32_FLOAT_S8X24_UINT",
+                        "SharedQueueMask": "Graphics"
+                    }
+                }
+            ],
+            "Connections": [
+                {
+                    "LocalSlot": "DepthOutput",
+                    "AttachmentRef": {
+                        "Pass": "This",
+                        "Attachment": "DepthAttachment"
+                    }
+                }
+            ],
+            "PassData": {
+                "$type": "RasterPassData",
+                "DrawListTag": "simplepass",
+                "BindViewSrg": true
+            }
+        }
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/SimpleMeshPipeline.pass

@@ -0,0 +1,36 @@
+{
+    "Type": "JsonSerialization",
+    "Version": 1,
+    "ClassName": "PassAsset",
+    "ClassData": {
+        "PassTemplate": {
+            "Name": "SimpleMeshPipelineTemplate",
+            "PassClass": "ParentPass",
+            "Slots": [
+                {
+                    // The Slot name must be exactly "PipelineOutput" because this the Parent Pass that
+                    // Describes a Render Pipeline and the C++ code looks for a PassSlotBinding
+                    // with this name, which will be connected to the SwapChain.
+                    "Name": "PipelineOutput",
+                    "SlotType": "InputOutput"
+                }
+            ],
+            "PassRequests": [     
+                {
+                    "Name": "SimpleMeshPass",
+                    "TemplateName": "SimpleMeshPassTemplate",
+                    "Enabled": true,
+                    "Connections": [
+                        {
+                            "LocalSlot": "LightingOutput",
+                            "AttachmentRef": {
+                                "Pass": "Parent",
+                                "Attachment": "PipelineOutput"
+                            }
+                        }
+                    ]
+                }
+            ]
+        }
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/billboard_visualize_rtt.attimage

@@ -0,0 +1,21 @@
+{
+    "Type": "JsonSerialization",
+    "Version": 1,
+    "ClassName": "AttachmentImageAsset",
+    "ClassData": {
+        "m_imageDescriptor": {
+            "BindFlags": [
+                "ShaderRead",
+                "ShaderWrite",
+                "Color"
+            ],
+            "Size": {
+                "Width": 512,
+                "Height": 512
+            },
+            "Format": 19 //AZ::RHI::Format::R8G8B8A8_UNORM
+        },
+        "Name": "$BillboardVisualizeRtt",
+        "IsUniqueName": true
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/billboard_visualize_rtt.material

@@ -0,0 +1,27 @@
+{
+    "materialType": "@gemroot:Atom_Feature_Common@/Assets/Materials/Types/StandardPBR.materialtype",
+    "materialTypeVersion": 5,
+    "propertyValues": {
+        "baseColor.color": [
+            0.800000011920929,
+            0.800000011920929,
+            0.800000011920929,
+            1.0
+        ],
+        // Commenting this out because it causes the `AssetProcessor` to never
+        // complete the compilation of this material. The solution is to run
+        // the Editor and open the `Material Instance Editor` and pick this
+        // same attachment image for the `baseColor.textureMap`. The actual
+        // GUID of the asset is stored in `ShaderLoadTest.prefab` (which is
+        // how Material Instance Properties are stored).
+        // "baseColor.textureMap": "Levels/ShaderReloadTest/billboard_visualize_rtt.attimage",
+        "emissive.color": [
+            0.0,
+            0.0,
+            0.0,
+            1.0
+        ],
+        "opacity.factor": 1.0,
+        "roughness.factor": 0.5527864098548889
+    }
+}

AutomatedTesting/Levels/ShaderReloadTest/shader_reload_test.py

@@ -0,0 +1,311 @@
+#
+# Copyright (c) Contributors to the Open 3D Engine Project.
+# For complete copyright and license terms please see the LICENSE at the root of this distribution.
+#
+# SPDX-License-Identifier: Apache-2.0 OR MIT
+#
+#
+import os
+import numpy as np
+import tempfile
+from PIL import Image
+
+import azlmbr.bus as azbus
+import azlmbr.legacy.general as azgeneral
+import azlmbr.editor as azeditor
+import azlmbr.atom as azatom
+import azlmbr.entity as azentity
+import azlmbr.camera as azcamera
+
+# See README.md for details.
+
+# Some global constants
+EXPECTED_LEVEL_NAME = "ShaderReloadTest"
+SUPPORTED_IMAGE_FILE_EXTENSIONS = {".ppm", ".bmp", ".tiff", ".tif", ".png"}
+
+def OverwriteFile(azslFilePath: str, begin: list, middle: list, end: list):
+    fileObj = open(azslFilePath, "w")
+    fileObj.writelines(begin)
+    fileObj.writelines(middle)
+    fileObj.writelines(end)
+    fileObj.close()
+
+
+def FlipTestColorLine(line: str) -> tuple[tuple[int, int, int], str]:
+    """
+    Flips/toggles "BLUE_COLOR" for "GREEN_COLOR" in @line
+    and viceversa. Also retuns the expected pixel color in addition
+    to the modified line.
+    """
+    if "BLUE_COLOR" in line:
+        expectedColor = "GREEN_COLOR"
+        newLine = line.replace("BLUE_COLOR", expectedColor)
+        return (0, 152, 15), newLine
+    elif "GREEN_COLOR" in line:
+        expectedColor = "BLUE_COLOR"
+        newLine = line.replace("GREEN_COLOR", expectedColor)
+        return (0, 1, 155), newLine
+    raise Exception("Can't find color")
+
+
+def FlipShaderColor(azslFilePath: str) -> tuple[int, int, int]:
+    """
+    Modifies the Shader code in @azslFilePath by flipping the line
+    that outputs the expected color.
+    If the line is found as:
+        const float3 TEST_COLOR = BLUE_COLOR;
+    It gets flipped to:
+        const float3 TEST_COLOR = GREEN_COLOR;
+    and viceversa.
+    """
+    begin = []
+    middle = []
+    end = []
+    fileObj = open(azslFilePath, 'rt')
+    sectionStart = False
+    sectionEnd = False
+    expectedColor = ""
+    for line in fileObj:
+        if ("ShaderReloadTest" in line):
+            if ("START" in line):
+                middle.append(line)
+                sectionStart = True
+                continue
+            elif ("END" in line):
+                middle.append(line)
+                sectionEnd = True
+                continue
+        if (not sectionStart) and (not sectionEnd):
+            begin.append(line)
+            continue
+        if sectionEnd:
+            end.append(line)
+            continue
+        if "TEST_COLOR" in line:
+            expectedColor, newLine = FlipTestColorLine(line)
+            middle.append(newLine)
+
+    fileObj.close()
+    OverwriteFile(azslFilePath, begin, middle, end)
+    return expectedColor
+    
+
+def UpdateShaderAndTestPixelResult(azslFilePath: str, screenUpdateWaitTime: float, captureCountOnFailure: int, screenshotImagePath: str, quickShaderOverwriteCount: int, quickShaderOverwriteWait: float) -> bool:
+    """
+    This function represents the work done for a single iteration of the shader modification test.
+    Modifies the content of the Shader (@azslFilePath), waits @screenUpdateWaitTime, and captures
+    the pixels of the Editor Viewport.  
+    Some leniency was added with the variable @captureCountOnFailure because sometimes @@screenUpdateWaitTime is not
+    enought time for the screen to update. This function retries @captureCountOnFailure times before
+    considering it a failure.
+    """
+    expectedPixelColor = FlipShaderColor(azslFilePath)
+    print(f"Expecting color {expectedPixelColor}")
+    for overwriteCount in range(quickShaderOverwriteCount):
+        azgeneral.idle_wait(quickShaderOverwriteWait)
+        expectedPixelColor = FlipShaderColor(azslFilePath)
+        print(f"Shader quickly overwritten {overwriteCount + 1} of {quickShaderOverwriteCount}")
+        print(f"Expecting color {expectedPixelColor}")
+    azgeneral.idle_wait(screenUpdateWaitTime)
+    # Capture the screenshot
+    captureCount = -1
+    success = False
+    while captureCount < captureCountOnFailure:
+        captureCount += 1
+        outcome = azatom.FrameCaptureRequestBus(
+                    azbus.Broadcast, "CaptureScreenshot", screenshotImagePath
+                )
+        if not outcome.IsSuccess():
+            frameCaptureError = outcome.GetError()
+            errorMsg = frameCaptureError.error_message
+            print(f"Failed to capture screenshot at outputImagePath='{screenshotImagePath}'\nError:\n{errorMsg}")
+            return False
+        azgeneral.idle_wait(screenUpdateWaitTime)
+        img = Image.open(screenshotImagePath)
+        width, height = img.size
+        r = int(height/2)
+        c = int(width/2)
+        image_array = np.array(img)
+        color = image_array[r][c]
+        print(f"captureCount {captureCount}: Center Pixel[{r},{c}] Color={color}, type:{type(color)}, r={color[0]}, g={color[1]}, b={color[2]}")
+        success = (color[0] == expectedPixelColor[0]) and (color[1] == expectedPixelColor[1]) and (color[2] == expectedPixelColor[2])
+        if success:
+            return success
+    return success 
+
+
+def ShaderReloadTest(iterationCountMax: int, screenUpdateWaitTime: float, captureCountOnFailure: int, screenshotImagePath: str, quickShaderOverwriteCount: int, quickShaderOverwriteWait: float) -> tuple[bool, int]:
+    """
+    This function is the main loop. Runs @iterationCountMax iterations and all iterations must PASS
+    to consider the test a success.
+    A single iteration modifies the Shader file, waits @screenUpdateWaitTime, captures the pixel content
+    of the Editor Viewport, and reads the center pixel of the image for an expected color.
+    """    
+    levelPath = azeditor.EditorToolsApplicationRequestBus(azbus.Broadcast, "GetCurrentLevelPath")
+    levelName = os.path.basename(levelPath)
+    if levelName != EXPECTED_LEVEL_NAME:
+        print(f"ERROR: This test suite expects a level named '{EXPECTED_LEVEL_NAME}', instead got '{levelName}'")
+        return False, 0
+    azslFilePath = os.path.join(levelPath, "SimpleMesh.azsl") 
+
+    iterationCount = 0
+    success = False
+    while iterationCount < iterationCountMax:
+        iterationCount += 1
+        print(f"Starting Retry {iterationCount} of {iterationCountMax}...")
+        success = UpdateShaderAndTestPixelResult(azslFilePath, screenUpdateWaitTime, captureCountOnFailure, screenshotImagePath, quickShaderOverwriteCount, quickShaderOverwriteWait)
+        if not success:
+            break
+    return success, iterationCount
+
+
+def ValidateImageExtension(screenshotImagePath: str) -> bool:
+    _, file_extension = os.path.splitext(screenshotImagePath)
+    if file_extension in SUPPORTED_IMAGE_FILE_EXTENSIONS:
+        return True
+    print(f"ERROR: Image path '{screenshotImagePath}' has an unsupported file extension.\nSupported extensions: {SUPPORTED_IMAGE_FILE_EXTENSIONS}")
+    return False
+
+
+def AdjustEditorCameraPosition(cameraEntityName: str) -> azentity.EntityId:
+    """
+    Searches for an entity named @cameraEntityName, assumes the entity has a Camera Component,
+    and forces the Editor Viewport to make it the Active Camera. This helps center the `Billboard`
+    entity because this test Samples the middle the of the screen for the correct Pixel color.
+    """
+    if not cameraEntityName:
+        return None
+    # Find the first entity with such name.
+    searchFilter = azentity.SearchFilter()
+    searchFilter.names = [cameraEntityName,]
+    entityList = azentity.SearchBus(azbus.Broadcast, "SearchEntities", searchFilter)
+    print(f"Found {len(entityList)} entities named {cameraEntityName}. Will use the first.")
+    if len(entityList) < 1:
+        print(f"No camera entity with name {cameraEntityName} was found. Viewport camera won't be adjusted.")
+        return None
+    cameraEntityId = entityList[0]
+    isActiveCamera = azcamera.EditorCameraViewRequestBus(azbus.Event, "IsActiveCamera", cameraEntityId)
+    if isActiveCamera:
+        print(f"Entity '{cameraEntityName}' is already the active camera")
+        return cameraEntityId
+    azcamera.EditorCameraViewRequestBus(azbus.Event, "ToggleCameraAsActiveView", cameraEntityId)
+    print(f"Entity '{cameraEntityName}' is now the active camera. Will wait 2 seconds for the screen to settle.")
+    print(f"REMARK: It is expected that the camera is located at [0, -1, 2] with all euler angles at 0.")
+    azgeneral.idle_wait(2.0)
+    return cameraEntityId
+
+
+def ClearViewportOfHelpers():
+    """
+    Makes sure all helpers and artifacts that add unwanted pixels
+    are hidden.
+    """
+    # Make sure no entity is selected when the test runs because entity selection adds unwanted colored pixels
+    azeditor.ToolsApplicationRequestBus(azbus.Broadcast, "SetSelectedEntities", [])
+    # Hide helpers
+    if azgeneral.is_helpers_shown():
+        azgeneral.toggle_helpers()
+    # Hide icons
+    if azgeneral.is_icons_shown():
+        azgeneral.toggle_icons()
+    #Hide FPS, etc
+    azgeneral.set_cvar_integer("r_displayInfo", 0)
+    # Wait a little for the screen to update.
+    azgeneral.idle_wait(0.25)
+
+
+# Quick Example on how to run this test from the Editor Console (See README.md for more details):
+# Runs 10 iterations:
+# pyRunFile C:\GIT\o3de\AutomatedTesting\Levels\ShaderReloadTest\shader_reload_test.py -i 10
+def MainFunc():
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        description="Records several frames of pass attachments as image files."
+    )
+
+    parser.add_argument(
+        "-i",
+        "--iterations",
+        type=int,
+        default=1,
+        help="How many times the Shader should be modified and the screen pixel validated.",
+    )
+
+    parser.add_argument(
+        "--screen_update_wait_time",
+        type=float,
+        default=3.0,
+        help="Minimum time to wait after modifying the shader and taking the screen snapshot to validate color output.",
+    )
+
+    parser.add_argument(
+        "--capture_count_on_failure",
+        type=int,
+        default=2,
+        help="How many times the screen should be recaptured if the pixel output failes.",
+    )
+
+    parser.add_argument(
+        "-p",
+        "--screenshot_image_path",
+        default="",
+        help="Absolute path of the file where the screenshot will be written to. Must include image extensions 'ppm', 'png', 'bmp', 'tif'. By default a temporary png path will be created",
+    )
+
+    parser.add_argument(
+        "-q",
+        "--quick_shader_overwrite_count",
+        type=int,
+        default=0,
+        help="How many times the shader should be overwritten before capturing the screenshot. This simulates real life cases where a shader file is updated and saved to the file system several times consecutively",
+    )
+
+    parser.add_argument(
+        "-w",
+        "--quick_shader_overwrite_wait",
+        type=float,
+        default=1.0,
+        help="Minimum time to wait in between quick shader overwrites.",
+    )
+
+    parser.add_argument(
+        "-c",
+        "--camera_entity_name",
+        default="Camera",
+        help="Name of the entity that contains a Camera Component. If found, the Editor camera will be set to it before starting the test.",
+    )
+
+    args = parser.parse_args()
+    iterationCountMax = args.iterations
+    screenUpdateWaitTime = args.screen_update_wait_time
+    captureCountOnFailure = args.capture_count_on_failure
+    screenshotImagePath = args.screenshot_image_path
+    quickShaderOverwriteCount = args.quick_shader_overwrite_count
+    quickShaderOverwriteWait = args.quick_shader_overwrite_wait
+    cameraEntityName = args.camera_entity_name
+    tmpDir = None
+    if not screenshotImagePath:
+        tmpDir = tempfile.TemporaryDirectory()
+        screenshotImagePath = os.path.join(tmpDir.name, "shader_reload.png")
+        print(f"The temporary file '{screenshotImagePath}' will be used to capture screenshots")
+    else:
+        if not ValidateImageExtension(screenshotImagePath):
+            return # Exit test suite.
+    
+    cameraEntityId = AdjustEditorCameraPosition(cameraEntityName)
+    ClearViewportOfHelpers()
+
+    result, iterationCount = ShaderReloadTest(iterationCountMax, screenUpdateWaitTime, captureCountOnFailure, screenshotImagePath, quickShaderOverwriteCount, quickShaderOverwriteWait)
+    if result:
+        print(f"ShaderReloadTest PASSED after retrying {iterationCount}/{iterationCountMax} times.")
+    else:
+        print(f"ShaderReloadTest FAILED after retrying {iterationCount}/{iterationCountMax} times.")
+    
+    if cameraEntityId is not None:
+        azcamera.EditorCameraViewRequestBus(azbus.Event, "ToggleCameraAsActiveView", cameraEntityId)
+    
+    if tmpDir:
+        tmpDir.cleanup()
+if __name__ == "__main__":
+    MainFunc()

AutomatedTesting/Levels/ShaderReloadTest/simple_mesh.material

@@ -0,0 +1,4 @@
+{
+    "materialType": "simple_mesh.materialtype",
+    "materialTypeVersion": 1
+}

AutomatedTesting/Levels/ShaderReloadTest/simple_mesh.materialtype

@@ -0,0 +1,9 @@
+{
+    "description": "For ShaderReloadTest.",
+    "version": 1,
+    "shaders": [
+        {
+            "file": "Levels/ShaderReloadTest/SimpleMesh.shader"
+        }
+    ]
+}

Code/Editor/Lib/Tests/test_ViewportTitleDlgPythonBindings.cpp

@@ -54,5 +54,7 @@ namespace ViewportTitleDlgFuncsUnitTests
 
         EXPECT_TRUE(behaviorContext->m_methods.find("toggle_helpers") != behaviorContext->m_methods.end());
         EXPECT_TRUE(behaviorContext->m_methods.find("is_helpers_shown") != behaviorContext->m_methods.end());
+        EXPECT_TRUE(behaviorContext->m_methods.find("toggle_icons") != behaviorContext->m_methods.end());
+        EXPECT_TRUE(behaviorContext->m_methods.find("is_icons_shown") != behaviorContext->m_methods.end());
     }
 }

Code/Editor/ViewportTitleDlg.cpp

@@ -319,6 +319,17 @@ inline double Round(double fVal, double fStep)
     {
         return AzToolsFramework::HelpersVisible();
     }
+
+    void PyToggleIcons()
+    {
+        AzToolsFramework::SetIconsVisible(!AzToolsFramework::IconsVisible());
+        AzToolsFramework::EditorSettingsAPIBus::Broadcast(&AzToolsFramework::EditorSettingsAPIBus::Events::SaveSettingsRegistryFile);
+    }
+
+    bool PyIsIconsShown()
+    {
+        return AzToolsFramework::IconsVisible();
+    }
 } // namespace
 
 namespace AzToolsFramework
@@ -337,6 +348,8 @@ namespace AzToolsFramework
 
             addLegacyGeneral(behaviorContext->Method("toggle_helpers", PyToggleHelpers, nullptr, "Toggles the display of helpers."));
             addLegacyGeneral(behaviorContext->Method("is_helpers_shown", PyIsHelpersShown, nullptr, "Gets the display state of helpers."));
+            addLegacyGeneral(behaviorContext->Method("toggle_icons", PyToggleIcons, nullptr, "Toggles the display of icons."));
+            addLegacyGeneral(behaviorContext->Method("is_icons_shown", PyIsIconsShown, nullptr, "Gets the display state of icons."));
         }
     }
 } // namespace AzToolsFramework

Code/Framework/AtomCore/AtomCore/Instance/InstanceData.cpp

@@ -36,25 +36,23 @@ namespace AZ
 
         void InstanceData::release()
         {
-            // It is possible that some other thread, not us, will delete this InstanceData after we
-            // decrement m_useCount. For example, another thread could create and release an instance
-            // immediately after we decrement. So we copy the necessary data to the callstack before
-            // decrementing. This ensures the call to ReleaseInstance() below won't crash even if this
-            // InstanceData gets deleted by another thread first.
-            InstanceDatabaseInterface* parentDatabase = m_parentDatabase;
-            InstanceId instanceId = GetId();
-
-            const int prevUseCount = m_useCount.fetch_sub(1);
-
-            AZ_Assert(prevUseCount >= 1, "m_useCount is negative");
-
-            if (prevUseCount == 1)
+            // If @m_parentDatabase is valid we can't just simply decrement the ref count
+            // InstanceDatabase also supports the case of Orphaned instances. The only
+            // way to guarantee correcteness is to delegate ref count subtraction to the
+            // instanceDatabase under its database mutex.
+            // TODO: Ideally we should call `m_useCount.fetch_sub(1)` first and if it reaches the value
+            //       0, then we'd ask @m_parentDatabase to release. Investigate why Mesh Model Loading
+            //       needs to Orphan instances. Once Orphaned instance API is removed then we can call
+            //       m_useCount.fetch_sub(1) before having to lock the instanceDatabase mutex.
+            if (m_parentDatabase)
             {
-                if (parentDatabase)
-                {
-                    parentDatabase->ReleaseInstance(this, instanceId);
-                }
-                else
+                m_parentDatabase->ReleaseInstance(this);
+            }
+            else
+            {
+                const int prevUseCount = m_useCount.fetch_sub(1);
+                AZ_Assert(prevUseCount >= 1, "m_useCount is negative");
+                if (prevUseCount == 1)
                 {
                     // This is a standalone object not created through the InstanceDatabase so
                     // we can just delete it.

Code/Framework/AtomCore/AtomCore/Instance/InstanceDatabase.h

@@ -17,6 +17,7 @@
 #include <AzCore/Module/Environment.h>
 #include <AzCore/std/parallel/shared_mutex.h>
 
+
 namespace AZStd
 {
     class any;
@@ -74,7 +75,7 @@ namespace AZ
         {
             friend class InstanceData;
         protected:
-            virtual void ReleaseInstance(InstanceData* instance, const InstanceId& instanceId) = 0;
+            virtual void ReleaseInstance(InstanceData* instance) = 0;
         };
 
         /**
@@ -235,7 +236,7 @@ namespace AZ
             Data::Instance<Type> EmplaceInstance(const InstanceId& id, const Data::Asset<AssetData>& asset, const AZStd::any* param);
 
             // Utility function called by InstanceData to remove the instance from the database.
-            void ReleaseInstance(InstanceData* instance, const InstanceId& instanceId) override;
+            void ReleaseInstance(InstanceData* instance) override;
 
             void ValidateSameAsset(InstanceData* instance, const Data::Asset<AssetData>& asset) const;
 
@@ -246,6 +247,12 @@ namespace AZ
             mutable AZStd::recursive_mutex m_databaseMutex;
             AZStd::unordered_map<InstanceId, Type*> m_database;
 
+            // There are several classes in Atom, like ShaderResourceGroup, that are not threadsafe
+            // because they share the same ShaderResourceGroupPool, so it is important that for each
+            // InstanceType there's a mutex that prevents several of those classes from being instantiated
+            // simultaneously.
+            AZStd::recursive_mutex m_instanceCreationMutex;
+
             // All instances created by this InstanceDatabase will be for assets derived from this type.
             AssetType m_baseAssetType;
 
@@ -316,19 +323,6 @@ namespace AZ
                 return nullptr;
             }
 
-            // Take a lock to guard the insertion.  Note that this will not guard against recursive insertions on the same thread.
-            AZStd::scoped_lock<AZStd::recursive_mutex> lock(m_databaseMutex);
-
-            // Search again in case someone else got here first.
-            auto iter = m_database.find(id);
-            if (iter != m_database.end())
-            {
-                InstanceData* data = static_cast<InstanceData*>(iter->second);
-                ValidateSameAsset(data, asset);
-
-                return iter->second;
-            }
-
             return EmplaceInstance(id, assetLocal, param);
         }
 
@@ -351,8 +345,6 @@ namespace AZ
                 return nullptr;
             }
 
-            // Take a lock to guard the insertion.  Note that this will not guard against recursive insertions on the same thread.
-            AZStd::scoped_lock<AZStd::recursive_mutex> lock(m_databaseMutex);
             return EmplaceInstance(id, assetLocal, param);
         }
 
@@ -391,53 +383,77 @@ namespace AZ
         Data::Instance<Type> InstanceDatabase<Type>::EmplaceInstance(
             const InstanceId& id, const Data::Asset<AssetData>& asset, const AZStd::any* param)
         {
-            // This assert is here to catch any potential non-randomness in our id generation. If it triggers,
-            // there might be a bug / race condition in the id generator. The same assert also occurs *after*
-            // instance creation to help differentiate between a non-random id vs recursive creation of the same id.
-            AZ_Assert(
-                !m_database.contains(id),
-                "Database already contains an instance for this id (%s), possibly a random id generation collision?",
-                id.ToString<AZStd::fixed_string<64>>().c_str());
-
-            // Emplace a new instance and return it.
+            // It's very important to have m_databaseMutex unlocked while an instance is being created because
+            // there can be cases like in StreamingImage(s), where multiple threads are involved and some of those threads
+            // attempt to release a StreamingImage, which in turn will lock m_databaseMutex and it could incurr
+            // in potential deadlocks.
+
+            // If the instance was created redundantly, it will be temporarily stored here for destruction
+            // before this function returns.
+            Data::Instance<Type> redundantInstance = nullptr;
+
             // It's possible for the m_createFunction call to recursively trigger another FindOrCreate call, so be aware that
             // the contents of m_database may change within this call.
             Data::Instance<Type> instance = nullptr;
-            if (!param)
-            {
-                instance = m_instanceHandler.m_createFunction(asset.Get());
-            }
-            else
+
             {
-                instance = m_instanceHandler.m_createFunctionWithParam(asset.Get(), param);
+                AZStd::scoped_lock<decltype(m_instanceCreationMutex)> lock(m_instanceCreationMutex);
+                if (!param)
+                {
+                    instance = m_instanceHandler.m_createFunction(asset.Get());
+                }
+                else
+                {
+                    instance = m_instanceHandler.m_createFunctionWithParam(asset.Get(), param);
+                }
             }
 
+            // Lock the database. There's still a chance that the same instance was created in parallel.
+            // in such case we return the first one that made it into the database and gracefully release the
+            // redundant one.
             if (instance)
             {
-                AZ_Assert(
-                    !m_database.contains(id),
-                    "Instance creation for asset id %s resulted in a recursive creation of that asset, which was unexpected. "
-                    "This asset might be erroneously referencing itself as a dependent asset.",
-                    asset.GetHint().c_str());
-
-                instance->m_id = id;
-                instance->m_parentDatabase = this;
-                instance->m_assetId = asset.GetId();
-                instance->m_assetType = asset.GetType();
-                m_database.emplace(id, instance.get());
+                AZStd::scoped_lock<AZStd::recursive_mutex> lock(m_databaseMutex);
+                auto iter = m_database.find(id);
+                if (iter != m_database.end())
+                {
+                    InstanceData* data = static_cast<InstanceData*>(iter->second);
+                    ValidateSameAsset(data, asset);
+                    redundantInstance = instance; // Will be destroyed as soon as we return from this function.
+                    instance = iter->second;
+                }
+                else
+                {
+                    instance->m_id = id;
+                    instance->m_parentDatabase = this;
+                    instance->m_assetId = asset.GetId();
+                    instance->m_assetType = asset.GetType();
+                    m_database.emplace(id, instance.get());
+                }
             }
+
             return instance;
         }
 
         template<typename Type>
-        void InstanceDatabase<Type>::ReleaseInstance(InstanceData* instance, const InstanceId& instanceId)
+        void InstanceDatabase<Type>::ReleaseInstance(InstanceData* instance)
         {
             AZStd::scoped_lock<AZStd::recursive_mutex> lock(m_databaseMutex);
             
+            const int prevUseCount = instance->m_useCount.fetch_sub(1);
+            AZ_Assert(prevUseCount >= 1, "m_useCount is negative");
+            if (prevUseCount > 1)
+            {
+                // This instance is still being used.
+                return;
+            }
+        
             // If instanceId doesn't exist in m_database that means the instance was already deleted on another thread.
             // We check and make sure the pointers match before erasing, just in case some other InstanceData was created with the same ID.
             // We re-check the m_useCount in case some other thread requested an instance from the database after we decremented m_useCount.
-            // We change m_useCount to -1 to be sure another thread doesn't try to clean up the instance (though the other checks probably cover that).
+            // We change m_useCount to -1 to be sure another thread doesn't try to clean up the instance (though the other checks
+            // probably cover that).
+            auto instanceId = instance->GetId();
             auto instanceItr = m_database.find(instanceId);
             int32_t expectedRefCount = 0;
             if (instanceItr != m_database.end() &&

Code/Framework/AzCore/AzCore/Memory/SystemAllocator.cpp

@@ -21,6 +21,19 @@
 #define AZCORE_SYSTEM_ALLOCATOR_HPHA 1
 #define AZCORE_SYSTEM_ALLOCATOR_MALLOC 2
 
+#if !defined(AZCORE_SYSTEM_ALLOCATOR)
+    // We are using here AZCORE_SYSTEM_ALLOCATOR_HPHA for the sake of passing unit tests.
+    // But it's been found that, when using Vulkan, and working with levels that have
+    // large amount of meshes, entering/Exiting game mode puts lost of stress in memory allocation that crashes
+    // when using HPHA. With AZCORE_SYSTEM_ALLOCATOR_MALLOC crashes don't occur.
+    // TODO: Review Github Issue #18597
+    #define AZCORE_SYSTEM_ALLOCATOR AZCORE_SYSTEM_ALLOCATOR_HPHA
+#endif
+
+#if (AZCORE_SYSTEM_ALLOCATOR != AZCORE_SYSTEM_ALLOCATOR_HPHA) && (AZCORE_SYSTEM_ALLOCATOR != AZCORE_SYSTEM_ALLOCATOR_MALLOC)
+    #error AZCORE_SYSTEM_ALLOCATOR is an invalid value, it needs to be either AZCORE_SYSTEM_ALLOCATOR_HPHA or AZCORE_SYSTEM_ALLOCATOR_MALLOC
+#endif
+
 #include <AzCore/Memory/HphaAllocator.h>
 
 namespace AZ

Code/Legacy/CryCommon/MultiThread_Containers.h

@@ -22,14 +22,14 @@ namespace CryMT
     //////////////////////////////////////////////////////////////////////////
 
     //////////////////////////////////////////////////////////////////////////
-    // Multi-Thread safe queue container, can be used instead of std::vector.
+    // Multi-Thread safe queue container.
     //////////////////////////////////////////////////////////////////////////
     template <class T, class Alloc = std::allocator<T> >
     class queue
     {
     public:
         typedef T   value_type;
-        typedef std::vector<T, Alloc>   container_type;
+        typedef std::queue<T, std::deque<T, Alloc>>   container_type;
         typedef AZStd::lock_guard<AZStd::recursive_mutex> AutoLock;
 
         //////////////////////////////////////////////////////////////////////////
@@ -37,7 +37,7 @@ namespace CryMT
         //////////////////////////////////////////////////////////////////////////
         const T& front() const      { AutoLock lock(m_cs); return v.front(); };
         const T& back() const { AutoLock lock(m_cs);    return v.back(); }
-        void    push(const T& x)    { AutoLock lock(m_cs); return v.push_back(x); };
+        void    push(const T& x)    { AutoLock lock(m_cs); return v.push(x); };
         void reserve(const size_t n) { AutoLock lock(m_cs); v.reserve(n); };
 
         AZStd::recursive_mutex& get_lock() const { return m_cs; }
@@ -57,7 +57,7 @@ namespace CryMT
             if (!v.empty())
             {
                 returnValue = v.front();
-                v.erase(v.begin());
+                v.pop();
                 return true;
             }
             return false;

Gems/Atom/RHI/DX12/Code/Source/RHI/StreamingImagePool.cpp

@@ -210,26 +210,36 @@ namespace AZ
 
             uint32_t totalTiles = request.m_sourceRegionSize.NumTiles;
 
-            // Check if heap memory is enough for the tiles. 
-            RHI::HeapMemoryUsage& memoryAllocatorUsage = GetDeviceHeapMemoryUsage();
-            size_t pageAllocationInBytes = m_tileAllocator.EvaluateMemoryAllocation(totalTiles);
-
-            // Try to release some memory if there isn't enough memory available in the pool
-            bool canAllocate = memoryAllocatorUsage.CanAllocate(pageAllocationInBytes);
-            if (!canAllocate && m_memoryReleaseCallback)
+            // protect access to m_tileAllocator
             {
-                // only try to release tiles the resource need
-                uint32_t maxUsedTiles = m_tileAllocator.GetTotalTileCount() - totalTiles;
-                bool releaseSuccess = m_memoryReleaseCallback(maxUsedTiles * m_tileAllocator.GetDescriptor().m_tileSizeInBytes);
+                AZStd::lock_guard<AZStd::mutex> lock(m_tileMutex);
+
+                // Check if heap memory is enough for the tiles.
+                RHI::HeapMemoryUsage& memoryAllocatorUsage = GetDeviceHeapMemoryUsage();
+                size_t pageAllocationInBytes = m_tileAllocator.EvaluateMemoryAllocation(totalTiles);
 
-                if (!releaseSuccess)
+                // Try to release some memory if there isn't enough memory available in the pool
+                bool canAllocate = memoryAllocatorUsage.CanAllocate(pageAllocationInBytes);
+                if (!canAllocate && m_memoryReleaseCallback)
                 {
-                    AZ_Warning("DX12::StreamingImagePool", false, "There isn't enough memory to allocate the image [%s]'s subresource %d. "
-                        "Using the default tile for the subresource. Try increase the StreamingImagePool memory budget", image.GetName().GetCStr(), subresourceIndex);
+                    // only try to release tiles the resource need
+                    uint32_t maxUsedTiles = m_tileAllocator.GetTotalTileCount() - totalTiles;
+                    bool releaseSuccess = m_memoryReleaseCallback(maxUsedTiles * m_tileAllocator.GetDescriptor().m_tileSizeInBytes);
+
+                    if (!releaseSuccess)
+                    {
+                        AZ_Warning(
+                            "DX12::StreamingImagePool",
+                            false,
+                            "There isn't enough memory to allocate the image [%s]'s subresource %d. "
+                            "Using the default tile for the subresource. Try increase the StreamingImagePool memory budget",
+                            image.GetName().GetCStr(),
+                            subresourceIndex);
+                    }
                 }
-            }
 
-            image.m_heapTiles[subresourceIndex] = m_tileAllocator.Allocate(totalTiles);
+                image.m_heapTiles[subresourceIndex] = m_tileAllocator.Allocate(totalTiles);
+            } // Unlock m_tileMutex.
 
             // If it failed to allocate tiles, use default tile for the sub-resource
             if (image.m_heapTiles[subresourceIndex].size() == 0)
@@ -340,12 +350,16 @@ namespace AZ
             image.m_tileLayout.GetSubresourceTileInfo(subresourceIndex, imageTileOffset, request.m_sourceCoordinate, request.m_sourceRegionSize);
             GetDevice().GetAsyncUploadQueue().QueueTileMapping(request);
 
-            // deallocate tiles and update image's sub-resource info
-            m_tileAllocator.DeAllocate(heapTilesList);
-            image.m_heapTiles[subresourceIndex] = {};
+            {
+                AZStd::lock_guard<AZStd::mutex> lock(m_tileMutex);
 
-            // Garbage collect the allocator immediately.
-            m_tileAllocator.GarbageCollect();
+                // deallocate tiles and update image's sub-resource info
+                m_tileAllocator.DeAllocate(heapTilesList);
+                image.m_heapTiles[subresourceIndex] = {};
+
+                // Garbage collect the allocator immediately.
+                m_tileAllocator.GarbageCollect();
+            }
         }
 
         void StreamingImagePool::AllocatePackedImageTiles(Image& image)
@@ -357,7 +371,6 @@ namespace AZ
             const ImageTileLayout& tileLayout = image.m_tileLayout;
             if (tileLayout.m_mipCountPacked)
             {
-                AZStd::lock_guard<AZStd::mutex> lock(m_tileMutex);
                 AllocateImageTilesInternal(image, tileLayout.GetPackedSubresourceIndex());
                 image.UpdateResidentTilesSizeInBytes(TileSizeInBytes);
             }
@@ -377,7 +390,6 @@ namespace AZ
             // Only proceed if the interval is still valid.
             if (mipInterval.m_min < mipInterval.m_max)
             {
-                AZStd::lock_guard<AZStd::mutex> lock(m_tileMutex);
                 for (uint32_t arrayIndex = 0; arrayIndex < descriptor.m_arraySize; ++arrayIndex)
                 {
                     for (uint32_t mipIndex = mipInterval.m_min; mipIndex < mipInterval.m_max; ++mipIndex)
@@ -411,7 +423,6 @@ namespace AZ
                 const Fence& fence = compiledFences.GetFence(RHI::HardwareQueueClass::Graphics);
                 GetDevice().GetAsyncUploadQueue().QueueWaitFence(fence, fence.GetPendingValue());
 
-                AZStd::lock_guard<AZStd::mutex> lock(m_tileMutex);
                 for (uint32_t arrayIndex = 0; arrayIndex < descriptor.m_arraySize; ++arrayIndex)
                 {
                     for (uint32_t mipIndex = mipInterval.m_min; mipIndex < mipInterval.m_max; ++mipIndex)
@@ -542,8 +553,8 @@ namespace AZ
         {
             Image& image = static_cast<Image&>(resourceBase);
 
-            // Wait for any upload of this image done. 
-            GetDevice().GetAsyncUploadQueue().WaitForUpload(image.GetUploadFenceValue());
+            // Wait for any upload of this image done.
+            WaitFinishUploading(image);
 
             if (auto* resolver = GetResolver())
             {
@@ -580,6 +591,9 @@ namespace AZ
         {
             Image& image = static_cast<Image&>(*request.m_image);
 
+            // Wait for any upload of this image done.
+            WaitFinishUploading(image);
+
             const uint32_t residentMipLevelBefore = image.GetResidentMipLevel();
             const uint32_t residentMipLevelAfter = residentMipLevelBefore - static_cast<uint32_t>(request.m_mipSlices.size());
 
@@ -612,7 +626,7 @@ namespace AZ
             Image& imageImpl = static_cast<Image&>(image);
 
             // Wait for any upload of this image done. 
-            GetDevice().GetAsyncUploadQueue().WaitForUpload(imageImpl.GetUploadFenceValue());
+            WaitFinishUploading(imageImpl);
 
             // Set streamed mip level to target mip level
             if (imageImpl.GetStreamedMipLevel() < targetMipLevel)
@@ -670,6 +684,11 @@ namespace AZ
         {
             return m_enableTileResource;
         }
+
+        void StreamingImagePool::WaitFinishUploading(const Image& image)
+        {
+            GetDevice().GetAsyncUploadQueue().WaitForUpload(image.GetUploadFenceValue());
+        }
     }
 }
 

Gems/Atom/RHI/DX12/Code/Source/RHI/StreamingImagePool.h

@@ -77,6 +77,10 @@ namespace AZ
             // Get the data reference of device heap memory usage 
             RHI::HeapMemoryUsage& GetDeviceHeapMemoryUsage();
 
+            // A helper function that makes sure any previous upload request
+            // is actually completed on @image.
+            void WaitFinishUploading(const Image& image);
+
             // whether to enable tiled resource
             bool m_enableTileResource = false;
 

Gems/Atom/RPI/Code/Include/Atom/RPI.Public/Material/Material.h

@@ -144,7 +144,8 @@ namespace AZ
             bool NeedsCompile() const;
 
             using OnMaterialShaderVariantReadyEvent = AZ::Event<>;
-            //! Connect a handler to listen to the event that a shader variant asset of the shaders used by this material is ready
+            //! Connect a handler to listen to the event that a shader variant asset of the shaders used by this material is ready.
+            //! This is a thread safe function.
             void ConnectEvent(OnMaterialShaderVariantReadyEvent::Handler& handler);
 
         private:
@@ -225,6 +226,9 @@ namespace AZ
 
             MaterialPropertyPsoHandling m_psoHandling = MaterialPropertyPsoHandling::Warning;
 
+            //! AZ::Event is not thread safe, so we have to do our own thread safe code
+            //! because MeshDrawPacket can connect to this event from different threads.
+            AZStd::recursive_mutex m_shaderVariantReadyEventMutex;
             OnMaterialShaderVariantReadyEvent m_shaderVariantReadyEvent;
         };
 

Gems/Atom/RPI/Code/Include/Atom/RPI.Public/Shader/Shader.h

@@ -52,7 +52,7 @@ namespace AZ
         //! If you need guarantee lifetime, it is safe to take a reference on the returned pipeline state.
         class Shader final
             : public Data::InstanceData
-            , public Data::AssetBus::Handler
+            , public Data::AssetBus::MultiHandler
             , public ShaderVariantFinderNotificationBus::Handler
         {
             friend class ShaderSystem;
@@ -183,6 +183,21 @@ namespace AZ
             //! A strong reference to the shader asset.
             Data::Asset<ShaderAsset> m_asset;
 
+            /////////////////////////////////////////////////////////////////////////////////////
+            //! The following variables are necessary to reliably reload the Shader
+            //! whenever the Shader source assets and dependencies change.
+            //! 
+            //! Each time the Shader is initialized, this variable
+            //! caches all the Assets that we are expecting to be reloaded whenever
+            //! the Shader asset changes. This includes m_asset + each Supervariant ShaderVariantAsset.
+            //! Typically most shaders only contain one Supervariant, so this variable becomes 2. 
+            size_t m_expectedAssetReloadCount = 0;
+            //! Each time one of the assets is reloaded we store it here, and when the
+            //! size of this dictionary equals @m_expectedAssetReloadCount then we know it is safe
+            //! to reload the Shader.
+            AZStd::unordered_map<Data::AssetId, Data::Asset<Data::AssetData>> m_reloadedAssets;
+            /////////////////////////////////////////////////////////////////////////////////////
+
             //! Selects current supervariant to be used.
             //! This value is defined at instantiation.
             SupervariantIndex m_supervariantIndex;

Gems/Atom/RPI/Code/Include/Atom/RPI.Reflect/Asset/AssetUtils.h

@@ -10,7 +10,7 @@
 
 #include <AzCore/Asset/AssetCommon.h>
 #include <AzCore/Asset/AssetManager.h>
-
+#include <AzCore/Component/TickBus.h>
 #include <AzFramework/Asset/AssetSystemBus.h>
 
 namespace AZ
@@ -193,7 +193,25 @@ namespace AZ
             //! multiple ebus functions to handle callbacks. It will invoke the provided callback function when the
             //! asset loads or errors. It will stop listening on destruction, so it should be held onto until the
             //! callback fires.
-            class AsyncAssetLoader : private Data::AssetBus::Handler
+            //! This class will always invoke the callback during OnSystemTick() to prevent
+            //! deadlocks related with StreamingImage assets. Here is a quick summary of the deadlock
+            //! this class avoids:
+             //** Main Thread                | ** Secondary Copy Queue Thread                                                
+             //AssetBus::lock(mutex)         |                                                 
+             //AssetBus::OnAssetReady        |                                                 
+             //StreamingImage::FindOrCreate  |                                                 
+             //AsyncUploadQueue::queueWork   |                                                 
+             //Wait For Work Complete        |                                                 
+             //                              |                      
+             //                              | workQueue signaled                              
+             //                              | Pop Work                                        
+             //                              | StreaminImage::Destructor()                     
+             //                              | AssetBus::Disconnect()                          
+             //                              | AssetBus::lock(mutex) <- Deadlocked             
+             //                                                                                                           
+            class AsyncAssetLoader :
+                private Data::AssetBus::Handler,
+                private SystemTickBus::Handler
             {
             public:
                 AZ_RTTI(AZ::RPI::AssetUtils::AsyncAssetLoader, "{E0FB5B08-B97D-40DF-8478-226249C0B654}");
@@ -219,6 +237,12 @@ namespace AZ
                 void OnAssetReady(Data::Asset<Data::AssetData> asset) override;
                 void OnAssetError(Data::Asset<Data::AssetData> asset) override;
 
+                // SystemTickBus::Handler overrides..
+                void OnSystemTick() override;
+
+                // This function should never be called directly under the scope
+                // of any of the AssetBus::OnAssetXXXX() functions to avoid deadlocks
+                // when working with StreaminImage assets.
                 void HandleCallback(Data::Asset<Data::AssetData> asset);
 
                 AssetCallback m_callback;

Gems/Atom/RPI/Code/Include/Atom/RPI.Reflect/Shader/ShaderAsset.h

@@ -258,6 +258,13 @@ namespace AZ
                 AZStd::vector<Supervariant> m_supervariants;
             };
 
+            //! Searches across all Supervariants for the matching AssetId of the each Root
+            //! ShaderVariantAsset. The first root ShaderVariantAsset that matches is replaced
+            //! with @shaderVariantAsset.
+            //! @returns true if a matching AssetId was found and replaced.
+            //! @remark This function is only useful during Shader::OnAssetReloaded.
+            bool UpdateRootShaderVariantAsset(Data::Asset<ShaderVariantAsset> shaderVariantAsset);
+
             bool PostLoadInit() override;
             void SetReady();
 

Gems/Atom/RPI/Code/Source/RPI.Private/Module.cpp

@@ -13,12 +13,14 @@
 #include <Atom/RPI.Public/Image/ImageTagSystemComponent.h>
 #include <Atom/RPI.Public/Model/ModelTagSystemComponent.h>
 #include <RPI.Private/RPISystemComponent.h>
+#include <RPI.Private/PassTemplatesAutoLoader.h>
 
 AZ::RPI::Module::Module()
 {
     m_descriptors.push_back(AZ::RPI::RPISystemComponent::CreateDescriptor());
     m_descriptors.push_back(AZ::RPI::ImageTagSystemComponent::CreateDescriptor());
     m_descriptors.push_back(AZ::RPI::ModelTagSystemComponent::CreateDescriptor());
+    m_descriptors.push_back(AZ::RPI::PassTemplatesAutoLoader::CreateDescriptor());
 }
 
 AZ::ComponentTypeList AZ::RPI::Module::GetRequiredSystemComponents() const
@@ -27,7 +29,8 @@ AZ::ComponentTypeList AZ::RPI::Module::GetRequiredSystemComponents() const
     {
         azrtti_typeid<AZ::RPI::RPISystemComponent>(),
         azrtti_typeid<AZ::RPI::ImageTagSystemComponent>(),
-        azrtti_typeid<AZ::RPI::ModelTagSystemComponent>()
+        azrtti_typeid<AZ::RPI::ModelTagSystemComponent>(),
+        azrtti_typeid<AZ::RPI::PassTemplatesAutoLoader>()
     };
 }
 

Gems/Atom/RPI/Code/Source/RPI.Private/PassTemplatesAutoLoader.cpp

@@ -0,0 +1,135 @@
+/*
+ * Copyright (c) Contributors to the Open 3D Engine Project.
+ * For complete copyright and license terms please see the LICENSE at the root of this distribution.
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ *
+ */
+
+
+#include <RPI.Private/PassTemplatesAutoLoader.h>
+
+#include <AzCore/Serialization/EditContext.h>
+#include <AzCore/Settings/SettingsRegistry.h>
+#include <AzCore/Utils/Utils.h>
+
+#include <AzFramework/Gem/GemInfo.h>
+
+#include <Atom/RPI.Reflect/Asset/AssetUtils.h>
+
+namespace AZ
+{
+    namespace RPI
+    {
+        void PassTemplatesAutoLoader::Reflect(ReflectContext* context)
+        {
+            if (auto* serializeContext = azrtti_cast<SerializeContext*>(context))
+            {
+                serializeContext
+                    ->Class<PassTemplatesAutoLoader, Component>()
+                    ->Version(0)
+                    ;
+
+                if (AZ::EditContext* ec = serializeContext->GetEditContext())
+                {
+                    ec->Class<PassTemplatesAutoLoader>("PassTemplatesAutoLoader", "A service that loads PassTemplates.")
+                        ->ClassElement(AZ::Edit::ClassElements::EditorData, "")
+                        ->Attribute(AZ::Edit::Attributes::AutoExpand, true)
+                        ;
+                }
+            }
+        }
+
+        void PassTemplatesAutoLoader::GetRequiredServices(ComponentDescriptor::DependencyArrayType& required)
+        {
+            required.push_back(AZ_CRC_CE("RPISystem"));
+        }
+
+        void PassTemplatesAutoLoader::GetProvidedServices(ComponentDescriptor::DependencyArrayType& provided)
+        {
+            provided.push_back(AZ_CRC_CE("PassTemplatesAutoLoader"));
+        }
+
+        void PassTemplatesAutoLoader::GetIncompatibleServices(AZ::ComponentDescriptor::DependencyArrayType& incompatible)
+        {
+            incompatible.push_back(AZ_CRC_CE("PassTemplatesAutoLoader"));
+        }
+
+        void PassTemplatesAutoLoader::Activate()
+        {
+            // Register the event handler.
+            m_loadTemplatesHandler = AZ::RPI::PassSystemInterface::OnReadyLoadTemplatesEvent::Handler(
+                [&]()
+                {
+                    LoadPassTemplates();
+                });
+            AZ::RPI::PassSystemInterface::Get()->ConnectEvent(m_loadTemplatesHandler);
+        }
+
+        void PassTemplatesAutoLoader::Deactivate()
+        {
+
+        }
+
+        void PassTemplatesAutoLoader::LoadPassTemplates()
+        {
+            auto* settingsRegistry = AZ::SettingsRegistry::Get();
+            AZStd::vector<AzFramework::GemInfo> gemInfoList;
+            if (!AzFramework::GetGemsInfo(gemInfoList, *settingsRegistry))
+            {
+                AZ_Warning(LogWindow, false, "%s Failed to get Gems info.\n", __FUNCTION__);
+                return;
+            }
+
+            auto* passSystemInterface = AZ::RPI::PassSystemInterface::Get();
+            AZStd::unordered_set<AZStd::string> loadedTemplates; // See CAVEAT below.
+            auto loadFunc = [&](const AZStd::string& assetPath)
+            {
+                if (loadedTemplates.count(assetPath) > 0)
+                {
+                    // CAVEAT: Most of the times Game Projects contain a Gem of the same name
+                    // inside of them, this is why we check first with @loadedTemplates before attempting to load
+                    // the PassTemplate asset located at <PROJECT_ROOT>/Passes/<PROJECT_NAME>/AutoLoadPassTemplates.azasset
+                    return;
+                }
+                const auto assetId = AssetUtils::GetAssetIdForProductPath(assetPath.c_str(), AssetUtils::TraceLevel::None);
+                if (!assetId.IsValid())
+                {
+                    // This is the most common scenario.
+                    return;
+                }
+                if (!passSystemInterface->LoadPassTemplateMappings(assetPath))
+                {
+                    AZ_Error(LogWindow, false, "Failed to load PassTemplates at '%s'.\n", assetPath.c_str());
+                    return;
+                }
+                AZ_Printf(LogWindow, "Successfully load PassTemplates from '%s'.\n", assetPath.c_str());
+                loadedTemplates.emplace(AZStd::move(assetPath));
+            };
+
+            for (const auto& gemInfo : gemInfoList)
+            {
+                AZStd::string assetPath = AZStd::string::format("Passes/%s/AutoLoadPassTemplates.azasset", gemInfo.m_gemName.c_str());
+                loadFunc(assetPath);
+            }
+
+            // Besides the Gems, a Game Project can also provide PassTemplates.
+            // <PROJECT_ROOT>/Assets/Passes/<PROJECT_NAME>/AutoLoadPassTemplates.azasset
+            // <PROJECT_ROOT>/Passes/<PROJECT_NAME>/AutoLoadPassTemplates.azasset
+            const auto projectName = AZ::Utils::GetProjectName(settingsRegistry);
+            if (!projectName.empty())
+            {
+                {
+                    AZStd::string assetPath = AZStd::string::format("Passes/%s/AutoLoadPassTemplates.azasset", projectName.c_str());
+                    loadFunc(assetPath);
+                }
+
+                {
+                    AZStd::string assetPath = AZStd::string::format("Assets/Passes/%s/AutoLoadPassTemplates.azasset", projectName.c_str());
+                    loadFunc(assetPath);
+                }
+            }
+        }
+
+    } // namespace RPI
+} // namespace AZ

Gems/Atom/RPI/Code/Source/RPI.Private/PassTemplatesAutoLoader.h

@@ -0,0 +1,81 @@
+/*
+ * Copyright (c) Contributors to the Open 3D Engine Project.
+ * For complete copyright and license terms please see the LICENSE at the root of this distribution.
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ *
+ */
+
+/**
+ * @file PassTemplatesAutoLoader.h
+ * @brief Contains the definition of the PassTemplatesAutoLoader which provides
+ *        a data-driven alternative for Gems and Projects to load custom PassTemplates.
+ */
+#pragma once
+
+#include <AzCore/Component/Component.h>
+
+#include <Atom/RPI.Public/Pass/PassSystemInterface.h>
+
+namespace AZ
+{
+    namespace RPI
+    {
+        /**
+         * @brief A data-driven System Component that loads PassTemplates across all Gems and the Game Project.
+         * @detail This service provides an opt-in mechanism for Gems and any Game Project to load
+         *         custom PassTemplates.azasset WITHOUT having to write C++ code.
+                   This system component works as a convenience service because there's already an API
+         *         in AZ::RPI::PassSystemInterface::OnReadyLoadTemplatesEvent that helps Gems and Game Projects
+         *         load their custom PassTemplates (*.azasset), the problem is, of course, that C++ code
+         *         needs to be written to invoque the API. And this is where this System Component comes
+         *         to the rescue.
+         *         How it works?
+         *         This service, at startup time, looks across all active Gems for assets with the following
+         *         naming convention:
+         *         "Passes/<Gem Name>/AutoLoadPassTemplates.azassset".
+         *         or (Applicable to the Game Project)
+         *         "Passes/<Project Name>/AutoLoadPassTemplates.azassset".
+         *         or (Applicable to the Game Project)
+         *         "Assets/Passes/<Project Name>/AutoLoadPassTemplates.azassset".
+         *         If any of those asset paths exist, this service will automatically add those
+         *         PassTemplates to the PassLibrary.
+         */
+        class PassTemplatesAutoLoader final
+            : public AZ::Component
+        {
+        public:
+            AZ_COMPONENT(PassTemplatesAutoLoader, "{75FEC6CC-ACA7-419C-8A63-4286998CBC0B}");
+
+            static void Reflect(AZ::ReflectContext* context);
+            static void GetRequiredServices(AZ::ComponentDescriptor::DependencyArrayType& required);
+            static void GetProvidedServices(AZ::ComponentDescriptor::DependencyArrayType& provided);
+            static void GetIncompatibleServices(AZ::ComponentDescriptor::DependencyArrayType& incompatible);
+
+            PassTemplatesAutoLoader() = default;
+            ~PassTemplatesAutoLoader() override = default;
+
+            void Activate() override;
+            void Deactivate() override;
+
+        private:
+            PassTemplatesAutoLoader(const PassTemplatesAutoLoader&) = delete;
+
+            static constexpr char LogWindow[] = "PassTemplatesAutoLoader";
+
+            //! Loads PassTemplates across all Gems and the Game Project according to the following
+            //! naming convention:
+            //! "Passes/<Gem Name>/AutoLoadPassTemplates.azassset".
+            //! Or (Applicable to the Game Project)
+            //! "Passes/<Project Name>/AutoLoadPassTemplates.azassset".
+            //! Or (Applicable to the Game Project)
+            //! "Assets/Passes/<Project Name>/AutoLoadPassTemplates.azassset".
+            void LoadPassTemplates();
+
+            //! We use this event handler to register with RPI::PassSystem to be notified
+            //! of the right time to load PassTemplates.
+            //! The callback will invoke this->LoadPassTemplates()
+            AZ::RPI::PassSystemInterface::OnReadyLoadTemplatesEvent::Handler m_loadTemplatesHandler;
+        };
+    } // namespace RPI
+} // namespace AZ

Gems/Atom/RPI/Code/Source/RPI.Public/Material/Material.cpp

@@ -66,6 +66,7 @@ namespace AZ
             m_generalShaderCollection = {};
             m_materialPipelineData = {};
             m_materialAsset = { &materialAsset, AZ::Data::AssetLoadBehavior::PreLoad };
+
             ShaderReloadNotificationBus::MultiHandler::BusDisconnect();
             if (!m_materialAsset.IsReady())
             {
@@ -322,10 +323,32 @@ namespace AZ
         {
             ShaderReloadDebugTracker::ScopedSection reloadSection("{%p}->Material::OnShaderVariantReinitialized %s", this, shaderVariant.GetShaderVariantAsset().GetHint().c_str());
 
+            // Move m_shaderVariantReadyEvent to a local AZ::Event in order to allow
+            // the handlers to be signaled outside of the mutex lock.
+            // This allows other threads to register their handlers while this thread
+            // is invoking Signal() on the current snapshot of handlers.
+            decltype(m_shaderVariantReadyEvent) localShaderVariantReadyEvent;
+            {
+                AZStd::scoped_lock lock(m_shaderVariantReadyEventMutex);
+                localShaderVariantReadyEvent = AZStd::move(m_shaderVariantReadyEvent);
+            }
+
             // Note: we don't need to re-compile the material if a shader variant is ready or changed
             // The DrawPacket created for the material need to be updated since the PSO need to be re-creaed.
-            // This event can be used to notify the owners to update their DrawPackets
-            m_shaderVariantReadyEvent.Signal();
+            // This event can be used to notify the owners to update their DrawPackets.
+            localShaderVariantReadyEvent.Signal();
+
+            // Finally restore m_shaderVariantReadyEvent but making sure to claim any new handlers that were added
+            // in other threads while Signal() was being called.
+            {
+                // Swap the local handlers with the current m_notifiers which
+                // will contain any handlers added during the signaling of the
+                // local event
+                AZStd::scoped_lock lock(m_shaderVariantReadyEventMutex);
+                AZStd::swap(m_shaderVariantReadyEvent, localShaderVariantReadyEvent);
+                // Append any added handlers to the m_notifier structure
+                m_shaderVariantReadyEvent.ClaimHandlers(AZStd::move(localShaderVariantReadyEvent));
+            }
         }
 
         void Material::ReInitKeepPropertyValues()
@@ -377,6 +400,7 @@ namespace AZ
 
         void Material::ConnectEvent(OnMaterialShaderVariantReadyEvent::Handler& handler)
         {
+            AZStd::scoped_lock lock(m_shaderVariantReadyEventMutex);
             handler.Connect(m_shaderVariantReadyEvent);
         }
 

Gems/Atom/RPI/Code/Source/RPI.Public/Shader/Shader.cpp

@@ -121,7 +121,7 @@ namespace AZ
 
         RHI::ResultCode Shader::Init(ShaderAsset& shaderAsset)
         {
-            Data::AssetBus::Handler::BusDisconnect();
+            Data::AssetBus::MultiHandler::BusDisconnect();
             ShaderVariantFinderNotificationBus::Handler::BusDisconnect();
 
             RHI::RHISystemInterface* rhiSystem = RHI::RHISystemInterface::Get();
@@ -171,15 +171,22 @@ namespace AZ
             }
 
             ShaderVariantFinderNotificationBus::Handler::BusConnect(m_asset.GetId());
-            Data::AssetBus::Handler::BusConnect(m_asset.GetId());
 
+            m_reloadedAssets.clear();
+            const auto& supervariants = m_asset->GetCurrentShaderApiData().m_supervariants;
+            m_expectedAssetReloadCount = 1 /*m_asset*/ + supervariants.size();
+            Data::AssetBus::MultiHandler::BusConnect(m_asset.GetId());
+            for (const auto& supervariant : supervariants)
+            {
+                Data::AssetBus::MultiHandler::BusConnect(supervariant.m_rootShaderVariantAsset.GetId());
+            }
             return RHI::ResultCode::Success;
         }
 
         void Shader::Shutdown()
         {
             ShaderVariantFinderNotificationBus::Handler::BusDisconnect();
-            Data::AssetBus::Handler::BusDisconnect();
+            Data::AssetBus::MultiHandler::BusDisconnect();
 
             if (m_pipelineLibraryHandle.IsValid())
             {
@@ -205,18 +212,75 @@ namespace AZ
         // AssetBus overrides
         void Shader::OnAssetReloaded(Data::Asset<Data::AssetData> asset)
         {
-            ShaderReloadDebugTracker::ScopedSection reloadSection("{%p}->Shader::OnAssetReloaded %s", this, asset.GetHint().c_str());
+            ShaderReloadDebugTracker::ScopedSection reloadSection("{%p}->Shader::OnAssetReloaded %s.\n",
+                this, asset.GetHint().c_str());
 
-            m_asset = asset;
+            m_reloadedAssets.emplace(asset.GetId(), asset);
 
             if (ShaderReloadDebugTracker::IsEnabled())
             {
+                ShaderReloadDebugTracker::Printf(
+                    "Current ShaderAssetPtr={%p} with RootVariantAssetPtr={%p}", m_asset.Get(), m_asset->GetRootVariantAsset().Get());
+
+                ShaderReloadDebugTracker::Printf("{%p} -> Shader::OnAssetReloaded so far only %zu of %zu assets have been reloaded.",
+                    this, m_reloadedAssets.size(), m_expectedAssetReloadCount);
+
                 AZStd::sys_time_t now = AZStd::GetTimeUTCMilliSecond();
+                if (asset.GetType() == AZ::AzTypeInfo<ShaderVariantAsset>::Uuid())
+                {
+                    ShaderReloadDebugTracker::Printf(
+                        "{%p}->Shader::OnRootVariantReloaded [current time %lld] got new variant {%p}'%s'",
+                        this,
+                        now,
+                        asset.Get(),
+                        asset.GetHint().c_str());
+                }
+                else
+                {
+                    const auto* newShaderAsset = asset.GetAs<ShaderAsset>();
+                    const auto shaderVariantAsset = newShaderAsset->GetRootVariantAsset();
+                    ShaderReloadDebugTracker::Printf(
+                        "{%p}->Shader::OnShaderAssetReloaded [current time %lld] got new shader {%p}'%s' with included variant {%p}'%s'",
+                        this,
+                        now,
+                        newShaderAsset,
+                        asset.GetHint().c_str(),
+                        shaderVariantAsset.Get(),
+                        shaderVariantAsset.GetHint().c_str());
+                }
+            }
+
+            if (m_reloadedAssets.size() != m_expectedAssetReloadCount)
+            {
+                return;
+            }
 
-                const auto shaderVariantAsset = m_asset->GetRootVariantAsset();
-                ShaderReloadDebugTracker::Printf("{%p}->Shader::OnAssetReloaded for shader '%s' [current time %lld] found variant '%s'",
-                    this, m_asset.GetHint().c_str(), now, shaderVariantAsset.GetHint().c_str());
+            // Time to update all references:
+            auto itor = m_reloadedAssets.find(m_asset.GetId());
+            if (itor == m_reloadedAssets.end())
+            {
+                AZ_Error("Shader", false, "Can not find the reloaded ShaderAsset with ID '%s'. Hint '%s'",
+                    m_asset.GetId().ToString<AZStd::string>().c_str(),
+                    m_asset.GetHint().c_str());
+                return;
             }
+
+            m_asset = itor->second;
+            m_reloadedAssets.erase(itor);
+            for (auto& [assetId, rootVariantAsset] : m_reloadedAssets)
+            {
+                AZ_Assert(rootVariantAsset.GetType() == AZ::AzTypeInfo<ShaderVariantAsset>::Uuid(),
+                    "Was expecting only ShaderVariantAsset(s)");
+                if (!m_asset->UpdateRootShaderVariantAsset(Data::static_pointer_cast<ShaderVariantAsset>(rootVariantAsset)))
+                {
+                    AZ_Error("Shader", false,
+                        "Failed to update Root ShaderVariantAsset {%p}'%s'",
+                        rootVariantAsset.Get(),
+                        rootVariantAsset.GetHint().c_str());
+                }
+            }
+            m_reloadedAssets.clear();
+
             Init(*m_asset.Get());
             ShaderReloadNotificationBus::Event(asset.GetId(), &ShaderReloadNotificationBus::Events::OnShaderReinitialized, *this);
         }

Gems/Atom/RPI/Code/Source/RPI.Reflect/Asset/AssetUtils.cpp

@@ -89,21 +89,26 @@ namespace AZ
             AsyncAssetLoader::~AsyncAssetLoader()
             {
                 Data::AssetBus::Handler::BusDisconnect();
+                SystemTickBus::Handler::BusDisconnect();
             }
 
             void AsyncAssetLoader::OnAssetReady(Data::Asset<Data::AssetData> asset)
             {
-                HandleCallback(asset);
+                Data::AssetBus::Handler::BusDisconnect();
+                m_asset = asset;
+                SystemTickBus::Handler::BusConnect();
+
             }
 
             void AsyncAssetLoader::OnAssetError(Data::Asset<Data::AssetData> asset)
             {
-                HandleCallback(asset);
+                Data::AssetBus::Handler::BusDisconnect();
+                m_asset = asset;
+                SystemTickBus::Handler::BusConnect();
             }
 
             void AsyncAssetLoader::HandleCallback(Data::Asset<Data::AssetData> asset)
             {
-                Data::AssetBus::Handler::BusDisconnect();
                 if (m_callback)
                 {
                     m_callback(asset);
@@ -113,6 +118,13 @@ namespace AZ
                 m_asset = {}; // Release the asset in case this AsyncAssetLoader hangs around longer than the asset needs to.
             }
 
+            // SystemTickBus::Handler overrides..
+            void AsyncAssetLoader::OnSystemTick()
+            {
+                SystemTickBus::Handler::BusDisconnect();
+                HandleCallback(m_asset);
+            }
+
         } // namespace AssetUtils
     } // namespace RPI
 } // namespace AZ

Gems/Atom/RPI/Code/Source/RPI.Reflect/Shader/ShaderAsset.cpp

@@ -574,6 +574,20 @@ namespace AZ
             return true;
         }
 
+        bool ShaderAsset::UpdateRootShaderVariantAsset(Data::Asset<ShaderVariantAsset> shaderVariantAsset)
+        {
+            auto& supervariants = GetCurrentShaderApiData().m_supervariants;
+            for (auto& supervariant : supervariants)
+            {
+                if (supervariant.m_rootShaderVariantAsset.GetId() == shaderVariantAsset.GetId())
+                {
+                    supervariant.m_rootShaderVariantAsset = shaderVariantAsset;
+                    return true;
+                }
+            }
+            return false;
+        }
+
         bool ShaderAsset::PostLoadInit()
         {
             ShaderVariantFinderNotificationBus::Handler::BusConnect(GetId());

Gems/Atom/RPI/Code/atom_rpi_private_files.cmake

@@ -13,4 +13,6 @@ set(FILES
     Source/RPI.Private/RPISystemComponent.h
     Source/RPI.Private/PerformanceCVarManager.cpp
     Source/RPI.Private/PerformanceCVarManager.h
+    Source/RPI.Private/PassTemplatesAutoLoader.cpp
+    Source/RPI.Private/PassTemplatesAutoLoader.h
 )

Gems/AtomLyIntegration/CommonFeatures/Code/Source/ImageBasedLights/ImageBasedLightComponentController.cpp

@@ -131,22 +131,43 @@ namespace AZ
 
         void ImageBasedLightComponentController::UpdateWithAsset(Data::Asset<Data::AssetData> updatedAsset)
         {
-            if (m_configuration.m_specularImageAsset.GetId() == updatedAsset.GetId())
+            // REMARK: This function is typically invoked within the context of one of the AssetBus::OnAssetXXX functions,
+            // and a deadlock may occur according to the following sequence:
+            // 1. Starting from Main thread, AssetBus locks a mutex.
+            // 2. AssetBus calls OnAssetReady and it enters in this function.
+            // 3. Start the instantiation of a new StreamingImage.
+            // 4. StreamingImage asynchronously queues work in the "Seconday Copy Queue".
+            // 5. StreamingImage waits until the work completes.
+            // 6. The thread of "Seconday Copy Queue" gets a new work item, which may hold a reference
+            //    to an old StreamingImage.
+            // 7. The old StreamingImage gets destroyed and it calls AssetBus::MultiHandler::BusDisconnect(GetAssetId());
+            // 8. When calling AssetBus::MultiHandler::BusDisconnect(GetAssetId()); it tries to lock the same mutex
+            //    from step 1. But the mutex is already locked on Main Thread in step 1.
+            // 9. The "Seconday Copy Queue" thread deadlocks and never completes the work.
+            // 10. Main thread is also deadlocked waiting for "Seconday Copy Queue" to complete.
+            // The solution is to enqueue texture update on the next tick.
+            auto postTickLambda = [=]()
             {
-                if (m_featureProcessor && HandleAssetUpdate(updatedAsset, m_configuration.m_specularImageAsset))
+                if (m_configuration.m_specularImageAsset.GetId() == updatedAsset.GetId())
                 {
-                    m_featureProcessor->SetSpecularImage(m_configuration.m_specularImageAsset);
-                    ImageBasedLightComponentNotificationBus::Event(m_entityId, &ImageBasedLightComponentNotifications::OnSpecularImageUpdated);
+                    if (m_featureProcessor && HandleAssetUpdate(updatedAsset, m_configuration.m_specularImageAsset))
+                    {
+                        m_featureProcessor->SetSpecularImage(m_configuration.m_specularImageAsset);
+                        ImageBasedLightComponentNotificationBus::Event(
+                            m_entityId, &ImageBasedLightComponentNotifications::OnSpecularImageUpdated);
+                    }
                 }
-            }
-            else if (m_configuration.m_diffuseImageAsset.GetId() == updatedAsset.GetId())
-            {
-                if (m_featureProcessor && HandleAssetUpdate(updatedAsset, m_configuration.m_diffuseImageAsset))
+                else if (m_configuration.m_diffuseImageAsset.GetId() == updatedAsset.GetId())
                 {
-                    m_featureProcessor->SetDiffuseImage(m_configuration.m_diffuseImageAsset);
-                    ImageBasedLightComponentNotificationBus::Event(m_entityId, &ImageBasedLightComponentNotifications::OnDiffuseImageUpdated);
+                    if (m_featureProcessor && HandleAssetUpdate(updatedAsset, m_configuration.m_diffuseImageAsset))
+                    {
+                        m_featureProcessor->SetDiffuseImage(m_configuration.m_diffuseImageAsset);
+                        ImageBasedLightComponentNotificationBus::Event(
+                            m_entityId, &ImageBasedLightComponentNotifications::OnDiffuseImageUpdated);
+                    }
                 }
-            }
+            };
+            AZ::TickBus::QueueFunction(AZStd::move(postTickLambda));
         }
 
         bool ImageBasedLightComponentController::HandleAssetUpdate(Data::Asset<Data::AssetData> updatedAsset, Data::Asset<RPI::StreamingImageAsset>& configAsset)

Gems/AtomLyIntegration/CommonFeatures/Code/Source/Material/MaterialComponentController.cpp

@@ -175,6 +175,13 @@ namespace AZ
 
         void MaterialComponentController::OnSystemTick()
         {
+            while (!m_notifiedMaterialAssets.empty())
+            {
+                auto materialAsset = m_notifiedMaterialAssets.front();
+                m_notifiedMaterialAssets.pop();
+                InitializeNotifiedMaterialAsset(materialAsset);
+            }
+
             if (m_queuedLoadMaterials)
             {
                 m_queuedLoadMaterials = false;
@@ -292,7 +299,7 @@ namespace AZ
             }
         }
 
-        void MaterialComponentController::InitializeMaterialInstance(const Data::Asset<Data::AssetData>& asset)
+        void MaterialComponentController::InitializeNotifiedMaterialAsset(Data::Asset<Data::AssetData> asset)
         {
             bool allReady = true;
             auto updateAsset = [&](AZ::Data::Asset<AZ::RPI::MaterialAsset>& materialAsset)
@@ -340,6 +347,13 @@ namespace AZ
             }
         }
 
+        void MaterialComponentController::InitializeMaterialInstance(Data::Asset<Data::AssetData> asset)
+        {
+            // See header file, where @m_notifiedMaterialAssets is declared for details.
+            m_notifiedMaterialAssets.push(asset);
+            SystemTickBus::Handler::BusConnect();
+        }
+
         void MaterialComponentController::ReleaseMaterials()
         {
             SystemTickBus::Handler::BusDisconnect();
@@ -354,6 +368,8 @@ namespace AZ
             {
                 materialPair.second.Release();
             }
+            decltype(m_notifiedMaterialAssets) tmpQueue;
+            AZStd::swap(m_notifiedMaterialAssets, tmpQueue);
         }
 
         MaterialAssignmentMap MaterialComponentController::GetDefaultMaterialMap() const

Gems/AtomLyIntegration/CommonFeatures/Code/Source/Material/MaterialComponentController.h

@@ -96,7 +96,11 @@ namespace AZ
             void OnSystemTick() override;
 
             void LoadMaterials();
-            void InitializeMaterialInstance(const Data::Asset<Data::AssetData>& asset);
+            // Typically called from thread context of Data::AssetBus::MultiHandler::OnAssetXXX.
+            void InitializeMaterialInstance(Data::Asset<Data::AssetData> asset);
+            // Must be called on main thread.
+            // Called for each asset in @m_notifiedMaterialAssets only during SystemTick.
+            void InitializeNotifiedMaterialAsset(Data::Asset<Data::AssetData> asset);
             void ReleaseMaterials();
             //! Queue applying property overrides to material instances until tick
             void QueuePropertyChanges(const MaterialAssignmentId& materialAssignmentId);
@@ -121,6 +125,24 @@ namespace AZ
             MaterialAssignmentMap m_defaultMaterialMap;
             AZStd::unordered_map<AZ::Data::AssetId, AZ::Data::Asset<AZ::RPI::MaterialAsset>> m_uniqueMaterialMap;
             AZStd::unordered_set<MaterialAssignmentId> m_materialsWithDirtyProperties;
+            //! We store here references to all the material assets for which we are connected to the
+            //! AssetBus for notifications. Instead of taking action upon being notified, we simply store
+            //! the asset here, and in the next SystemTick we'll process the assets. This is done, for the following reason:
+            //! When AssetBus::OnAssetXXX functions are called, a deadlock may occur according to the following sequence:
+            // 1. Starting from Main thread, AssetBus locks a mutex.
+            // 2. AssetBus calls OnAssetReady and it enters in this function.
+            // 3. Start the instantiation of a new StreamingImage.
+            // 4. StreamingImage asynchronously queues work in the "Seconday Copy Queue".
+            // 5. StreamingImage waits until the work completes.
+            // 6. The thread of "Seconday Copy Queue" gets a new work item, which may hold a reference
+            //    to an old StreamingImage.
+            // 7. The old StreamingImage gets destroyed and it calls AssetBus::MultiHandler::BusDisconnect(GetAssetId());
+            // 8. When calling AssetBus::MultiHandler::BusDisconnect(GetAssetId()); it tries to lock the same mutex
+            //    from step 1. But the mutex is already locked on Main Thread in step 1.
+            // 9. The "Seconday Copy Queue" thread deadlocks and never completes the work.
+            // 10. Main thread is also deadlocked waiting for "Seconday Copy Queue" to complete.
+            // The solution is to enqueue texture update on the next tick.
+            AZStd::queue<Data::Asset<Data::AssetData>> m_notifiedMaterialAssets;
             bool m_queuedMaterialsCreatedNotification = false;
             bool m_queuedMaterialsUpdatedNotification = false;
             bool m_queuedLoadMaterials = false;

Gems/EditorPythonBindings/Editor/Scripts/level_load_unload_stress.py

@@ -0,0 +1,116 @@
+#
+# Copyright (c) Contributors to the Open 3D Engine Project.
+# For complete copyright and license terms please see the LICENSE at the root of this distribution.
+#
+# SPDX-License-Identifier: Apache-2.0 OR MIT
+#
+#
+import os
+
+import azlmbr.legacy.general as azgeneral
+
+def RunLevelReloadTest(levels: list[str],
+                       iterationsCount: int = 3,
+                       iterationsEnterExit: int = 1,
+                       idleFramesOpen: int = 3,
+                       idleFramesEnter: int = 3,
+                       idleFramesExit: int = 3):
+    """
+    For each level name listed in @levels:
+    1- Loads the level
+    2- Waits (if requested)
+    3- Enters/Exit Game mode as many times defined in @iterationsEnterExit
+    
+    Repeats all of the above @iterationsCount times.
+    """
+    for idx in range(iterationsCount):
+        print(f"Iteration {idx}")
+        for levelName in levels:
+            azgeneral.open_level_no_prompt(levelName)
+            if idleFramesOpen > 0:
+                azgeneral.idle_wait_frames(idleFramesOpen)
+            for jdx in range(iterationsEnterExit):
+                print(f"Iteration {idx}. Enter/Exit {jdx}")
+                azgeneral.enter_game_mode()
+                if idleFramesEnter > 0:
+                    azgeneral.idle_wait_frames(idleFramesEnter)
+                azgeneral.exit_game_mode()
+                if idleFramesExit > 0:
+                    azgeneral.idle_wait_frames(idleFramesExit)
+
+
+# Quick Example on how to run this test using default levels from AutomatedTesting for 10 iterations:
+# pyRunFile C:\GIT\o3de\Gems\EditorPythonBindings\Editor\Scripts\level_load_unload_stress.py -i 10
+# pyRunFile C:\GIT\o3de\Gems\EditorPythonBindings\Editor\Scripts\level_load_unload_stress.py -i 10 --level CommsCenter,CypunkAptInterior,PoliceStation,TempleOfEnlightment
+# pyRunFile C:\GIT\o3de\Gems\EditorPythonBindings\Editor\Scripts\level_load_unload_stress.py -n 1 -x 1 --level CommsCenter,CypunkAptInterior,PoliceStation,TempleOfEnlightment
+def MainFunc():
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        description="Level loading test that validates that there are no crashes due to changing levels quickly."
+    )
+
+    parser.add_argument(
+        "-i",
+        "--iterations",
+        type=int,
+        default=3,
+        help="How many times will load all levels in the list",
+    )
+
+    parser.add_argument(
+        "-e",
+        "--enter_exit_iterations",
+        type=int,
+        default=2,
+        help="How many times will enter and exit game mode, for each level in the list.",
+    )
+
+    parser.add_argument(
+        "-o",
+        "--idle_frames_open",
+        type=int,
+        default=3,
+        help="How many frames to wait after opening a level",
+    )
+
+    parser.add_argument(
+        "-n",
+        "--idle_frames_enter",
+        type=int,
+        default=3,
+        help="How many frames to wait after entering game mode.",
+    )
+
+    parser.add_argument(
+        "-x",
+        "--idle_frames_exit",
+        type=int,
+        default=3,
+        help="How many frames to wait after exiting game mode.",
+    )
+
+    parser.add_argument(
+        "--levels",
+        default="Graphics/hermanubis,Graphics/macbeth_shaderballs",
+        help="Comma seperated list of level names.",
+    )
+
+    args = parser.parse_args()
+    iterations = args.iterations
+    iterationsEnterExit = args.enter_exit_iterations
+    idleFramesOpen = args.idle_frames_open
+    idleFramesEnter = args.idle_frames_enter
+    idleFramesExit = args.idle_frames_exit
+    levels = args.levels.split(",")
+    print(f"Original levels list:\n{levels}")
+    nonEmptyNames = []
+    for levelName in levels:
+        if levelName:
+            nonEmptyNames.append(levelName)
+    print(f"Valid levels list:\n{nonEmptyNames}")
+    RunLevelReloadTest(nonEmptyNames, iterations, iterationsEnterExit, idleFramesOpen, idleFramesEnter, idleFramesExit)
+    print(f"PASSED. Completed {iterations} iterations!")
+
+if __name__ == "__main__":
+    MainFunc()

Gems/Terrain/Code/Source/TerrainRenderer/Components/TerrainSurfaceMaterialsListComponent.cpp

@@ -327,6 +327,30 @@ namespace Terrain
     }
 
     void TerrainSurfaceMaterialsListComponent::OnAssetReady(AZ::Data::Asset<AZ::Data::AssetData> asset)
+    {
+        // REMARK: This function is typically invoked within the context of one of the AssetBus::OnAssetXXX functions,
+        // and a deadlock may occur according to the following sequence:
+        // 1. Starting from Main thread, AssetBus locks a mutex.
+        // 2. AssetBus calls OnAssetReady and it enters in this function.
+        // 3. Start the instantiation of a new StreamingImage.
+        // 4. StreamingImage asynchronously queues work in the "Seconday Copy Queue".
+        // 5. StreamingImage waits until the work completes.
+        // 6. The thread of "Seconday Copy Queue" gets a new work item, which may hold a reference
+        //    to an old StreamingImage.
+        // 7. The old StreamingImage gets destroyed and it calls AssetBus::MultiHandler::BusDisconnect(GetAssetId());
+        // 8. When calling AssetBus::MultiHandler::BusDisconnect(GetAssetId()); it tries to lock the same mutex
+        //    from step 1. But the mutex is already locked on Main Thread in step 1.
+        // 9. The "Seconday Copy Queue" thread deadlocks and never completes the work.
+        // 10. Main thread is also deadlocked waiting for "Seconday Copy Queue" to complete.
+        // The solution is to enqueue texture update on the next tick.
+        auto postTickLambda = [=]()
+        {
+            OnAssetReadyPostTick(asset);
+        };
+        AZ::TickBus::QueueFunction(AZStd::move(postTickLambda));
+    }
+
+    void TerrainSurfaceMaterialsListComponent::OnAssetReadyPostTick(AZ::Data::Asset<AZ::Data::AssetData> asset)
     {
         // Find the missing material instance with the correct id.
         auto handleCreateMaterial = [&](TerrainSurfaceMaterialMapping& mapping, const AZ::Data::Asset<AZ::Data::AssetData>& asset)
@@ -363,6 +387,16 @@ namespace Terrain
     }
 
     void TerrainSurfaceMaterialsListComponent::OnAssetReloaded(AZ::Data::Asset<AZ::Data::AssetData> asset)
+    {
+        // REMARK: See OnAssetReady for details on why we postpone the work on the next tick.
+        auto postTickLambda = [=]()
+        {
+            OnAssetReloadedPostTick(asset);
+        };
+        AZ::TickBus::QueueFunction(AZStd::move(postTickLambda));
+    }
+
+    void TerrainSurfaceMaterialsListComponent::OnAssetReloadedPostTick(AZ::Data::Asset<AZ::Data::AssetData> asset)
     {
         // Find the material instance with the correct id.
         auto handleUpdateMaterial = [&](TerrainSurfaceMaterialMapping& mapping, const AZ::Data::Asset<AZ::Data::AssetData>& asset)

Gems/Terrain/Code/Source/TerrainRenderer/Components/TerrainSurfaceMaterialsListComponent.h

@@ -106,6 +106,9 @@ namespace Terrain
         void OnAssetReady(AZ::Data::Asset<AZ::Data::AssetData> asset) override;
         void OnAssetReloaded(AZ::Data::Asset<AZ::Data::AssetData> asset) override;
 
+        void OnAssetReadyPostTick(AZ::Data::Asset<AZ::Data::AssetData> asset);
+        void OnAssetReloadedPostTick(AZ::Data::Asset<AZ::Data::AssetData> asset);
+
         TerrainSurfaceMaterialsListConfig m_configuration;
 
         AZ::Aabb m_cachedAabb{ AZ::Aabb::CreateNull() };