atomic.odin 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453
  1. package sync
  2. import "base:intrinsics"
  3. /*
  4. This procedure may lower CPU consumption or yield to a hyperthreaded twin
  5. processor. It's exact function is architecture specific, but the intent is to
  6. say that you're not doing much on a CPU.
  7. */
  8. cpu_relax :: intrinsics.cpu_relax
  9. /*
  10. Describes memory ordering for an atomic operation.
  11. Modern CPU's contain multiple cores and caches specific to those cores. When a
  12. core performs a write to memory, the value is written to cache first. The issue
  13. is that a core doesn't typically see what's inside the caches of other cores.
  14. In order to make operations consistent CPU's implement mechanisms that
  15. synchronize memory operations across cores by asking other cores or by
  16. pushing data about writes to other cores.
  17. Due to how these algorithms are implemented, the stores and loads performed by
  18. one core may seem to happen in a different order to another core. It also may
  19. happen that a core reorders stores and loads (independent of how compiler put
  20. them into the machine code). This can cause issues when trying to synchronize
  21. multiple memory locations between two cores. Which is why CPU's allow for
  22. stronger memory ordering guarantees if certain instructions or instruction
  23. variants are used.
  24. In Odin there are 5 different memory ordering guaranties that can be provided
  25. to an atomic operation:
  26. - `Relaxed`: The memory access (load or store) is unordered with respect to
  27. other memory accesses. This can be used to implement an atomic counter.
  28. Multiple threads access a single variable, but it doesn't matter when
  29. exactly it gets incremented, because it will become eventually consistent.
  30. - `Consume`: No loads or stores dependent on a memory location can be
  31. reordered before a load with consume memory order. If other threads released
  32. the same memory, it becomes visible.
  33. - `Acquire`: No loads or stores on a memory location can be reordered before a
  34. load of that memory location with acquire memory ordering. If other threads
  35. release the same memory, it becomes visible.
  36. - `Release`: No loads or stores on a memory location can be reordered after a
  37. store of that memory location with release memory ordering. All threads that
  38. acquire the same memory location will see all writes done by the current
  39. thread.
  40. - `Acq_Rel`: Acquire-release memory ordering: combines acquire and release
  41. memory orderings in the same operation.
  42. - `Seq_Cst`: Sequential consistency. The strongest memory ordering. A load will
  43. always be an acquire operation, a store will always be a release operation,
  44. and in addition to that all threads observe the same order of writes.
  45. Non-explicit atomics will always be sequentially consistent.
  46. Atomic_Memory_Order :: enum {
  47. Relaxed = 0, // Unordered
  48. Consume = 1, // Monotonic
  49. Acquire = 2,
  50. Release = 3,
  51. Acq_Rel = 4,
  52. Seq_Cst = 5,
  53. }
  54. **Note(i386, x64)**: x86 has a very strong memory model by default. It
  55. guarantees that all writes are ordered, stores and loads aren't reordered. In
  56. a sense, all operations are at least acquire and release operations. If `lock`
  57. prefix is used, all operations are sequentially consistent. If you use explicit
  58. atomics, make sure you have the correct atomic memory order, because bugs likely
  59. will not show up in x86, but may show up on e.g. arm. More on x86 memory
  60. ordering can be found
  61. [[here; https://www.cs.cmu.edu/~410-f10/doc/Intel_Reordering_318147.pdf]]
  62. */
  63. Atomic_Memory_Order :: intrinsics.Atomic_Memory_Order
  64. /*
  65. Establish memory ordering.
  66. This procedure establishes memory ordering, without an associated atomic
  67. operation.
  68. */
  69. atomic_thread_fence :: intrinsics.atomic_thread_fence
  70. /*
  71. Establish memory ordering between a current thread and a signal handler.
  72. This procedure establishes memory ordering between a thread and a signal
  73. handler, that run on the same thread, without an associated atomic operation.
  74. This procedure is equivalent to `atomic_thread_fence`, except it doesn't
  75. issue any CPU instructions for memory ordering.
  76. */
  77. atomic_signal_fence :: intrinsics.atomic_signal_fence
  78. /*
  79. Atomically store a value into memory.
  80. This procedure stores a value to a memory location in such a way that no other
  81. thread is able to see partial reads. This operation is sequentially-consistent.
  82. */
  83. atomic_store :: intrinsics.atomic_store
  84. /*
  85. Atomically store a value into memory with explicit memory ordering.
  86. This procedure stores a value to a memory location in such a way that no other
  87. thread is able to see partial reads. The memory ordering of this operation is
  88. as specified by the `order` parameter.
  89. */
  90. atomic_store_explicit :: intrinsics.atomic_store_explicit
  91. /*
  92. Atomically load a value from memory.
  93. This procedure loads a value from a memory location in such a way that the
  94. received value is not a partial read. The memory ordering of this operation is
  95. sequentially-consistent.
  96. */
  97. atomic_load :: intrinsics.atomic_load
  98. /*
  99. Atomically load a value from memory with explicit memory ordering.
  100. This procedure loads a value from a memory location in such a way that the
  101. received value is not a partial read. The memory ordering of this operation
  102. is as specified by the `order` parameter.
  103. */
  104. atomic_load_explicit :: intrinsics.atomic_load_explicit
  105. /*
  106. Atomically add a value to the value stored in memory.
  107. This procedure loads a value from memory, adds the specified value to it, and
  108. stores it back as an atomic operation. This operation is an atomic equivalent
  109. of the following:
  110. dst^ += val
  111. The memory ordering of this operation is sequentially-consistent.
  112. */
  113. atomic_add :: intrinsics.atomic_add
  114. /*
  115. Atomically add a value to the value stored in memory.
  116. This procedure loads a value from memory, adds the specified value to it, and
  117. stores it back as an atomic operation. This operation is an atomic equivalent
  118. of the following:
  119. dst^ += val
  120. The memory ordering of this operation is as specified by the `order` parameter.
  121. */
  122. atomic_add_explicit :: intrinsics.atomic_add_explicit
  123. /*
  124. Atomically subtract a value from the value stored in memory.
  125. This procedure loads a value from memory, subtracts the specified value from it,
  126. and stores the result back as an atomic operation. This operation is an atomic
  127. equivalent of the following:
  128. dst^ -= val
  129. The memory ordering of this operation is sequentially-consistent.
  130. */
  131. atomic_sub :: intrinsics.atomic_sub
  132. /*
  133. Atomically subtract a value from the value stored in memory.
  134. This procedure loads a value from memory, subtracts the specified value from it,
  135. and stores the result back as an atomic operation. This operation is an atomic
  136. equivalent of the following:
  137. dst^ -= val
  138. The memory ordering of this operation is as specified by the `order` parameter.
  139. */
  140. atomic_sub_explicit :: intrinsics.atomic_sub_explicit
  141. /*
  142. Atomically replace the memory location with the result of AND operation with
  143. the specified value.
  144. This procedure loads a value from memory, calculates the result of AND operation
  145. between the loaded value and the specified value, and stores it back into the
  146. same memory location as an atomic operation. This operation is an atomic
  147. equivalent of the following:
  148. dst^ &= val
  149. The memory ordering of this operation is sequentially-consistent.
  150. */
  151. atomic_and :: intrinsics.atomic_and
  152. /*
  153. Atomically replace the memory location with the result of AND operation with
  154. the specified value.
  155. This procedure loads a value from memory, calculates the result of AND operation
  156. between the loaded value and the specified value, and stores it back into the
  157. same memory location as an atomic operation. This operation is an atomic
  158. equivalent of the following:
  159. dst^ &= val
  160. The memory ordering of this operation is as specified by the `order` parameter.
  161. */
  162. atomic_and_explicit :: intrinsics.atomic_and_explicit
  163. /*
  164. Atomically replace the memory location with the result of NAND operation with
  165. the specified value.
  166. This procedure loads a value from memory, calculates the result of NAND operation
  167. between the loaded value and the specified value, and stores it back into the
  168. same memory location as an atomic operation. This operation is an atomic
  169. equivalent of the following:
  170. dst^ = ~(dst^ & val)
  171. The memory ordering of this operation is sequentially-consistent.
  172. */
  173. atomic_nand :: intrinsics.atomic_nand
  174. /*
  175. Atomically replace the memory location with the result of NAND operation with
  176. the specified value.
  177. This procedure loads a value from memory, calculates the result of NAND operation
  178. between the loaded value and the specified value, and stores it back into the
  179. same memory location as an atomic operation. This operation is an atomic
  180. equivalent of the following:
  181. dst^ = ~(dst^ & val)
  182. The memory ordering of this operation is as specified by the `order` parameter.
  183. */
  184. atomic_nand_explicit :: intrinsics.atomic_nand_explicit
  185. /*
  186. Atomically replace the memory location with the result of OR operation with
  187. the specified value.
  188. This procedure loads a value from memory, calculates the result of OR operation
  189. between the loaded value and the specified value, and stores it back into the
  190. same memory location as an atomic operation. This operation is an atomic
  191. equivalent of the following:
  192. dst^ |= val
  193. The memory ordering of this operation is sequentially-consistent.
  194. */
  195. atomic_or :: intrinsics.atomic_or
  196. /*
  197. Atomically replace the memory location with the result of OR operation with
  198. the specified value.
  199. This procedure loads a value from memory, calculates the result of OR operation
  200. between the loaded value and the specified value, and stores it back into the
  201. same memory location as an atomic operation. This operation is an atomic
  202. equivalent of the following:
  203. dst^ |= val
  204. The memory ordering of this operation is as specified by the `order` parameter.
  205. */
  206. atomic_or_explicit :: intrinsics.atomic_or_explicit
  207. /*
  208. Atomically replace the memory location with the result of XOR operation with
  209. the specified value.
  210. This procedure loads a value from memory, calculates the result of XOR operation
  211. between the loaded value and the specified value, and stores it back into the
  212. same memory location as an atomic operation. This operation is an atomic
  213. equivalent of the following:
  214. dst^ ~= val
  215. The memory ordering of this operation is sequentially-consistent.
  216. */
  217. atomic_xor :: intrinsics.atomic_xor
  218. /*
  219. Atomically replace the memory location with the result of XOR operation with
  220. the specified value.
  221. This procedure loads a value from memory, calculates the result of XOR operation
  222. between the loaded value and the specified value, and stores it back into the
  223. same memory location as an atomic operation. This operation is an atomic
  224. equivalent of the following:
  225. dst^ ~= val
  226. The memory ordering of this operation is as specified by the `order` parameter.
  227. */
  228. atomic_xor_explicit :: intrinsics.atomic_xor_explicit
  229. /*
  230. Atomically exchange the value in a memory location, with the specified value.
  231. This procedure loads a value from the specified memory location, and stores the
  232. specified value into that memory location. Then the loaded value is returned,
  233. all done in a single atomic operation. This operation is an atomic equivalent
  234. of the following:
  235. tmp := dst^
  236. dst^ = val
  237. return tmp
  238. The memory ordering of this operation is sequentially-consistent.
  239. */
  240. atomic_exchange :: intrinsics.atomic_exchange
  241. /*
  242. Atomically exchange the value in a memory location, with the specified value.
  243. This procedure loads a value from the specified memory location, and stores the
  244. specified value into that memory location. Then the loaded value is returned,
  245. all done in a single atomic operation. This operation is an atomic equivalent
  246. of the following:
  247. tmp := dst^
  248. dst^ = val
  249. return tmp
  250. The memory ordering of this operation is as specified by the `order` parameter.
  251. */
  252. atomic_exchange_explicit :: intrinsics.atomic_exchange_explicit
  253. /*
  254. Atomically compare and exchange the value with a memory location.
  255. This procedure checks if the value pointed to by the `dst` parameter is equal
  256. to `old`, and if they are, it stores the value `new` into the memory location,
  257. all done in a single atomic operation. This procedure returns the old value
  258. stored in a memory location and a boolean value signifying whether `old` was
  259. equal to `new`.
  260. This procedure is an atomic equivalent of the following operation:
  261. old_dst := dst^
  262. if old_dst == old {
  263. dst^ = new
  264. return old_dst, true
  265. } else {
  266. return old_dst, false
  267. }
  268. The strong version of compare exchange always returns true, when the returned
  269. old value stored in location pointed to by `dst` and the `old` parameter are
  270. equal.
  271. Atomic compare exchange has two memory orderings: One is for the
  272. read-modify-write operation, if the comparison succeeds, and the other is for
  273. the load operation, if the comparison fails. The memory ordering for both of
  274. of these operations is sequentially-consistent.
  275. */
  276. atomic_compare_exchange_strong :: intrinsics.atomic_compare_exchange_strong
  277. /*
  278. Atomically compare and exchange the value with a memory location.
  279. This procedure checks if the value pointed to by the `dst` parameter is equal
  280. to `old`, and if they are, it stores the value `new` into the memory location,
  281. all done in a single atomic operation. This procedure returns the old value
  282. stored in a memory location and a boolean value signifying whether `old` was
  283. equal to `new`.
  284. This procedure is an atomic equivalent of the following operation:
  285. old_dst := dst^
  286. if old_dst == old {
  287. dst^ = new
  288. return old_dst, true
  289. } else {
  290. return old_dst, false
  291. }
  292. The strong version of compare exchange always returns true, when the returned
  293. old value stored in location pointed to by `dst` and the `old` parameter are
  294. equal.
  295. Atomic compare exchange has two memory orderings: One is for the
  296. read-modify-write operation, if the comparison succeeds, and the other is for
  297. the load operation, if the comparison fails. The memory ordering for these
  298. operations is as specified by `success` and `failure` parameters respectively.
  299. */
  300. atomic_compare_exchange_strong_explicit :: intrinsics.atomic_compare_exchange_strong_explicit
  301. /*
  302. Atomically compare and exchange the value with a memory location.
  303. This procedure checks if the value pointed to by the `dst` parameter is equal
  304. to `old`, and if they are, it stores the value `new` into the memory location,
  305. all done in a single atomic operation. This procedure returns the old value
  306. stored in a memory location and a boolean value signifying whether `old` was
  307. equal to `new`.
  308. This procedure is an atomic equivalent of the following operation:
  309. old_dst := dst^
  310. if old_dst == old {
  311. // may return false here
  312. dst^ = new
  313. return old_dst, true
  314. } else {
  315. return old_dst, false
  316. }
  317. The weak version of compare exchange may return false, even if `dst^ == old`.
  318. On some platforms running weak compare exchange in a loop is faster than a
  319. strong version.
  320. Atomic compare exchange has two memory orderings: One is for the
  321. read-modify-write operation, if the comparison succeeds, and the other is for
  322. the load operation, if the comparison fails. The memory ordering for both
  323. of these operations is sequentially-consistent.
  324. */
  325. atomic_compare_exchange_weak :: intrinsics.atomic_compare_exchange_weak
  326. /*
  327. Atomically compare and exchange the value with a memory location.
  328. This procedure checks if the value pointed to by the `dst` parameter is equal
  329. to `old`, and if they are, it stores the value `new` into the memory location,
  330. all done in a single atomic operation. This procedure returns the old value
  331. stored in a memory location and a boolean value signifying whether `old` was
  332. equal to `new`.
  333. This procedure is an atomic equivalent of the following operation:
  334. old_dst := dst^
  335. if old_dst == old {
  336. // may return false here
  337. dst^ = new
  338. return old_dst, true
  339. } else {
  340. return old_dst, false
  341. }
  342. The weak version of compare exchange may return false, even if `dst^ == old`.
  343. On some platforms running weak compare exchange in a loop is faster than a
  344. strong version.
  345. Atomic compare exchange has two memory orderings: One is for the
  346. read-modify-write operation, if the comparison succeeds, and the other is for
  347. the load operation, if the comparison fails. The memory ordering for these
  348. operations is as specified by the `success` and `failure` parameters
  349. respectively.
  350. */
  351. atomic_compare_exchange_weak_explicit :: intrinsics.atomic_compare_exchange_weak_explicit