ext_buffer.html 23 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689
  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4. <title>String Buffer Library</title>
  5. <meta charset="utf-8">
  6. <meta name="Copyright" content="Copyright (C) 2005-2023">
  7. <meta name="Language" content="en">
  8. <link rel="stylesheet" type="text/css" href="bluequad.css" media="screen">
  9. <link rel="stylesheet" type="text/css" href="bluequad-print.css" media="print">
  10. <style type="text/css">
  11. .lib {
  12. vertical-align: middle;
  13. margin-left: 5px;
  14. padding: 0 5px;
  15. font-size: 60%;
  16. border-radius: 5px;
  17. background: #c5d5ff;
  18. color: #000;
  19. }
  20. </style>
  21. </head>
  22. <body>
  23. <div id="site">
  24. <a href="https://luajit.org"><span>Lua<span id="logo">JIT</span></span></a>
  25. </div>
  26. <div id="head">
  27. <h1>String Buffer Library</h1>
  28. </div>
  29. <div id="nav">
  30. <ul><li>
  31. <a href="luajit.html">LuaJIT</a>
  32. <ul><li>
  33. <a href="https://luajit.org/download.html">Download <span class="ext">&raquo;</span></a>
  34. </li><li>
  35. <a href="install.html">Installation</a>
  36. </li><li>
  37. <a href="running.html">Running</a>
  38. </li></ul>
  39. </li><li>
  40. <a href="extensions.html">Extensions</a>
  41. <ul><li>
  42. <a href="ext_ffi.html">FFI Library</a>
  43. <ul><li>
  44. <a href="ext_ffi_tutorial.html">FFI Tutorial</a>
  45. </li><li>
  46. <a href="ext_ffi_api.html">ffi.* API</a>
  47. </li><li>
  48. <a href="ext_ffi_semantics.html">FFI Semantics</a>
  49. </li></ul>
  50. </li><li>
  51. <a class="current" href="ext_buffer.html">String Buffers</a>
  52. </li><li>
  53. <a href="ext_jit.html">jit.* Library</a>
  54. </li><li>
  55. <a href="ext_c_api.html">Lua/C API</a>
  56. </li><li>
  57. <a href="ext_profiler.html">Profiler</a>
  58. </li></ul>
  59. </li><li>
  60. <a href="https://luajit.org/status.html">Status <span class="ext">&raquo;</span></a>
  61. </li><li>
  62. <a href="https://luajit.org/faq.html">FAQ <span class="ext">&raquo;</span></a>
  63. </li><li>
  64. <a href="https://luajit.org/list.html">Mailing List <span class="ext">&raquo;</span></a>
  65. </li></ul>
  66. </div>
  67. <div id="main">
  68. <p>
  69. The string buffer library allows <b>high-performance manipulation of
  70. string-like data</b>.
  71. </p>
  72. <p>
  73. Unlike Lua strings, which are constants, string buffers are
  74. <b>mutable</b> sequences of 8-bit (binary-transparent) characters. Data
  75. can be stored, formatted and encoded into a string buffer and later
  76. converted, extracted or decoded.
  77. </p>
  78. <p>
  79. The convenient string buffer API simplifies common string manipulation
  80. tasks, that would otherwise require creating many intermediate strings.
  81. String buffers improve performance by eliminating redundant memory
  82. copies, object creation, string interning and garbage collection
  83. overhead. In conjunction with the FFI library, they allow zero-copy
  84. operations.
  85. </p>
  86. <p>
  87. The string buffer library also includes a high-performance
  88. <a href="serialize">serializer</a> for Lua objects.
  89. </p>
  90. <h2 id="use">Using the String Buffer Library</h2>
  91. <p>
  92. The string buffer library is built into LuaJIT by default, but it's not
  93. loaded by default. Add this to the start of every Lua file that needs
  94. one of its functions:
  95. </p>
  96. <pre class="code">
  97. local buffer = require("string.buffer")
  98. </pre>
  99. <p>
  100. The convention for the syntax shown on this page is that <tt>buffer</tt>
  101. refers to the buffer library and <tt>buf</tt> refers to an individual
  102. buffer object.
  103. </p>
  104. <p>
  105. Please note the difference between a Lua function call, e.g.
  106. <tt>buffer.new()</tt> (with a dot) and a Lua method call, e.g.
  107. <tt>buf:reset()</tt> (with a colon).
  108. </p>
  109. <h3 id="buffer_object">Buffer Objects</h3>
  110. <p>
  111. A buffer object is a garbage-collected Lua object. After creation with
  112. <tt>buffer.new()</tt>, it can (and should) be reused for many operations.
  113. When the last reference to a buffer object is gone, it will eventually
  114. be freed by the garbage collector, along with the allocated buffer
  115. space.
  116. </p>
  117. <p>
  118. Buffers operate like a FIFO (first-in first-out) data structure. Data
  119. can be appended (written) to the end of the buffer and consumed (read)
  120. from the front of the buffer. These operations may be freely mixed.
  121. </p>
  122. <p>
  123. The buffer space that holds the characters is managed automatically
  124. &mdash; it grows as needed and already consumed space is recycled. Use
  125. <tt>buffer.new(size)</tt> and <tt>buf:free()</tt>, if you need more
  126. control.
  127. </p>
  128. <p>
  129. The maximum size of a single buffer is the same as the maximum size of a
  130. Lua string, which is slightly below two gigabytes. For huge data sizes,
  131. neither strings nor buffers are the right data structure &mdash; use the
  132. FFI library to directly map memory or files up to the virtual memory
  133. limit of your OS.
  134. </p>
  135. <h3 id="buffer_overview">Buffer Method Overview</h3>
  136. <ul>
  137. <li>
  138. The <tt>buf:put*()</tt>-like methods append (write) characters to the
  139. end of the buffer.
  140. </li>
  141. <li>
  142. The <tt>buf:get*()</tt>-like methods consume (read) characters from the
  143. front of the buffer.
  144. </li>
  145. <li>
  146. Other methods, like <tt>buf:tostring()</tt> only read the buffer
  147. contents, but don't change the buffer.
  148. </li>
  149. <li>
  150. The <tt>buf:set()</tt> method allows zero-copy consumption of a string
  151. or an FFI cdata object as a buffer.
  152. </li>
  153. <li>
  154. The FFI-specific methods allow zero-copy read/write-style operations or
  155. modifying the buffer contents in-place. Please check the
  156. <a href="#ffi_caveats">FFI caveats</a> below, too.
  157. </li>
  158. <li>
  159. Methods that don't need to return anything specific, return the buffer
  160. object itself as a convenience. This allows method chaining, e.g.:
  161. <tt>buf:reset():encode(obj)</tt> or <tt>buf:skip(len):get()</tt>
  162. </li>
  163. </ul>
  164. <h2 id="create">Buffer Creation and Management</h2>
  165. <h3 id="buffer_new"><tt>local buf = buffer.new([size [,options]])<br>
  166. local buf = buffer.new([options])</tt></h3>
  167. <p>
  168. Creates a new buffer object.
  169. </p>
  170. <p>
  171. The optional <tt>size</tt> argument ensures a minimum initial buffer
  172. size. This is strictly an optimization when the required buffer size is
  173. known beforehand. The buffer space will grow as needed, in any case.
  174. </p>
  175. <p>
  176. The optional table <tt>options</tt> sets various
  177. <a href="#serialize_options">serialization options</a>.
  178. </p>
  179. <h3 id="buffer_reset"><tt>buf = buf:reset()</tt></h3>
  180. <p>
  181. Reset (empty) the buffer. The allocated buffer space is not freed and
  182. may be reused.
  183. </p>
  184. <h3 id="buffer_free"><tt>buf = buf:free()</tt></h3>
  185. <p>
  186. The buffer space of the buffer object is freed. The object itself
  187. remains intact, empty and may be reused.
  188. </p>
  189. <p>
  190. Note: you normally don't need to use this method. The garbage collector
  191. automatically frees the buffer space, when the buffer object is
  192. collected. Use this method, if you need to free the associated memory
  193. immediately.
  194. </p>
  195. <h2 id="write">Buffer Writers</h2>
  196. <h3 id="buffer_put"><tt>buf = buf:put([str|num|obj] [,…])</tt></h3>
  197. <p>
  198. Appends a string <tt>str</tt>, a number <tt>num</tt> or any object
  199. <tt>obj</tt> with a <tt>__tostring</tt> metamethod to the buffer.
  200. Multiple arguments are appended in the given order.
  201. </p>
  202. <p>
  203. Appending a buffer to a buffer is possible and short-circuited
  204. internally. But it still involves a copy. Better combine the buffer
  205. writes to use a single buffer.
  206. </p>
  207. <h3 id="buffer_putf"><tt>buf = buf:putf(format, …)</tt></h3>
  208. <p>
  209. Appends the formatted arguments to the buffer. The <tt>format</tt>
  210. string supports the same options as <tt>string.format()</tt>.
  211. </p>
  212. <h3 id="buffer_putcdata"><tt>buf = buf:putcdata(cdata, len)</tt><span class="lib">FFI</span></h3>
  213. <p>
  214. Appends the given <tt>len</tt> number of bytes from the memory pointed
  215. to by the FFI <tt>cdata</tt> object to the buffer. The object needs to
  216. be convertible to a (constant) pointer.
  217. </p>
  218. <h3 id="buffer_set"><tt>buf = buf:set(str)<br>
  219. buf = buf:set(cdata, len)</tt><span class="lib">FFI</span></h3>
  220. <p>
  221. This method allows zero-copy consumption of a string or an FFI cdata
  222. object as a buffer. It stores a reference to the passed string
  223. <tt>str</tt> or the FFI <tt>cdata</tt> object in the buffer. Any buffer
  224. space originally allocated is freed. This is <i>not</i> an append
  225. operation, unlike the <tt>buf:put*()</tt> methods.
  226. </p>
  227. <p>
  228. After calling this method, the buffer behaves as if
  229. <tt>buf:free():put(str)</tt> or <tt>buf:free():put(cdata,&nbsp;len)</tt>
  230. had been called. However, the data is only referenced and not copied, as
  231. long as the buffer is only consumed.
  232. </p>
  233. <p>
  234. In case the buffer is written to later on, the referenced data is copied
  235. and the object reference is removed (copy-on-write semantics).
  236. </p>
  237. <p>
  238. The stored reference is an anchor for the garbage collector and keeps the
  239. originally passed string or FFI cdata object alive.
  240. </p>
  241. <h3 id="buffer_reserve"><tt>ptr, len = buf:reserve(size)</tt><span class="lib">FFI</span><br>
  242. <tt>buf = buf:commit(used)</tt><span class="lib">FFI</span></h3>
  243. <p>
  244. The <tt>reserve</tt> method reserves at least <tt>size</tt> bytes of
  245. write space in the buffer. It returns an <tt>uint8_t&nbsp;*</tt> FFI
  246. cdata pointer <tt>ptr</tt> that points to this space.
  247. </p>
  248. <p>
  249. The available length in bytes is returned in <tt>len</tt>. This is at
  250. least <tt>size</tt> bytes, but may be more to facilitate efficient
  251. buffer growth. You can either make use of the additional space or ignore
  252. <tt>len</tt> and only use <tt>size</tt> bytes.
  253. </p>
  254. <p>
  255. The <tt>commit</tt> method appends the <tt>used</tt> bytes of the
  256. previously returned write space to the buffer data.
  257. </p>
  258. <p>
  259. This pair of methods allows zero-copy use of C read-style APIs:
  260. </p>
  261. <pre class="code">
  262. local MIN_SIZE = 65536
  263. repeat
  264. local ptr, len = buf:reserve(MIN_SIZE)
  265. local n = C.read(fd, ptr, len)
  266. if n == 0 then break end -- EOF.
  267. if n &lt; 0 then error("read error") end
  268. buf:commit(n)
  269. until false
  270. </pre>
  271. <p>
  272. The reserved write space is <i>not</i> initialized. At least the
  273. <tt>used</tt> bytes <b>must</b> be written to before calling the
  274. <tt>commit</tt> method. There's no need to call the <tt>commit</tt>
  275. method, if nothing is added to the buffer (e.g. on error).
  276. </p>
  277. <h2 id="read">Buffer Readers</h2>
  278. <h3 id="buffer_length"><tt>len = #buf</tt></h3>
  279. <p>
  280. Returns the current length of the buffer data in bytes.
  281. </p>
  282. <h3 id="buffer_concat"><tt>res = str|num|buf .. str|num|buf […]</tt></h3>
  283. <p>
  284. The Lua concatenation operator <tt>..</tt> also accepts buffers, just
  285. like strings or numbers. It always returns a string and not a buffer.
  286. </p>
  287. <p>
  288. Note that although this is supported for convenience, this thwarts one
  289. of the main reasons to use buffers, which is to avoid string
  290. allocations. Rewrite it with <tt>buf:put()</tt> and <tt>buf:get()</tt>.
  291. </p>
  292. <p>
  293. Mixing this with unrelated objects that have a <tt>__concat</tt>
  294. metamethod may not work, since these probably only expect strings.
  295. </p>
  296. <h3 id="buffer_skip"><tt>buf = buf:skip(len)</tt></h3>
  297. <p>
  298. Skips (consumes) <tt>len</tt> bytes from the buffer up to the current
  299. length of the buffer data.
  300. </p>
  301. <h3 id="buffer_get"><tt>str, … = buf:get([len|nil] [,…])</tt></h3>
  302. <p>
  303. Consumes the buffer data and returns one or more strings. If called
  304. without arguments, the whole buffer data is consumed. If called with a
  305. number, up to <tt>len</tt> bytes are consumed. A <tt>nil</tt> argument
  306. consumes the remaining buffer space (this only makes sense as the last
  307. argument). Multiple arguments consume the buffer data in the given
  308. order.
  309. </p>
  310. <p>
  311. Note: a zero length or no remaining buffer data returns an empty string
  312. and not <tt>nil</tt>.
  313. </p>
  314. <h3 id="buffer_tostring"><tt>str = buf:tostring()<br>
  315. str = tostring(buf)</tt></h3>
  316. <p>
  317. Creates a string from the buffer data, but doesn't consume it. The
  318. buffer remains unchanged.
  319. </p>
  320. <p>
  321. Buffer objects also define a <tt>__tostring</tt> metamethod. This means
  322. buffers can be passed to the global <tt>tostring()</tt> function and
  323. many other functions that accept this in place of strings. The important
  324. internal uses in functions like <tt>io.write()</tt> are short-circuited
  325. to avoid the creation of an intermediate string object.
  326. </p>
  327. <h3 id="buffer_ref"><tt>ptr, len = buf:ref()</tt><span class="lib">FFI</span></h3>
  328. <p>
  329. Returns an <tt>uint8_t&nbsp;*</tt> FFI cdata pointer <tt>ptr</tt> that
  330. points to the buffer data. The length of the buffer data in bytes is
  331. returned in <tt>len</tt>.
  332. </p>
  333. <p>
  334. The returned pointer can be directly passed to C functions that expect a
  335. buffer and a length. You can also do bytewise reads
  336. (<tt>local&nbsp;x&nbsp;=&nbsp;ptr[i]</tt>) or writes
  337. (<tt>ptr[i]&nbsp;=&nbsp;0x40</tt>) of the buffer data.
  338. </p>
  339. <p>
  340. In conjunction with the <tt>skip</tt> method, this allows zero-copy use
  341. of C write-style APIs:
  342. </p>
  343. <pre class="code">
  344. repeat
  345. local ptr, len = buf:ref()
  346. if len == 0 then break end
  347. local n = C.write(fd, ptr, len)
  348. if n &lt; 0 then error("write error") end
  349. buf:skip(n)
  350. until n >= len
  351. </pre>
  352. <p>
  353. Unlike Lua strings, buffer data is <i>not</i> implicitly
  354. zero-terminated. It's not safe to pass <tt>ptr</tt> to C functions that
  355. expect zero-terminated strings. If you're not using <tt>len</tt>, then
  356. you're doing something wrong.
  357. </p>
  358. <h2 id="serialize">Serialization of Lua Objects</h2>
  359. <p>
  360. The following functions and methods allow <b>high-speed serialization</b>
  361. (encoding) of a Lua object into a string and decoding it back to a Lua
  362. object. This allows convenient storage and transport of <b>structured
  363. data</b>.
  364. </p>
  365. <p>
  366. The encoded data is in an <a href="#serialize_format">internal binary
  367. format</a>. The data can be stored in files, binary-transparent
  368. databases or transmitted to other LuaJIT instances across threads,
  369. processes or networks.
  370. </p>
  371. <p>
  372. Encoding speed can reach up to 1 Gigabyte/second on a modern desktop- or
  373. server-class system, even when serializing many small objects. Decoding
  374. speed is mostly constrained by object creation cost.
  375. </p>
  376. <p>
  377. The serializer handles most Lua types, common FFI number types and
  378. nested structures. Functions, thread objects, other FFI cdata and full
  379. userdata cannot be serialized (yet).
  380. </p>
  381. <p>
  382. The encoder serializes nested structures as trees. Multiple references
  383. to a single object will be stored separately and create distinct objects
  384. after decoding. Circular references cause an error.
  385. </p>
  386. <h3 id="serialize_methods">Serialization Functions and Methods</h3>
  387. <h3 id="buffer_encode"><tt>str = buffer.encode(obj)<br>
  388. buf = buf:encode(obj)</tt></h3>
  389. <p>
  390. Serializes (encodes) the Lua object <tt>obj</tt>. The stand-alone
  391. function returns a string <tt>str</tt>. The buffer method appends the
  392. encoding to the buffer.
  393. </p>
  394. <p>
  395. <tt>obj</tt> can be any of the supported Lua types &mdash; it doesn't
  396. need to be a Lua table.
  397. </p>
  398. <p>
  399. This function may throw an error when attempting to serialize
  400. unsupported object types, circular references or deeply nested tables.
  401. </p>
  402. <h3 id="buffer_decode"><tt>obj = buffer.decode(str)<br>
  403. obj = buf:decode()</tt></h3>
  404. <p>
  405. The stand-alone function deserializes (decodes) the string
  406. <tt>str</tt>, the buffer method deserializes one object from the
  407. buffer. Both return a Lua object <tt>obj</tt>.
  408. </p>
  409. <p>
  410. The returned object may be any of the supported Lua types &mdash;
  411. even <tt>nil</tt>.
  412. </p>
  413. <p>
  414. This function may throw an error when fed with malformed or incomplete
  415. encoded data. The stand-alone function throws when there's left-over
  416. data after decoding a single top-level object. The buffer method leaves
  417. any left-over data in the buffer.
  418. </p>
  419. <p>
  420. Attempting to deserialize an FFI type will throw an error, if the FFI
  421. library is not built-in or has not been loaded, yet.
  422. </p>
  423. <h3 id="serialize_options">Serialization Options</h3>
  424. <p>
  425. The <tt>options</tt> table passed to <tt>buffer.new()</tt> may contain
  426. the following members (all optional):
  427. </p>
  428. <ul>
  429. <li>
  430. <tt>dict</tt> is a Lua table holding a <b>dictionary of strings</b> that
  431. commonly occur as table keys of objects you are serializing. These keys
  432. are compactly encoded as indexes during serialization. A well-chosen
  433. dictionary saves space and improves serialization performance.
  434. </li>
  435. <li>
  436. <tt>metatable</tt> is a Lua table holding a <b>dictionary of metatables</b>
  437. for the table objects you are serializing.
  438. </li>
  439. </ul>
  440. <p>
  441. <tt>dict</tt> needs to be an array of strings and <tt>metatable</tt> needs
  442. to be an array of tables. Both starting at index 1 and without holes (no
  443. <tt>nil</tt> in between). The tables are anchored in the buffer object and
  444. internally modified into a two-way index (don't do this yourself, just pass
  445. a plain array). The tables must not be modified after they have been passed
  446. to <tt>buffer.new()</tt>.
  447. </p>
  448. <p>
  449. The <tt>dict</tt> and <tt>metatable</tt> tables used by the encoder and
  450. decoder must be the same. Put the most common entries at the front. Extend
  451. at the end to ensure backwards-compatibility &mdash; older encodings can
  452. then still be read. You may also set some indexes to <tt>false</tt> to
  453. explicitly drop backwards-compatibility. Old encodings that use these
  454. indexes will throw an error when decoded.
  455. </p>
  456. <p>
  457. Metatables that are not found in the <tt>metatable</tt> dictionary are
  458. ignored when encoding. Decoding returns a table with a <tt>nil</tt>
  459. metatable.
  460. </p>
  461. <p>
  462. Note: parsing and preparation of the options table is somewhat
  463. expensive. Create a buffer object only once and recycle it for multiple
  464. uses. Avoid mixing encoder and decoder buffers, since the
  465. <tt>buf:set()</tt> method frees the already allocated buffer space:
  466. </p>
  467. <pre class="code">
  468. local options = {
  469. dict = { "commonly", "used", "string", "keys" },
  470. }
  471. local buf_enc = buffer.new(options)
  472. local buf_dec = buffer.new(options)
  473. local function encode(obj)
  474. return buf_enc:reset():encode(obj):get()
  475. end
  476. local function decode(str)
  477. return buf_dec:set(str):decode()
  478. end
  479. </pre>
  480. <h3 id="serialize_stream">Streaming Serialization</h3>
  481. <p>
  482. In some contexts, it's desirable to do piecewise serialization of large
  483. datasets, also known as <i>streaming</i>.
  484. </p>
  485. <p>
  486. This serialization format can be safely concatenated and supports streaming.
  487. Multiple encodings can simply be appended to a buffer and later decoded
  488. individually:
  489. </p>
  490. <pre class="code">
  491. local buf = buffer.new()
  492. buf:encode(obj1)
  493. buf:encode(obj2)
  494. local copy1 = buf:decode()
  495. local copy2 = buf:decode()
  496. </pre>
  497. <p>
  498. Here's how to iterate over a stream:
  499. </p>
  500. <pre class="code">
  501. while #buf ~= 0 do
  502. local obj = buf:decode()
  503. -- Do something with obj.
  504. end
  505. </pre>
  506. <p>
  507. Since the serialization format doesn't prepend a length to its encoding,
  508. network applications may need to transmit the length, too.
  509. </p>
  510. <h3 id="serialize_format">Serialization Format Specification</h3>
  511. <p>
  512. This serialization format is designed for <b>internal use</b> by LuaJIT
  513. applications. Serialized data is upwards-compatible and portable across
  514. all supported LuaJIT platforms.
  515. </p>
  516. <p>
  517. It's an <b>8-bit binary format</b> and not human-readable. It uses e.g.
  518. embedded zeroes and stores embedded Lua string objects unmodified, which
  519. are 8-bit-clean, too. Encoded data can be safely concatenated for
  520. streaming and later decoded one top-level object at a time.
  521. </p>
  522. <p>
  523. The encoding is reasonably compact, but tuned for maximum performance,
  524. not for minimum space usage. It compresses well with any of the common
  525. byte-oriented data compression algorithms.
  526. </p>
  527. <p>
  528. Although documented here for reference, this format is explicitly
  529. <b>not</b> intended to be a 'public standard' for structured data
  530. interchange across computer languages (like JSON or MessagePack). Please
  531. do not use it as such.
  532. </p>
  533. <p>
  534. The specification is given below as a context-free grammar with a
  535. top-level <tt>object</tt> as the starting point. Alternatives are
  536. separated by the <tt>|</tt> symbol and <tt>*</tt> indicates repeats.
  537. Grouping is implicit or indicated by <tt>{…}</tt>. Terminals are
  538. either plain hex numbers, encoded as bytes, or have a <tt>.format</tt>
  539. suffix.
  540. </p>
  541. <pre>
  542. object → nil | false | true
  543. | null | lightud32 | lightud64
  544. | int | num | tab | tab_mt
  545. | int64 | uint64 | complex
  546. | string
  547. nil → 0x00
  548. false → 0x01
  549. true → 0x02
  550. null → 0x03 // NULL lightuserdata
  551. lightud32 → 0x04 data.I // 32 bit lightuserdata
  552. lightud64 → 0x05 data.L // 64 bit lightuserdata
  553. int → 0x06 int.I // int32_t
  554. num → 0x07 double.L
  555. tab → 0x08 // Empty table
  556. | 0x09 h.U h*{object object} // Key/value hash
  557. | 0x0a a.U a*object // 0-based array
  558. | 0x0b a.U a*object h.U h*{object object} // Mixed
  559. | 0x0c a.U (a-1)*object // 1-based array
  560. | 0x0d a.U (a-1)*object h.U h*{object object} // Mixed
  561. tab_mt → 0x0e (index-1).U tab // Metatable dict entry
  562. int64 → 0x10 int.L // FFI int64_t
  563. uint64 → 0x11 uint.L // FFI uint64_t
  564. complex → 0x12 re.L im.L // FFI complex
  565. string → (0x20+len).U len*char.B
  566. | 0x0f (index-1).U // String dict entry
  567. .B = 8 bit
  568. .I = 32 bit little-endian
  569. .L = 64 bit little-endian
  570. .U = prefix-encoded 32 bit unsigned number n:
  571. 0x00..0xdf → n.B
  572. 0xe0..0x1fdf → (0xe0|(((n-0xe0)>>8)&0x1f)).B ((n-0xe0)&0xff).B
  573. 0x1fe0.. → 0xff n.I
  574. </pre>
  575. <h2 id="error">Error handling</h2>
  576. <p>
  577. Many of the buffer methods can throw an error. Out-of-memory or usage
  578. errors are best caught with an outer wrapper for larger parts of code.
  579. There's not much one can do after that, anyway.
  580. </p>
  581. <p>
  582. OTOH, you may want to catch some errors individually. Buffer methods need
  583. to receive the buffer object as the first argument. The Lua colon-syntax
  584. <tt>obj:method()</tt> does that implicitly. But to wrap a method with
  585. <tt>pcall()</tt>, the arguments need to be passed like this:
  586. </p>
  587. <pre class="code">
  588. local ok, err = pcall(buf.encode, buf, obj)
  589. if not ok then
  590. -- Handle error in err.
  591. end
  592. </pre>
  593. <h2 id="ffi_caveats">FFI caveats</h2>
  594. <p>
  595. The string buffer library has been designed to work well together with
  596. the FFI library. But due to the low-level nature of the FFI library,
  597. some care needs to be taken:
  598. </p>
  599. <p>
  600. First, please remember that FFI pointers are zero-indexed. The space
  601. returned by <tt>buf:reserve()</tt> and <tt>buf:ref()</tt> starts at the
  602. returned pointer and ends before <tt>len</tt> bytes after that.
  603. </p>
  604. <p>
  605. I.e. the first valid index is <tt>ptr[0]</tt> and the last valid index
  606. is <tt>ptr[len-1]</tt>. If the returned length is zero, there's no valid
  607. index at all. The returned pointer may even be <tt>NULL</tt>.
  608. </p>
  609. <p>
  610. The space pointed to by the returned pointer is only valid as long as
  611. the buffer is not modified in any way (neither append, nor consume, nor
  612. reset, etc.). The pointer is also not a GC anchor for the buffer object
  613. itself.
  614. </p>
  615. <p>
  616. Buffer data is only guaranteed to be byte-aligned. Casting the returned
  617. pointer to a data type with higher alignment may cause unaligned
  618. accesses. It depends on the CPU architecture whether this is allowed or
  619. not (it's always OK on x86/x64 and mostly OK on other modern
  620. architectures).
  621. </p>
  622. <p>
  623. FFI pointers or references do not count as GC anchors for an underlying
  624. object. E.g. an <tt>array</tt> allocated with <tt>ffi.new()</tt> is
  625. anchored by <tt>buf:set(array,&nbsp;len)</tt>, but not by
  626. <tt>buf:set(array+offset,&nbsp;len)</tt>. The addition of the offset
  627. creates a new pointer, even when the offset is zero. In this case, you
  628. need to make sure there's still a reference to the original array as
  629. long as its contents are in use by the buffer.
  630. </p>
  631. <p>
  632. Even though each LuaJIT VM instance is single-threaded (but you can
  633. create multiple VMs), FFI data structures can be accessed concurrently.
  634. Be careful when reading/writing FFI cdata from/to buffers to avoid
  635. concurrent accesses or modifications. In particular, the memory
  636. referenced by <tt>buf:set(cdata,&nbsp;len)</tt> must not be modified
  637. while buffer readers are working on it. Shared, but read-only memory
  638. mappings of files are OK, but only if the file does not change.
  639. </p>
  640. <br class="flush">
  641. </div>
  642. <div id="foot">
  643. <hr class="hide">
  644. Copyright &copy; 2005-2023
  645. <span class="noprint">
  646. &middot;
  647. <a href="contact.html">Contact</a>
  648. </span>
  649. </div>
  650. </body>
  651. </html>