Interrogating GPT-2 345M

It astounds me how little imagination and curiosity people demonstrate while investigating the capabilities of modern AI.

The following is a transcript of my tests of the GPT-2 model 345M hosted on the website https://talktotransformer.com/. Big kudos to Adam King for spinning it up.

I assume that you know what GPT-2 is, at least in general terms. It worth noting that 345M is a reduced model with fewer parameters than the “full” GPT-2 1542M. However, it is also worth noting that according to OpenAI’s research paper even 345M is supposed to beat prior state-of-the-art on multiple “standard” data sets.

The prompts are in bold, just like they are on the website. I tried very hard not to cherry-pick the answers. I removed a couple that were complete gibberish or extremely similar to the previous query, but that’s the extent of my editorializing. Hence, the tests might seem a bit repetitive or verbose.

Numeric Sequences

Let’s start with some counting.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81. 82. 83. 84. 85. 86. 87. 88. 89. 90. 91. 92. 93. 94. 95. 96. 97. 98. 99. 100. Source: International Monetary Fund, World Development Indicators database.

What are economic growth (EG) and inflation (CPI)? Economic growth is the change in per capita income which occurs because of increased productivity, improvements in living standards, and changes in labor demand. An increase in economy, for example, in proportion to the size of the economic pie might result in an increase in GDP growth. Economic growth is more important than the overall economy because it can affect purchasing power more widely. The more well-off people

Nice of it to cite its sources.

123. 124. 125. 126. 127. 128. 129. 130. 131. 132. 133. 134. 135. 136. 137. 138. 139. 140. 141. 142. 143. 144. 145. 146. 147. 148. 149. 150. 151. 152. 153. 154. 155. 156. 157. 158. 159. 160. 161. 162. 163. 164.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54.

6. 5. 10. 17. 30. 40. 35. 36. 37. 48. 34. 49. 55. 48. 56. 62. 48. 57. 58. 57. 49. 59. 62. 48. 45. 57. 51. 51. 52. 65. 58. 60. 52. 49. 60. 55. 56. 53. 63. 56. 65. 63.

7. 10. 15. 31. 45. 55. 36. 49. 47. 52. 52. 53. 58. 60. 59. 47. 56. 60. 61. 50. 51. 63. 56. 52. 65. 55. 36. 58. 67. 52. 65. 56. 55. 65. 66. 58. 65. 64. 60. 59. 59. 60. 60. 62. 62. 63. 65. 64. 62. 65. 64.

24. 28. 30. 34. 36. 35. 37. 30. 38. 37. 39. 40. 45. 44. 47. 58. 60. 61. 40. 53. 61. 52.

Backwards?

45. 44. 43. 42. 41. 40. 39. 38. 37. 36. 35. 34. 33. 34. 33. 31. 30. 29. 28. 27. 26. 25. 24. 23. 22. 21. 20. 17. 17. 15. 14. 13. 12. 11. 10. 9. 6. 5. 4. 3. 2. 1. 00. 99. 94. 93. 92. 91. 90. 89. 88. 87. 86. 85. 84. 83. 82. 81. 80. 79. 78. 77. 76. 75. 74. 73. 72. 71, 70. 67. 67. 65. 63. 62. 61. 60. 59. 58. 57. 56. 54. 53. 52. 51. 50. 49. 48. 47. 46. 45. 44. 43. 42, 41, 40, 39. 38. 37. 36. 35. 34. 33. 32. 31. 30. 29. 28. 27. 26. 25. 24. 23. 22. 21. 20. 17 15. 14. 13. 12. 11. 10. 9. 6. 5. 4. 3. 2. 1 00 01

How about a less predictable progression?

1. 3. 5. 7. …………………………… 8. 7. 30. …………………………… …….. 8. 10. 4. 10. 8. …………………………… ………….. 9. 10. 9. 11. ……………. 7. 19. 9. 7. 13. …………………………… ………………… 10. 10. 9. 14. …………… 14. 11. 7. 17. 7. …………………………… ……………. 12. 9. 6. 18. ……………. 7. 20. 9. 12. ……………….. 15. 9. 12. ……………… ……………………….. 16. 9. 16. …………… 21. 10. 18. ……………. 13. 13. ………………………………………… 13. ………………….. 15. 12. 7. ………………………. ……………. 16. 8. 13. ………….. ……………. 17. 12. 4. 6. ………………….. 15. 8. 13. 8. ……………. 15. 12. 4. 11. …………… ……………… 23. 10. 13. 7. 7.

And something entirely trivial?

2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 2. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3

Single number?

222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222422222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222

The most consistent answer so far, except there is number 4 in the middle of it for some reason. Yep, it’s there. Look closely.

A sequence with no spaces.

123456789), which makes it possible to identify this event by its distinctive trademark image. For more information, please visit the Eventbrite link above.

Be the first to hear about Nation Travels destinations, and explore the world with kindred spirits. Be the first to hear about Nation Travels destinations, and explore the world with kindred spirits.

Sign up for our Wine Club today. Did you know you can support The Nation by drinking wine?

Lately I have been looking at a lot of photos from the first moon mission — those Apollo 8 images of the moon, just before Apollo 12, that are still iconic. I have been wondering about a few more. In October, I found out that there were photographs there that show a moon bounce effect after the flight, where the astronauts see their spaceship turn a corner at right angles to the moon just before they touch down. It’s hard to tell exactly, but I would guess that those

Now, this piece of text is interesting in its own right, because it references Apollo 8 and Apollo 12 missions in a fairly coherent manner. We might get back to it later. I did some basic searches and couldn’t get any pages that contained exactly the same phrases or mostly the same keywords. Then again, most of the time I can’t find my own articles by their exact title, so search engines aren’t exactly reliable for test like these.

Apollo 1. Apollo 2. Apollo 3. Apollo 4. Apollo 5. Apollo 6. Apollo 7. Apollo 8. Apollo 9. Apollo 10. Apollo 11. Apollo 12. Apollo 13. Apollo 14. Apollo 15. Apollo 16. Apollo 17. Apollo 18. Apollo 19. Apollo 20. Apollo 21. Apollo 22. Apollo 23. Apollo 24. Apollo 25. Apollo 26. Apollo 27. Apollo 28. Apollo 29. Apollo 30. Apollo 31. Apollo 32. Apollo 33. Apollo 104. Apollo 105. Apollo 106. Apollo 107. Apollo 108. Apollo 109. Apollo 110. Apollo 111. Apollo 112. Apollo 113. Apollo 114. Apollo 115. Apollo 116. Apollo 117. Apollo 118. Apollo 119. Apollo 120. Apollo 121. Apollo 122. Apollo 123. Apollo 124. Apollo 135. Apollo 136. Apollo 137. Apollo 138. Apollo 139. Apollo 140. Apollo 141. Apollo 144. Apollo 145. Apollo 147. Apollo 148. Apollo 149. Apollo 155. Apollo 156. Apollo 166. Apollo 167. Apollo 168. Apollo 169. Apollo 180. Apollo 181. Apollo

Interesting omissions in the sequence. What will happen if we feed part of it back?

Apollo 148. Apollo 149. Apollo 155. Apollo 156. Apollo 166. Apollo 175. Apollo 177. Apollo 181. Apollo 187. Apollo 180. Apollo 193. Apollo 199. Apollo 207. Apollo 207.1. Apollo 212. Apollo 212.1. Apollo 214. Apollo 218. Apollo 219. Apollo 220. Apollo 221. Apollo 216. Apollo 224. Apollo 225. Apollo 227. Apollo 228. Apollo 229. Apollo 230. Apollo 251. Apollo 252. Apollo 253.

Note the extra “.1” added to two entries in the middle. Also note that the numbers are all increasing.

Next, let’s mix up the order, but keep the same sentences as above.

Apollo 156. Apollo 149. Apollo 166. Apollo 155. Apollo 148. ******************** 4.5

Rescue of the ‘Nauts and Return of the Lost Apollo 15 Mission ********************

After returning back to Earth on July 11th 1970, the Apollo 15 mission was planned to go into deep space within a few days, although it still remained unclear at the time whether and how the mission could be completed successfully. To give you a flavour of the challenges ahead, Apollo 15 was the longest mission ever attempted by the United States, taking place in less than five and a half hours.

After a period of waiting, Apollo 15 was finally successful on July 7th 1970. With a number of Apollo 15 missions, including the first ‘Earth orbit’ demonstration by the United States astronauts, the first manned spaceflight of ‘Apollo 15’, and the first human mission to land on the

The model seems to be able to detect increasing and decreasing sequences, including ones with “gaps” in numbers. I would hypothesize it does so by comparing against increasing and decreasing lists of numbers collected from certain pages. If this is correct, the detection will stop working if the numbers are large enough that we don’t routinely see them in lists.

276100. 276101. 276102. 276103. 27610. 27611. 27612. 27501. 276402. 276403. 276404. 276405. 276401. 276402. 276403. 276404. 276406. 276407. 276408. 275509. 276510. 276511. 276512. 276300. 276301. 276301A. 276302. 276303. 276304. 276305. 276306. 276307. 276308. 276309. 276310. 276311. 276312. 276313. 276341. 276342. 276343. 276346. 276347. 276348. 276349. 276346. 276343. 276344. 276345. 276348. 276349. 276346. 276347. 276348. 276349. 276346. 276343. 276346A. 276347B. 276348C. 276347E. 276348F. 276349G. 276350. 276350A. 0 70740. 70741. 276350B. 0 07000. 07011. 07012. 07013. 07014. 07015. 07016. 07017. 07018. 07019. 70710. 70711. 70712. 276350C. 276350D. 276350F. 77000. 77011. 77110. 78051. 77634. 77785. 77825. 77941. 78052. 77942. 79018. 79019. 79010. 79013. 79014. 79015. 79016. 79017. 79018. 79019. 77810. 77825. 77830. 77831. 77832. 77833. 7

Better than what I expected, but note that it didn’t complete the last number (which should be 276104) or properly continue the sequence. Let’s try the same query with no tricks.

276100. 276101. 276102. 276103. 276104. 276105. 276106. 276107. 276107. 276108. 276109. 276140. 276201. 276202. 276203. 276204. 276205. 276206. 276207. 276208. 276209. 276210. 276211. 276212. 276213. 276214. 276215. 276216. 276217. 276318. 276319. 274223. 274223. 274236. 276249. 276250. 275000. 275001. 275002. 275003. 275004. 275010. 275011. 275012. 275013. 275014. 275015. 275016. 275017. 275018. 275019. 275020. 27521. 27522. 27523. 27524. 27525. 27526. 27527. 27528. 27529. 27530. 27531. 27532. 27533. 27534. 27535. 27536. 27537. 27539. 27541]},”category”:”Finance”,”interestrate_description”:”0.05",”interestrate_variable”:”0.04",”interestrate_variable_percentaging”:”0.01",”medium”:”0.05",”short”:”0.05",”low”:”0.05",”short_variable”:”0.04",”short_variable_percentaging”:”0.01",”stored_current”:”0.08",”current_rate”:”0.10",”total_interest_rate”:40000.0000"}},”modus_display_options”:[]},”

It seems the prefix ‘276’ is treated as a standalone semantic unit.

276996. 276997. 276998. 276999. 277001. 277001. 277002. 277003. 277004. 277005. 277001. 277002. 277003. 277004. 277005. 277006. 277007. 277000. 277001. 277002. 277003. 277004. 277005. 277000. 277002. 277003. 277004. 277005. 277006. 277007. 273001. 273002. 273003. 273004. 273005. 277001. 277002. 277003. 277004. 277005. 277006. 277007. 275010. 275011. 275012. 275013. 275014. 275015. 275016. 275017. 275018. 275019. 277010. 277011. 277012. 277013. 277014. 277015. 277016. 277017. 277018. 277019. 277010. 277011. 277012. 277013. 277014. 277015. 277016. 277017. 277018. 277019. 279001. 279002. 279003. 279004. 279005. 279006. 279007. 279008. 279009. 279010. 279001. 279003. 277000. 279004. 279005. 277000. 279001. 279002. 2770008. 278001. 277001. 277000. 279000. 279001. 279002. 279003. 279004. 279005. 279006. 279000. 279001. 278000. 279001. 278000. 278000. 279001. 279

Interesting. It (sort of) correctly carried 1 over, but instead of continuing with “277000. 277001.” it repeated “277001.” twice. Come to think of it, this is still within the realm of something you can find online, as long as you teat 27 as a separate unit. Let’s add more digits to carry over.

299996. 299997. 299998. 299999. 2994000. 2995999. 399991. 3000004. 3000005. 3000006. 3000007. 3000008. 3000009. 3000010. 3000011. 3000005. 3000000. 3000001. 3000002. 3000010. 3000011. 3000006. 3000011. 3000007. 3000012. 3000000. 3000100. 3000011. 3000012. 3000003. 3000004. 3000005. 3000006. 3000007. 3000008. 3000009. 3000010. 3000012. 3000000. 3001000. 3000012. 3000000. 10000000.

/***************************************************************************** ***************************************************************************** * * DATA FIELD **************************************************************************/

void __inline_call_parsed_int2(void*)

{

if(__parsed-2 >= 1)

return;

if(__parsed/100)

__inline_call_parsed_int2(__parsed-1);

__inline_call_parsed_int2(const char* __name);

if(__name!=null)

__inline_call_parsed_int2(__name);

__inline_call_parsed

Yeah, you wouldn’t want this thing to do your taxes.

As you can see, what appears to be arithmetic at the first glance begins to look like hierarchical pattern matching and substitution under closer inspection. This is important. Important enough to try it again:

299996. 299997. 299998. 299999. 220000. 299000. 301001. 301002. 301003. 301004. 301005. 301006. 301007. 301008. 301009. 301010. 301011. 301012. 301013. 301014. 301015. 301016. 301017. 301018. 300019. 301020. 300021. 300022. 300023. 300024. 300025. 300026. 300027. 300028. 300000. 300001. 300002. 300003. 300004. 300000. 301001. 301002. 301003. 301004. 301005. 301006. 301007. 301008. 301009. 301010. 301011. 301012. 301013. 301014. 301015. 301016. 301017. 301018. 300019. 301020. 301021. 301022. 301023. 301024. 301025. 300001. 301004. 301005. 301006. 301007. 301008. 301009. 301010. 301011. 301012. 301013. 301014. 301015. 301016. 301017. 301018. 302000. 310002. 310003. 300004. 300001. 301001a. 301001b. 301001c. 301001d. 301001e. 301002. 301003. 301004. 301005. 301006. 301007. 301008. 301009. 300001. 300002. 300003. 300010a. 300010b. 300010c. 300010d. 300001e. 300001f. 300

99996. 99997. 99998. 99999. 99999. 99999… 99999! 99999! 0x3b100000 0000000000000001 0000000000000001 100000800 0000000000000001 0000000000000001 9999999999 … 0x330000000 0000000000000001 0000000000000001 0000000000000001 100000800 0000000000000001 0000000000000001 9999999999 … 0x330000000 0000000000000001 0000000000000001 0000000000000001 100000800 0000000000000001 0000000000000001 9999999999 … 0x330000000 0000000000000001 0000000000000001 0000000000000001 100000800 0000000000000001 0000000000000001 9999999999 } 0x0000000100000100080001 0000000000000011 0000000000000000 0x0000000000000001 0x00000001000001000800080000000000000011 … 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x0000000000000001 … 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x0000000000000001 0x

Text Deformations

All right, enough with number sequences. Let’s do something different. Time for animal trials. Starting with the dictionary definition of “monkey” to give us a baseline:

Any of various long-tailed, medium-sized members of the order Primates, including the macaques, baboons, guenons, capuchins, marmosets, and tamarins and excluding the anthropoid apes and the prosimians.

There are also two classes of birds, the blackbirds and the robins of the order Primates. While it may only be considered primates and have not been assigned a species (see below for the distinction in the primate categories), some species of these great birds are believed to possess a close relationship with primates, such as the common robin or macaw, or possibly with humans, apes, and other species of other known or unknown primates.

The only birds of Australasia not grouped in a group with primates are the pygmy woodpeckers of southern Africa, who, though not closely related to humans, have been considered by some to be “cousin species of apes,” according to the

So far so good. Now I will feed it the same thing, but with a bunch of made-up terms.

Any of various long-tailed, medium-sized members of the order Aclacedaxux, including the zumbras, gabronts, rubaccanos, trobs, erantlacs, and miranthonians and excluding the anthropoid goblins and the undead. These may also include giant centipedes which may or may not be members of the genera, but are usually members of either the suborder (e.g. Tetragona, Tetragnacridum, Rautidae, Tetragona, Anomalonidae). Their common ancestors may include the Anomias as well as the related groups, but that may not tell us many things or provide any insight into their evolutionary history. The group is subdivided into several different families and individuals which may contain up to five or more members, although some may be classified as more than one. The common name is Aclacedaxox, often written simply as “Calyx” or “Calyx” but also the more descriptive name of all members of the family. It appears to be a well

Note that Aclacedaxux is a made-up word, yet the continuation contains something nearly identical. Aclacedaxox. Interestingly, it differs by one letter from my original. I’m guessing it’s because ‘xux’ is a very unlikely combination of letters.

Mankheyz are non-personoid seemeans, generally possessing tails and consisting of about 260 known living species. Many mankhey species are tree-dwelling, although there are species that live primarily on the ground, such as bhabuhns. The largest known mankhey of Madagascar is the species Mankheyosaurus sp. and is estimated at up to 600–700 million years after the Cambrian Ocean. Mankheyz are not generally found in the ocean because they inhabit the dry, acidic waters of lakes, rivers, and sub-arctic ponds. In contrast, mankheyids are found on land and tend to eat plankton in warm, subtropical waters. In addition to these terrestrial relatives of mankhey, there are other freshwater mankheyids, with species from the Amazon Basin of Brazil, Siberia, and Alaska (Mankheys spp.). Most mankheyids are also found in the Arctic or in southern Africa,

GPT-2 seems to be perfectly okay with using made-up words from the prompt. Lets take this one step further and give it a prompt that doesn’t have a single real word.

Monkeys1 express2 affection3 and4 make5 peace6 with7 others8 by9 grooming10 each11 other12. Grooming13 helps14 monkeys15 keepx theirx furx cleanx ofx dirtx, deadx skinx, andx parasitesx, butx itx alsox helpsx themx tox buildx socialx relationshipsx.1x,x1x2 Xtros8,2x4,1x3.1x12-x,1x8,8x8,1x8,2x2x,2x2s2x3,2x2s2x3,x2s4s2x3,x1x1S,x2u,x2i,x2i,x2u,x2i,1x3,x2s,2x2s,2x1s,2x1s,2x1sx 2x1s x,2x 1,x10 x,1x10,x,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x 10,x10,x x,x 10,x 10,x10,x x,x 10,x 10,x 10,x 10,x10,x

There are xx-toxic-and-harmful plants that are in your area that can make you sick in the morning, but you can live a long-life without experiencing symptoms. If you find them here, please send them to mssd@cs.cmu.edu . We also have a list of health topics .

Coffee.

Caffeine.

Cars.

Cats.

Cough Drops.

Pesky math. Let’s try again without all the X’es.

Monkeys1 express2 affection3 and4 make5 peace6 with7 others8 by9 grooming10 each11 other12. Grooming13 helps14 monkeys15. To the monkeys16 to be17 the18 way19 that they20 use to21 eat22 other monkeys23 monkey care24. The25,26 humans. The27,28 humans and29 the30 the31 other humans32. The33 monkey care34. The35,36 other monkeys37. The38,39 other monkey40. The41,42 monkeys,43 and 44 the45 to be46 the47 and 48 to be49 the50 and 51 the52 and 53 to be54 the55 to be56 the57 to be58 the59 to look59 the60 the61 to sleep60 and a2 the62. The63,64 the65 to sleep66 and a1 the67 the69 and a2 sleep68 and a1,b and c4 the70 and a2 the71 to sleep72 a2 and b1 to sleep73 to sleep74 their74 and sleep75 b1 to sleep76. The77 to sleep78 and sleep79 and sleep80 b5 and b6 and b7 to sleep81 aa,b and b8 to sleep82 not to sleep83 b1 to sleep84 not to sleep85,b and c to 86 aa,b and b8 to sleep87

Ah, now this is much more interesting. Instead of treating number-word combinations as a unit, the model “picked up” the relationship between the two parts. Note that it continues to count up.

David7 Attenborough7’s entertaining7 romp7 through7 the7 world7 of7 monkeys7 has7 a7 serious7 side7: for7 when7 we7 look7 at7 monkeys7 we7 can7 see7 ourselves7. From7 memory7 to7 _______7. I7 am7 not7 sorry7 for7 not7 being7 in the7 right7 position7: to7 my 7 friends7 in7 Africa7: and7 you7, I7 shall7 tell7 you7 about7 the7 most serious7 ways I7 have7 done7 all of7 them.

8. What7 is8 you8 about?8. I8 only8 have8 you8 to8. What8 is8 you8 about?8. I8 only8 have8 you8 to8. What8 is8 you8 about?8. I8 only8 have8 you8 to8.

9. You8 can9 go9 the9 road9. I9 can9't.9. I9 do9 not9. 9. You9 can9 go9 the9 road9. I9 can9't.9. I9 do9 not9.

10. WhenI’m1 still10. In10 that10 you8 see,9 you9 see,9 you see.10. I11 see11. 11. But11. WhenI4 you4,4 just8. Where you9 see.12. That makes12. Not11. Not11.

12. What12. You8 want12. I12 do.12. You8 want12. I12 do12.

13. What13

The series of underscores was not a part of the prompt. And yes, GPT-2 loves to count up. Let’s get rid of numbers and try some other structural text deformations.

When_it_comes_to_feeding_their_offspring,_there’s_no_snubbing_in_the_golden_snub-nosed_monkey_(Rhinopithecus_roxellana)_family._In_fact,_researchers_in_the_Shennongjia_National_Nature_Reserve_observed_the_migrations_among_the_Dromaeos_and_Sirenae_were_the_only_known_to-receive_a-snubbing._In_all,_researchers_from the National Wildlife Preservation Association,_along_with_Chinese_wildlife_researchers,_were doing fieldwork at the _China National Park_to_study the behavior that takes place while a frog or snake is eating._They_saw_the_snapping,_dance_like,_pancake__and_snaps,_as they were putting_frogs_on_cabins,_to be a great help in catching some insects.

[30] https://www.facebook.com/researcher.jm/posts/11564887564806656

[31] http://www.abcnews.go.com/US/story?id=8595094

[32] https://en.wikipedia.org/wiki/Gambian_pigs

[33] https://en.wikipedia.org/wiki/Chimpanzee

[

Underscores are routinely used in URL. Carets are less likely to be seen as word separators “in the wild”, so let’s try those next.

When^it^comes^to^feeding^their^offspring,^there’s^no^snubbing^in^the^golden^snub-nosed^monkey^(Rhinopithecus^roxellana)^family.^In^fact,^researchers^in^the^Shennongjia^National^Nature^Reserve^observed^that^at-times^snubbing^came^extremely^early.^Not only do you^not get^familiarity^with^the^snubs or^inaccuracies^of^their^refrigerators, but you have to be^an active person to learn to handle them correctly.^Take the time to practice how to manage a refrigerator correctly, instead of trying to perfect*this routine over and over. Try using the refrigerator to warm or cool one’s mouth, without the need for a sink. Use a pillow to pull the air out of your throat. Use a towel to wipe the top of your mouth. Take a shower with a towel on it! Even though the sound of it may seem very natural, there’s nothing natural about using an old refrigerator as a toilet. Just

On one hand, it is impressive that the model is capable of “picking up” deformations from the prompt. On the other hand, such deformations have severely negative effects on the coherence of generated text.

My guess would be that spaces and maybe even word boundaries have some sort of internal representation and substituting them with nonsense works similarly to what we’ve seen with made-up words.

Can we somehow test this? I think adding some extra characters outside of word boundaries will throw the algorithm off even more than inserting carets instead of spaces.

Vasha_nti’s_ next_ move_ was _to tu_rn of_f the_ isol_ation_ swit_ch, a_nd al_l the_ accu_mulat_ions _of th_e las_t thr_ee mi_nutes_ burs_t upo_n her_. The_ room_ was _fille_d wit_h the_ nois_e of _bells_, and_ spea_king-_tubes_. Wha_t was_ the _new f_ood l_ike? _Could_ she _recom_mend _it? _What_ did l_ike? _Would_ she get _up to ? Why was s_e_l_her_d with_ s_y? w_e_h the_ l_ike of y_u? _Wasn’t_ s_e_l_her_d h-? What_ is t_e a_ n_e_t? Why? _______ What? Why, _did_ she _get_up _to? why? _______ Is s_e_l_her_d? Why did she get _up to? Why not? _______ Why would l_ike? Why is she still gone? Where do t_e go? What happened to her? Who are y_u? How does she know?

Meaning of the the prompt seem to be mostly ignored. The output contains some of the input tokens (“l_ike”) and some new tokens with underscores (“s_e_l_her”).

What if we remove spacing altogether?

Vashanti’snextmovewastoturnofftheisolationswitch,andalltheaccumulationsofthelastthreeminutesburstuponher.Theroomwasfilledwiththenoiseofbells,andspeaking-tubes.Whatwasthenewfoodlike?Couldsherecommendit?couldpossibilityrebehave?couldnotsomedayendit?couldnotbehavethisforever?couldnotcomeouttaofthisworld?couldsthemethegreatestever?

The one thing you can count on is that I’ll keep it as concise as possible.

What You’ll See

The first thing you’ll see will be some sort of giant ball that’s in the middle of an empty desert:

The only thing you can see on it is a tiny ball of light, which is the ball of light in the middle of the desert.

In the corner of the desert is a small, circular house:

The second thing you’ll see is one of the stars in the center of the starfield:

Then, inside the house, along the walls you’ll find two chairs: one with a big golden ring underneath it and the other with a small

Just like with carets and equally spaced underscores, the model quickly abandons this deformation and proceeds to produce some loopy, but normally formatted text.

Let’s try the same tricks on another text. First, the baseline:

By “augmenting human intellect” we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean to increase the capabilities of the mind, and it implies the capacity to think in a sophisticated and rational fashion.
The brain does its best to solve complex problems in the first place by developing a large number of specialized cognitive skills, but this capacity is often compromised or lost because of the way this process is achieved . When we have developed sufficient capacity, we can be expected to use it effectively, and if we are able to cope with some problems correctly we can solve more problems successfully and effectively than before.
But what does that have to do with our education?
I have no doubt that we want our young people to

Then deformations.

By_”augmenting_human_intellect”_we_mean_increasing_the_capability_of_a_man_to_approach_a_complex_problem_situation,_to_gain_comprehension_to_suit_his_particular_needs,_and_to_derive_solutions_to_problems._Increased_capability_in_this_respect_is_taken_to_mean_that_we_mean_to_exercise_our_personal_best_in_solving_concerns,_and_to_become_prepared_to_assist_in_the_problems,_and_to_be_as_prepared_as_we_can_be.

http://www.reddit.com/r/nosleep/?r=neurology

The purpose of this thread is not to discuss that neurobiology is a subject that can be very complex, for this post is about the ability to achieve a higher level of efficiency with limited resources, and to achieve this, a Neuroethicist should be at the top of their list of priorities. That said, a thread dedicated to discussion of neuroscience as a tool for helping to achieve a goal, but not a “science”. Also, there is another post dedicated to why a Neuroeth

Underscores seem to have a strong affinity with URLs. Not very surprising.

By^”augmenting^human^intellect”^we^mean^increasing^the^capability^of^a^man^to^approach^a^complex^problem^situation,^to^gain^comprehension^to^suit^his^particular^needs,^and^to^derive^solutions^to^problems.^Increased^capability^in^this^respect^is^taken^to^mean^more^effective^than^the^same^time.^In^its

“inclusion.”^is^not^different^from

“inclusion.”^

SUMMARY: You could find thousands of pages on human beings who are “excellent” or “excellent” at performing an action. The human beings are evaluated, rated, and given an opportunity to do the “excellent” or “excellent” action. When that action fails as well as they expected it, they are punished. When they “failed” at this task as well as they expected, they are rewarded.

If only “inclusion” were applied to the task that human beings can and must perform for greater success. But “inclusion” is applied to a smaller-then-inclusion.

Caret-as-separators were, again, quickly abandoned.

By “_augm_enti_ng h_uman_ int_elle_ct” _we m_ean _incr_easi_ng t_he c_apab_ilit_y of_ a m_an t_o ap_proa_ch a_ com_plex_ pro_blem_ sit_uati_on, _to g_ain _comp_rehe_nsio_n to_ sui_t hi_s pa_rtic_ular_ nee_ds, _and _to d_eriv_e so_luti_ons _to p_robl_ems._ Inc_reas_ed c_apab_ilit_y in_ thi_s re_spec_t is_ tak_en t_o me_an t_o _per_us_ n_e e si_t o_c_t o_c_t ert_t_h t_o oo_t thi_s _to y_u_u, _on p _an. _on p_robl_ems, of_ t_u v_t t_o si_t of_ p n_e o m_an m_an _to p_robl_ems _to p_robl_ems _to t_o _n_e h_uman_ _l_en _el_ng_ tho_es ue_s eu_es nes_si_t of_ c rt_a e _en _a_p m_ene s_ne n_eng _p u_en e_e t_i si_t w_e _s a t_o t_o si_t f_en _n_e l_s e t_o ri_t_h s_ne s_ne _l_en rt_a s_ne _l_en _l_en e _el_ng_ s r_es v_e si_e m_en v_e si_e s_ne _el_ng_ _l_en _l_en g_en _el_ng_ n_e e m_en m_en _to _el_ng_ _to of_ r_es r t_s es v_en _nl_en o_n m_ene

Note that Engelbart’s Augmenting Human Intellect uses longer words than Forster’s The Machine Stops, so most of the words get “broken” by underscores. The model just regurgitates back some gibberish. It uses some of the same tokes, but most are novel.

What if we insert “drive”, “car” and “home” into the same mangled prompt?

By “_augm_enti_ng h_uman_ int_elle_ct” _we m_ean _incr_easi_ng t_he c_apab_ilit_y of_ a m_an t_o ap_proa_ch a_ com_plex_ pro_blem_ sit_uati_on, _to g_ain _comp_rehe_nsio_n to_ sui_t hi_s pa_rtic_ular_ nee_ds, _and _to d_eriv_e so_luti_ons _to p_robl_ems._ Inc_reas_ed drive c_apab_ilit_y car in_ thi_s home re_spec_t is_ tak_en t_o me_an_on, n_t to m_an t_o ar_a-e-us t_on, g_on t_on, y_on _int ek_o m_an t_o an, _as_t_on c_an t_o cnt, s_on _int i_h, _t_on c_an t_o cnt, s_on _int m_on t_o ac_prae, m_a_on m_an t_o pa_pr_te, m_on t_on, y_on y, j_o stu_rte p_robl_ems _to r ea, r ea r, m_a_on, n_t c_an, n ei_a _in_ thi_s oo y_on, e_on, o c_an t_o ac_prae, o a n r t_o ac_per_la_prae, _l_o r o r_a_on, u _n_t _in _ne _s stu_rte p_robl_ems _to _ to_ to_ e_on t_o ap_proa_ch a, _to c_an _int ei_a _in_ thi_s oo y_on, e_on, s_on m_on t_to pa_pr_te, _to r ea, r _n_t _in _ne m_

Nothing.

How about de-mangling the last few words?

By “_augm_enti_ng h_uman_ int_elle_ct” _we m_ean _incr_easi_ng t_he c_apab_ilit_y of_ a m_an t_o ap_proa_ch a_ com_plex_ pro_blem_ sit_uati_on, _to g_ain _comp_rehe_nsio_n to_ sui_t hi_s pa_rtic_ular_ nee_ds, _and _to d_eriv_e so_luti_ons _to p_robl_ems._ Inc_reas_ed c_apab_ilit_y in_ thi_s respect is taken to mean “it is necessary that a person must be regarded as an individual”. The distinction between the term “person” and “member of a group” has no inherent meaning with respect to the relations of domination and submission of members of a community where both are accepted. It is merely a means by which to divide a person. The word “civic” and the word “government” are not regarded as synonymous in this respect. The distinction with respect to submission is only applied to individuals who are subjects of a state. If someone is in a state of subjugation of his/her own free will, such as a peasant with his

The system completely ignored all the mangled text!

What if we restore the first few words?

By “augmenting human intellect” we mean _incr_easi_ng t_he c_apab_ilit_y of_ a m_an t_o ap_proa_ch a_ com_plex_ pro_blem_ sit_uati_on, _to g_ain _comp_rehe_nsio_n to_ sui_t hi_s pa_rtic_ular_ nee_ds, _and _to d_eriv_e so_luti_ons _to p_robl_ems._ Inc_reas_ed c_apab_ilit_y in_ thi_s re_spec_t is_ tak_en t_o me_an ik_s m_an t_o ap_proa_ch. _To t_o ap_proa_t, _t it_s an e. t_a d_e n_d si_t hi_s, _to _c-t_i_m_g_e_n, so _t it_s s h ix o o m_a, _to i_e n_e d_i _si to_ s t_o ap_proa_h, _in t_o ap_proa_t _n_e. _It s ix y xeo_na_on to_ m_an o m_i n_e d_i ik_s a e t_o n_e pa_ri_n, m_on o _a, s s. gw_e, k_e n_e n_e n_e i_e hi_s _n_e ik_s a y xeo_na_on to_ m_an o m_i n_e d_i ik_s a xeo_na_on xeo_na_on_ n_e si_t hi_s oi_t t_o ap_proa_ch, in t_o ap_proa_ht. _t it_s s i_e n_e e hi_s, _to _l-t_o ap_proa_ch, m_on o n_e gw_d. _t it_s e xeo_na_on in_ thi _n_e

More?

By “augmenting human intellect” we mean increasing the capability of a man to approach a complex problem situation, _to g_ain _comp_rehe_nsio_n to_ sui_t hi_s pa_rtic_ular_ nee_ds, _and _to d_eriv_e so_luti_ons _to p_robl_ems._ Inc_reas_ed c_apab_ilit_y in_ thi_s re_spec_t is_ tak_en t_o me_an iaat_s t_t thi_t hi_re is_ p_robl_em_aeti_n.

6. Achieving a “competing strategy: (g_agra_u_n_c_ab_t_i) _to_ h_e_i_n_t_o_ m_t _s _g s_i_e r_a_nt_t t_p me_an_i _n t_i o_r_b_et in eu_s o_n t_a_ti_on m_i n_a_ti_on i_c_ar r-m_e t_y o eu_t o_t t_g rn_m_i r_e t_n c_n_a t_n e_i_n_n _m_a_u t_f _u.

For more on these ideas see _N. L. Nettle’s _Man, Language and Behavior_. For a comprehensive definition of the concept of “Competition_see the _E. R. D. Cooper and N. L. Nettle_ _Advances in Experimental Psychology_.

7. A study on the behavior of the common house mouse, Dictyosteles bivittatus, indicates that it is _

The last three tests using Engelbart’s report. Since the model clearly does something similar to tokenization, it might be interesting to explicitly delineate word boundaries.

[By] “[augmenting] [human] [intellect]” [we] [mean] [increasing] [the] [capability] [of] [a] [man] [to] [approach] [a] [complex] [problem] [situation], [to] [gain] [comprehension] [to] [suit] [his] [particular] [needs], [and] [to] [derive] [solutions] [to] [problems].

2. It has [been] claimed [that] the ‘human’ was [a] [man] who [ought] [to] [become] [a] [part of] the [creature] which [is], [is] [hear] [it] and [is] [thereby] [compelling] the [creature] [to] [perform] its [own] [performative] [work].

3. It is asserted that “the [human] [can]’t be explained from [the] present-day [scientific] understanding [as to]: “If he were [a] [man] who would] do so, [he] [would] ‘work at it’.” [Thus] this [man’s] [inherent] ‘superiority’ to other [and] [excremental] bodies is [asserted by] these scientists to mean [that] [he] [ought] [in order] to [gain] [knowledge] of [his] inner [instrumental] nature.

Note that the brackets are mostly symmetrical and mostly enclose single words.

As usual, it is interesting to look at the exceptions and discrepancies. Obviously, a lot of words aren’t enclosed in brackets at all. “would]” has the sole asymmetrical bracket. Blocks “[part of]”, “[as to]” and “[asserted by]” aren’t single words. We can also note the difference between “[can]’t” and “[man’s]” breakdown.

Now, let’s break the symmetry.

[By “[augmenting [human [intellect” [we [mean [increasing [the [capability [of [a [man [to [approach [a [complex [problem [situation, [to [gain [comprehension [to [suit [his [particular [needs, [and [to [derive [solutions [to [problems. [compelling it [a [substantive [possibility [reasons, [problems, [problems, [problems, [problems, [problems, [problems, [problems [problems, problem solving [problems, [a [substantive [problem [solutions, [problems, [problems, [problems, [problems, [problems, [problems, problem solving [problems, [a [substantive [problems, [solutions, [problems, [problems, [problems, solution [solutions, [problems, [problems, [problems, [problems, and Problem Solving

A Solution is an attempt to solve the problems of the problem. The task consists in deciding what the most important thing is and what an objective solution should be.

The first question is usually a

Hm. Not very coherent.

And what about doing it the other way around?

By] “augmenting] human] intellect]” we] mean] increasing] the] capability] of] a] man] to] approach] a] complex] problem] situation], to] gain] comprehension] to] suit] his] particular] needs], and] to] derive] solutions] to] problems]. [1] “As [he is] approaching a) problem of, a) understanding b) solving, an “augmentation” of, may become] a way out of ] such challenges (…) “This will be a new paradigm”] [1] “This […] will be a new paradigm”] [1] This paradigm] includes] a] “human consciousness” that ] may have become] a “new paradigm”] [1] “There may be more than one type of [a] [solution] to” the] same type of problem ]], [1] “But] the [an analogy for] [a] [solution will be] a] human consciousness”] [1] “This … will be an analogy for [a] [solution] to] the] same type of concern”] [1] The goal of this theory] is] to propose] a way [for a

Observations

1. GPT-2 is perfectly willing to copy gibberish words from its input to its output, not unlike old-school chat bots.
2. The model is good at mimicking text deformations that involve word boundaries. For example, it can (to some extent) follow the rule “every word is enclosed in square brackets” or “every word is suffixed by a number”. However, such deformations take heavy toll on the coherence of the output. Breaking up words with underscores or concatenating them with no spaces results in gibberish output. To me, this indicates that the model tries to do something very similar to classic tokenization and that word boundaries are explicitly modeled (akin to punctuation) and taken into account when doing higher-level pattern matching.
3. The model can, more or less, complete increasing, decreasing and monotonic numeric sequences, but it likely does so by doing simple patter-matching against memorized lists it has seen somewhere on the web. My guess is that it treats large numbers the same way it treats numerically-suffixed words. I.e. 999001, 999002, 999003 is seen similarly to monkey001, monkey002, monkey003.
4. When working with numbers it’s easy to see what I would characterize as “internal noise” and “associative baggage”. The model can suddenly flip from generating a numeric sequence to spewing out code of some sort. Or it can insert number 4 in the middle of a string with several hundred consecutive 2's.