Better Language Models and their Implications

If you are into Natural Language Processing (NLP), the last few years sure have been exciting for you. We’ve experienced Google Assistant making human-like calls, and open AI making the terminator treat seem credible with its GPT and GPT 2 models doing wonders. This article is aimed to provide a perspective to the latter. What exactly did the researchers come up with that is too sensitive to even release?

To put things in perspective, imagine a model capable of fabricating a short story on its own based on a line or two worth of prompts. A model that can answer general knowledge questions, which outperforms most humans in standardised text comprehension tests. A model that can teach a 3-year-old to construct sentences on its own. GPT-II released by open AI is potent enough to be seen as a harbinger of doom.

Here is a sample of text written entirely by the model. Judge it by the fluency of language and grammar and not by the reality of the passage:

The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.

Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them — they were so close they could touch their horns.

While examining these bizarre creatures, the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.”

Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America.

While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilisation. According to Pérez, “In South America, such incidents seem to be quite common.”

However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution or at least a change in social organisation,” said the scientist.

As you may have inferred, the model is capable of generating samples from a variety of prompts that feel close to the human quality and show coherence over a page or more of text. These samples have substantial policy implications: large language models are becoming increasingly easy to steer towards scalable, customised, coherent text generation, which in turn could be used in several beneficial as well as malicious ways.

Abroad, general language models could have significant societal impacts, and also have many near-term applications. We can anticipate how systems like GPT-2 could be used to create:

These findings, combined with earlier results on synthetic imagery, audio, and video, imply that technologies are reducing the cost of generating fake content and waging disinformation campaigns. The public at large will need to become more sceptical of text they find online, just as the “deep fakes” phenomenon calls for more scepticism about images.

Now that you have been shaken up properly let us introduce the model formally — GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a data set of 8 million web pages. GPT-2 is prepared with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the data-set causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10x the parameters and trained on more than 10x the amount of data.

As a prudent move, Open AI decided against releasing the full model for obvious reasons (?). They were noticeable enough when the model started spewing fake news articles and anti recycling texts on its own. The demonstration model armed with 117 million parameters is also optimised to outperform the previous monster, BERT on the GLUE index. Here is a sample of that too :

The dog on the ship ran off, and the dog was found by the crew.

The toned-down GPT 2 model wrote the bold part in the above sentence. This is the beginning of a new era in the field of NLP and the simple nature of the model design of GPT 2 is suggestive of the fact that many more sophisticated models are yet to come.

Related posts:

Mich Hancock is one busy gal. Besides being CEO of 100th Monkey Media, a company focused on creating quality human connections and interactions for B2B and B2C clients through social media, she is…

The digital transformation in the healthcare sector is rapidly transforming the way healthcare services are coordinated. Much of the transformations that have been evidenced are in line with…

Seringkali kita mendapatkan sms spam, misal sms pemenang undian, iklan agen pulsa, penawaran pinjaman online, ataupun sms mama minta pulsa. Tentu menyebalkan, bukan? Dengan bantuan fitur yang ada di…