{"id":1808,"date":"2021-10-19T17:24:19","date_gmt":"2021-10-19T22:24:19","guid":{"rendered":"https:\/\/singularityumexicosummit.com\/?p=1808"},"modified":"2021-10-19T17:24:19","modified_gmt":"2021-10-19T22:24:19","slug":"microsofts-massive-new-language-ai-is-triple-the-size-of-openais-gpt-3","status":"publish","type":"post","link":"https:\/\/singularityumexico.com\/en\/microsofts-massive-new-language-ai-is-triple-the-size-of-openais-gpt-3\/","title":{"rendered":"Microsoft\u2019s Massive New Language AI Is Triple the Size of OpenAI\u2019s GPT-3"},"content":{"rendered":"

Just under a year and a half ago OpenAI announced completion of GPT-3<\/a>, its natural language processing algorithm that was, at the time, the largest and most complex model of its type. This week, Microsoft and Nvidia introduced<\/a> a new model they\u2019re calling \u201cthe world\u2019s largest and most powerful generative language model.\u201d The Megatron-Turing Natural Language Generation model (MT-NLG) is more than triple the size of GPT-3 at 530 billion parameters.<\/p>\n\n\n\n

GPT-3\u2019s 175 billion parameters was already a lot; its predecessor, GPT-2<\/a>, had a mere 1.5 billion parameters, and Microsoft\u2019s Turing Natural Language Generation<\/a> model, released in February 2020, had 17 billion.<\/p>\n\n\n\n

A parameter is an attribute a machine learning model defines based on its training data, and tuning more of them requires upping the amount of data the model is trained on. It\u2019s essentially learning to predict how likely it is that a given word will be preceded or followed by another word, and how much that likelihood changes based on other words in the sentence.<\/p>\n\n\n\n

As you can imagine, getting to 530 billion parameters required quite a lot of input data and just as much computing power. The algorithm was trained using an Nvidia supercomputer made up of 560 servers, each holding eight 80-gigabyte GPUs. That\u2019s 4,480 GPUs total, and an estimated cost<\/a> of over $85 million.<\/p>\n\n\n\n

For training data, Megatron-Turing\u2019s creators used The Pile<\/a>, a dataset put together by open-source language model research group Eleuther AI. Comprised of everything from PubMed to Wikipedia to Github, the dataset totals 825GB, broken down into 22 smaller datasets. Microsoft and Nvidia curated the dataset, selecting subsets they found to be \u201cof the highest relative quality.\u201d They added data from Common Crawl<\/a>, a non-profit that scans the open web every month and downloads content from billions of HTML pages then makes it available in a special format for large-scale data mining. GPT-3 was also trained using Common Crawl data.<\/p>\n\n\n\n

Microsoft\u2019s blog post<\/a> on Megatron-Turing says the algorithm is skilled at tasks like completion prediction, reading comprehension, commonsense reasoning, natural language inferences, and word sense disambiguation. But stay tuned\u2014there will likely be more skills added to that list once the model starts being widely utilized.<\/p>\n\n\n\n

GPT-3 turned out to have capabilities beyond what its creators anticipated, like writing code, doing math, translating between languages, and autocompleting images (oh, and writing a short film<\/a> with a twist ending). This led some to speculate<\/a> that GPT-3 might be the gateway to artificial general intelligence<\/a>. But the algorithm\u2019s variety of talents, while unexpected, still fell within the language domain (including programming languages), so that\u2019s a bit of a stretch.<\/p>\n\n\n\n

However, given the tricks GPT-3 had up its sleeve based on its 175 billion parameters, it\u2019s intriguing to wonder what the Megatron-Turing model may surprise us with at 530 billion. The algorithm likely won\u2019t be commercially available for some time, so it\u2019ll be a while before we find out.<\/p>\n\n\n\n

The new model\u2019s creators, though, are highly optimistic. \u201cWe look forward to how MT-NLG will shape tomorrow\u2019s products and motivate the community to push the boundaries of natural language processing even further,\u201d they wrote in the\u00a0blog post<\/a>. \u201cThe journey is long and far from complete, but we are excited by what is possible and what lies ahead.\u201d<\/p>\n\n\n\n

\n\n\n\n

Image Credit:\u00a0Kranich17<\/a>\u00a0from\u00a0Pixabay<\/a><\/em><\/p>\n\n\n\n

Author:<\/strong><\/p>\n\n\n\n

Vanessa is senior editor of Singularity Hub. She’s interested in renewable energy, health and medicine, international development, and countless other topics. When she’s not reading or writing you can usually find her outdoors, in water, or on a plane. Learn <\/a>More<\/a><\/p>\n\n\n\n

Original Article<\/a><\/p>","protected":false},"excerpt":{"rendered":"

Just under a year and a half ago OpenAI announced completion of GPT-3, its natural language processing algorithm that was, at the time, the largest and most complex model of its type. This week, Microsoft and Nvidia introduced a new model they\u2019re calling \u201cthe world\u2019s largest and most powerful generative language model.\u201d The Megatron-Turing Natural Language Generation model […]\n","protected":false},"author":1,"featured_media":1809,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"episode_type":"","audio_file":"","podmotor_file_id":"","podmotor_episode_id":"","cover_image":"","cover_image_id":"","duration":"","filesize":"","filesize_raw":"","date_recorded":"","explicit":"","block":"","footnotes":""},"categories":[13],"tags":[26,27],"series":[],"class_list":["post-1808","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articulos-ingles","tag-artificial-intelligence","tag-inteligencia-artificial-2"],"episode_featured_image":"https:\/\/singularityumexico.com\/wp-content\/uploads\/2021\/10\/natural-language-processing-Microsoft-Megatron-Turing-model-AI.jpg","episode_player_image":"https:\/\/singularityumexico.com\/wp-content\/uploads\/2023\/05\/11711533-1673157178559-89a95be153719-4-scaled.jpg","download_link":"","player_link":"","audio_player":false,"episode_data":{"playerMode":"dark","subscribeUrls":{"apple_podcasts":{"key":"apple_podcasts","url":"","label":"Apple Podcasts","class":"apple_podcasts","icon":"apple-podcasts.png"},"stitcher":{"key":"stitcher","url":"","label":"Stitcher","class":"stitcher","icon":"stitcher.png"},"google_podcasts":{"key":"google_podcasts","url":"","label":"Google Podcasts","class":"google_podcasts","icon":"google-podcasts.png"},"spotify":{"key":"spotify","url":"","label":"Spotify","class":"spotify","icon":"spotify.png"}},"rssFeedUrl":"https:\/\/singularityumexico.com\/en\/feed\/podcast\/the-feedback-loop-by-singularity","embedCode":"

Microsoft\u2019s Massive New Language AI Is Triple the Size of OpenAI\u2019s GPT-3<\/a><\/blockquote>