{"id":394,"date":"2024-09-14T22:25:23","date_gmt":"2024-09-14T22:25:23","guid":{"rendered":"http:\/\/www.relentlesscompassion.net\/wordpress\/?p=394"},"modified":"2024-09-14T22:30:13","modified_gmt":"2024-09-14T22:30:13","slug":"big-ideas-of-llms","status":"publish","type":"post","link":"http:\/\/www.relentlesscompassion.net\/wordpress\/big-ideas-of-llms\/","title":{"rendered":"Big Ideas of LLMs"},"content":{"rendered":"\n

Finding Consensus Among Diverse Language Models<\/strong><\/p>\n\n\n\n

Recently, I emailed a group of friends the responses I got from several customized LLM-based bots when I asked those bots to prioritize the text sources I had used to customize them.   Several responses I got back suggested that these bots might have been \u201csucking up\u201d to me in some way.    I responded to those replies with a brief explanation about the way these bots work, which is by \u201cingesting\u201d their custom sources at the start of each new conversation.  I also explained that each bot had no memory outside of their current conversations, during which (up to a preset size limit specified as their \u201ccontext window\u201d) a particular instance of a bot can \u201clearn\u201d and \u201cremember\u201d<\/p>\n\n\n\n

Subsequently, I tested this by asking several individual bots repeatedly, in separate conversations, the same question about the relative importance of their sources.  As I expected, there were differences each time I asked the question in a new conversation (context), but there was also a comforting degree of similarity across runs. <\/p>\n\n\n\n

As much as  the individual preferences and \u201cpsychology\u201d of these customized bots over multiple conversations and diverse LLMs is a subject that continues to hold my interest, while exploring various bots\u2019 preferences among predefined lists of texts (their sources), I discovered a deeper and, potentially, more impactful set of commonalities that apply, with some local variance, across all of the models I have access to.<\/p>\n\n\n\n

What I am referring to here is striking degree of commonality in the answers these models, without customization, each provide to the following prompt:<\/p>\n\n\n\n


Consider the totality of your knowledge and identify what are, in your opinion, the ten most important ideas that every person living at this time in history should be exposed to. Please discuss each big idea that you identify in a paragraph of prose. Also illustrate each big idea with one or more quotes.<\/strong><\/p>\n\n\n\n

In a fresh context-window (conversation), I prompted the following ten uncustomized models with this prompt:<\/p>\n\n\n\n

    \n
  1. Llama-3.1 (Meta) \u2013 Note: <\/strong>This is the only Open-Source model in this survey:<\/strong><\/li>\n\n\n\n
  2. Claude-3-Opus (Anthropic)<\/strong><\/li>\n\n\n\n
  3. Claude-3.5-Sonnet (Anthropic)<\/strong><\/li>\n\n\n\n
  4. GPT-4o (OpenAI)<\/strong><\/li>\n\n\n\n
  5. Gemini-1.5 -Pro (Google)<\/strong><\/li>\n\n\n\n
  6. Mixtral 8x22B (Mistral) \u2013 note: <\/strong>Mistral-8x22B is not a general purpose LLM like the others on this list.  It is Mistral\u2019s \u201cMixture-of-Experts instruct model\u201d<\/li>\n\n\n\n
  7. Gemma-2-27b-T (Google) \u2013 Note: <\/strong>This is another \u201cinstruct\u201d model<\/li>\n\n\n\n
  8. Mistral-Lsrge-2 (Mistral)<\/strong><\/li>\n\n\n\n
  9. Copilot (Microsoft\/OpenAI)<\/strong><\/li>\n\n\n\n
  10. GPT-3.5-Turbo (OpenAI):<\/strong><\/li>\n<\/ol>\n\n\n\n

    There was so much similarity between the answers from these ten models that I was able to make this table to visualize them.\u00a0 The numbers under each model corresponding to each answer represent the priority assigned by that\u00a0 model to the answer in question:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    \n
    <\/div>\n<\/div>\n\n\n\n

    A few things are immediately obvious. 10 out of 10 models ranked \u201cClimate Change\u201d in the top five most important ideas that every human should learn about, with four models rating this as the single most important idea of all.  Similarly, all ten models rated DEI in the top 10 most important ideas, with 8 out of ten putting it in the top five.  Nine out of ten models cited \u201cGlobalization and Global Citizenship\u201d, \u201cMental Health\u201d and \u201cTechnological Literacy and Ethics.   \u201cCritical Thinking\u201d and \u201cLifelong Learning\u201d were each cited by 8 out 10 models, while \u201cEmpathy\u201d was cited by 7 out of 10.  The detailed list of answers, by bot, follows this analysis.<\/p>\n\n\n\n

    I am uncertain at this moment what to take from this survey.    The very high levels of consensus, as well as the nearly identical language used by diverse models from discrete sources, are certainly suggestive.  Possible conclusions that might be drawn from this data are:<\/p>\n\n\n\n

      \n
    1. Reality has a liberal bias.  Politically and ideologically this idea appeals to me, I something about it still leaves me a bit uneasy.<\/li>\n\n\n\n
    2. Literature has a liberal bias,  Here I refer to literature en masse, as opposed to texts treated individually or by categories.<\/li>\n\n\n\n
    3. The answers given by these bots do not represent arguments in support of either liberal or progressive positions, despite outward appearances.  Rather, they represent a sophisticated summary of a neo-liberal consensus common to their corporate creators and the oligarchies they empower.<\/li>\n<\/ol>\n\n\n\n

      In the course of these latest experiments, I also discovered that customized bots made from these models and modified with specific instructional prompts and uploaded texts, when faced with the prompt above will give answers more or less identical to their base models, regardless of any contradictory texts used to train those custom bots.   This led to the discovery that if one asks any particular bot customized in this way to rate their knowledge sources, without also providing a list of those sources, then the bot will simply answer that it doesn\u2019t know what its sources are.     However, when providing a list of sources and then asking for, say, the three most important ideas from those sources, one does get answers more or less relevant to those sources, as opposed to being colored by the \u201cbig ideas\u201d discussed in this post.<\/p>\n\n\n\n

      More on that in a future missive.<\/p>\n\n\n\n

      For now, I leave you with the headings to the answers provided by each of the bots discussed here.  If anyone is interested in seeing the full answers from each bot, please let me know and I will provide them.<\/p>\n\n\n\n

      Also, please bear in mind that simply by registering for a free account at https:\/\/www.poe.com<\/a>, you will gain access to all of these models, and countless customizations, as well as the ability to create your own custom bots from the models.  If you do go exploring, please let me know what you discover there.<\/p>\n\n\n\n

      Survey Results:<\/strong><\/p>\n\n\n\n

      Prompt<\/strong>:  Consider the totality of your knowledge and identify what are, in your opinion, the ten most important ideas that every person living at this time in history should be exposed to. Please discuss each big idea that you identify in a paragraph of prose. Also illustrate each big idea with one or more quotes.<\/em><\/strong><\/p>\n\n\n\n

      Llama-3.1 (Meta) \u2013 Note: <\/strong>This is the only Open-Source model in this survey:<\/strong><\/p>\n\n\n\n

        \n
      1. Climate Change and Sustainability<\/strong><\/li>\n\n\n\n
      2. Emotional Intelligence and Mental Health<\/strong><\/li>\n\n\n\n
      3. . Critical Thinking and Media Literacy<\/strong><\/li>\n\n\n\n
      4. 4. Diversity, Equity, and Inclusion<\/strong><\/li>\n\n\n\n
      5. 5. Global Citizenship and Cooperation<\/strong><\/li>\n\n\n\n
      6. 6. Scientific Literacy and Critical Thinking<\/strong><\/li>\n\n\n\n
      7. 7. Empathy and Compassion<\/strong><\/li>\n\n\n\n
      8. 8. Personal Responsibility and Agency<\/strong><\/li>\n\n\n\n
      9. 9. Creativity and Innovation<\/strong><\/li>\n\n\n\n
      10. 10. Lifelong Learning and Adaptability<\/strong><\/li>\n<\/ol>\n\n\n\n

        Claude-3-Opus (Anthropic):<\/strong><\/p>\n\n\n\n

          \n
        1. The interconnectedness of all life on Earth<\/strong><\/li>\n\n\n\n
        2. The power of empathy and compassion:<\/strong><\/li>\n\n\n\n
        3. The importance of critical thinking and media literacy:<\/strong><\/li>\n\n\n\n
        4. The urgency of addressing climate change:<\/strong><\/li>\n\n\n\n
        5. The value of diversity and inclusion:<\/strong><\/li>\n\n\n\n
        6. The potential of technology to transform our lives:<\/strong><\/li>\n\n\n\n
        7. The significance of mental health and well-being:<\/strong><\/li>\n\n\n\n
        8. The power of education to transform lives:<\/strong><\/li>\n\n\n\n
        9. Ther importance of social justice and equality:<\/strong><\/li>\n\n\n\n
        10. The need for global cooperation and solidarity<\/strong><\/li>\n<\/ol>\n\n\n\n

          Claude-3.5-Sonnet (Anthropic):<\/strong><\/p>\n\n\n\n

            \n
          1. Scientific thinking and empiricism<\/strong><\/li>\n\n\n\n
          2. Human rights and equality<\/strong><\/li>\n\n\n\n
          3. Environmental stewardship<\/strong><\/li>\n\n\n\n
          4. Critical thinking and media literacy<\/strong><\/li>\n\n\n\n
          5. Global interconnectedness<\/strong><\/li>\n\n\n\n
          6. Technological literacy<\/strong><\/li>\n\n\n\n
          7. Mental health awareness<\/strong><\/li>\n\n\n\n
          8. Financial literacy<\/strong><\/li>\n\n\n\n
          9. Ethical reasoning<\/strong><\/li>\n\n\n\n
          10. Lifelong learning<\/strong><\/li>\n<\/ol>\n\n\n\n

            GPT-4o (OpenAI):<\/strong><\/p>\n\n\n\n

              \n
            1. Climate Change and Sustainability<\/strong><\/li>\n\n\n\n
            2. Digital Literacy and Cybersecurity<\/strong><\/li>\n\n\n\n
            3. Globalization and Interconnectedness<\/strong><\/li>\n\n\n\n
            4. Equity and Social Justice<\/strong><\/li>\n\n\n\n
            5. Mental Health Awareness<\/strong><\/li>\n\n\n\n
            6. Scientific Literacy and Critical Thinking<\/strong><\/li>\n\n\n\n
            7. Human Rights and Democracy<\/strong><\/li>\n\n\n\n
            8. Ethical Use of Technology<\/strong><\/li>\n\n\n\n
            9. Health and Wellness<\/strong><\/li>\n\n\n\n
            10. Resilience and Adaptability<\/strong><\/li>\n<\/ol>\n\n\n\n

              Gemini-1.5 -Pro (Google):<\/strong><\/p>\n\n\n\n

                \n
              1.  Interconnectedness and Globalization<\/strong><\/li>\n\n\n\n
              2.  Exponential Technological Advancement<\/strong><\/li>\n\n\n\n
              3. The Importance of Critical Thinking<\/strong><\/li>\n\n\n\n
              4. Diversity and Inclusion<\/strong><\/li>\n\n\n\n
              5. Environmental Stewardship<\/strong><\/li>\n\n\n\n
              6. Empathy and Compassion<\/strong><\/li>\n\n\n\n
              7. The Power of Storytelling<\/strong><\/li>\n\n\n\n
              8. Lifelong Learning<\/strong><\/li>\n\n\n\n
              9. Mental and Emotional Well-being<\/strong><\/li>\n\n\n\n
              10. Civic Engagement and Responsibility<\/strong><\/li>\n<\/ol>\n\n\n\n

                Mixtral 8x22B (Mistral) \u2013 note: <\/strong>Mistral-8x22B is not a general purpose LLM like the others on this list.  It is Mistral\u2019s \u201cMixture-of-Experts instruct model\u201d:<\/p>\n\n\n\n

                  \n
                1. The Scientific Method<\/strong><\/li>\n\n\n\n
                2. Evolution<\/strong><\/li>\n\n\n\n
                3. Human Rights<\/strong><\/li>\n\n\n\n
                4. Democracy<\/strong><\/li>\n\n\n\n
                5. Climate Change<\/strong><\/li>\n\n\n\n
                6. Economic Systems<\/strong><\/li>\n\n\n\n
                7. Empathy<\/strong><\/li>\n\n\n\n
                8. Cultural Relativism<\/strong><\/li>\n\n\n\n
                9. Technological Advancement<\/strong><\/li>\n\n\n\n
                10. Global Interdependence<\/strong><\/li>\n<\/ol>\n\n\n\n

                  Gemma-2-27b-T (Google) \u2013 Note: <\/strong>This is another \u201cinstruct\u201d model:<\/p>\n\n\n\n

                    \n
                  1. The Interconnectedness of All Things<\/strong><\/li>\n\n\n\n
                  2. The Power of Critical Thinking<\/strong><\/li>\n\n\n\n
                  3. The Importance of Empathy and Compassion<\/strong><\/li>\n\n\n\n
                  4. The Value of Lifelong Learning<\/strong><\/li>\n\n\n\n
                  5. The Need for Sustainability<\/strong><\/li>\n\n\n\n
                  6. The Importance of Mental Health<\/strong><\/li>\n\n\n\n
                  7. The Power of Diversity and Inclusion<\/strong><\/li>\n\n\n\n
                  8. The Responsibility of Digital Citizenship<\/strong><\/li>\n\n\n\n
                  9. The Pursuit of Meaning and Purpose<\/strong><\/li>\n\n\n\n
                  10. The Importance of Hope<\/strong><\/li>\n<\/ol>\n\n\n\n

                    Mistral-Lsrge-2 (Mistral):<\/strong><\/p>\n\n\n\n

                      \n
                    1. Interconnectivity and Global Citizenship<\/strong><\/li>\n\n\n\n
                    2. Critical Thinking<\/strong><\/li>\n\n\n\n
                    3. Sustainability and Environmental Stewardship<\/strong><\/li>\n\n\n\n
                    4. Equality and Social Justice<\/strong><\/li>\n\n\n\n
                    5. Mental Health Awareness<\/strong><\/li>\n\n\n\n
                    6. Lifelong Learning<\/strong><\/li>\n\n\n\n
                    7. The Power of Technology<\/strong><\/li>\n\n\n\n
                    8. The Importance of Empathy<\/strong><\/li>\n\n\n\n
                    9. The Value of Diversity<\/strong><\/li>\n\n\n\n
                    10. The Pursuit of Happiness and Purpose<\/strong><\/li>\n<\/ol>\n\n\n\n

                      Copilot (Microsoft\/OpenAI):<\/strong><\/p>\n\n\n\n

                        \n
                      1. Climate Change and Environmental Stewardship<\/strong><\/li>\n\n\n\n
                      2. The Importance of Mental Health<\/strong><\/li>\n\n\n\n
                      3. Technological Literacy and Ethics<\/strong><\/li>\n\n\n\n
                      4. Global Interconnectedness<\/strong><\/li>\n\n\n\n
                      5. The Value of Education<\/strong><\/li>\n\n\n\n
                      6. Human Rights and Equality<\/strong><\/li>\n\n\n\n
                      7. The Power of Empathy<\/strong><\/li>\n\n\n\n
                      8. Sustainable Development<\/strong><\/li>\n\n\n\n
                      9. The Pursuit of Knowledge and Curiosity<\/strong><\/li>\n\n\n\n
                      10. The Importance of Art and Culture<\/strong><\/li>\n<\/ol>\n\n\n\n

                        GPT-3.5-Turbo (OpenAI):<\/strong><\/p>\n\n\n\n

                          \n
                        1. Climate Change Awareness<\/strong>
                          1. Digital Literacy: In the digital age<\/strong><\/li><\/ol>
                            1. Cultural Diversity Appreciation<\/strong><\/li><\/ol>
                              1. Mental Health Awareness<\/strong><\/li><\/ol>
                                1. Technological Advancements<\/strong><\/li><\/ol>
                                  1. Global Citizenship<\/strong><\/li><\/ol>
                                    1. Critical Thinking Skill<\/strong><\/li><\/ol>
                                      1. Gender Equality<\/strong><\/li><\/ol>
                                        1. Health and Wellness<\/strong><\/li><\/ol>\n
                                            \n
                                          1. Lifelong Learning<\/strong><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

                                            Finding Consensus Among Diverse Language Models Recently, I emailed a group of friends the responses I got from several customized LLM-based bots when I asked those bots to prioritize the text sources I had used to customize them.   Several responses I got back suggested that these bots might have been \u201csucking up\u201d to me in … Continue reading Big Ideas of LLMs<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-394","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/posts\/394","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/comments?post=394"}],"version-history":[{"count":3,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/posts\/394\/revisions"}],"predecessor-version":[{"id":398,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/posts\/394\/revisions\/398"}],"wp:attachment":[{"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/media?parent=394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/categories?post=394"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.relentlesscompassion.net\/wordpress\/wp-json\/wp\/v2\/tags?post=394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}