Entropy for Information Systems

Posted on 30th August 2010 by ideonexus in Geeking Out,Ionian Enchantment

Entropy is a fairly easy concept to define, the measure of disorder in a closed system, and a rather difficult concept to grasp, but one that furnishes us with wonderful insights into the way the world around us operates. The amount of entropy in the Universe is ever-increasing, the energy concentrated in our sun is constantly radiating away in light and heat, dissipating into an unusable state, absolute undifferentiation.

Sunflower
Sunflower
Credit: riandreu

Living things form “pockets of resistance” to the force of entropy. They do this through syntropy, or negentropy, which is the entropy we export to reduce our internal entropy; in other words, it’s the waste energy we generate to keep our soma in an organized working state. We collect the sun’s waste energy and use it to organize ourselves through syntropy.

How Much Information Entropy?
How Much Information Entropy?
Credit: Moi

In Information Systems, entropy, known as Shannon entropy for Claude Shannon, is the measure of uncertainty in a random variable. A coin toss has one bit of entropy for the 50/50 chance of it turning up heads or tails, 0 or 1. A six-sided dice carries three bits of entropy for the possible outcomes it may produce with each roll (1 (000), 2 (001), 3 (010), 4 (011), 5 (100), 6 (101)). The weather has an amount of entropy difficult to quantify, but it varies from location to location. The weather in New York has more entropy than the weather in Southern California because Southern California has a more consistent climate. Similarly, in our first example, if we were dealing with a rigged coin, one that turned up heads more often than tails, then there would be less than one bit of entropy in each coin toss because we would expect heads more frequently than tails.


On the face of it, the only thing in common with the thermodynamic and information theory definitions is that entropy is a measure of disorder, but the two are analogous in other ways. In our thermodynamic Universe things move toward a state of increasing entropy, and a similar tendency towards a state of total uniformity occurs in an information system, only in reverse.

A living organism in an information system starts out in a world of absolute entropy, nothing is known. As that life interacts with its Universe, the amount of entropy in the Universe decreases for that being, and the amount of its internal entropy increases as what it knows becomes more of a variable to its peers. As the beings living in an information system decrease the entropy of their universe, it tends toward a state of absolute syntropy, absolute predictability.

We exist in a thermodynamic system, and it powers the information systems in our brains, the information systems we construct, and the information system these combine to form in our civilization. The increase of syntropy in an information system comes at the cost of an increase of entropy in the thermodynamic system powering it. Our thermodynamic system is winding down, a bad thing for us, but our information system is becoming increasingly more sophisticated, more syntropic. This brings a deeper insight to H.G. Wells’ prescient observation: “Human history becomes more and more a race between education and catastrophe.”


Notes:

  • With this understanding of Information Entropy, apply this deeper understanding to the Monty Hall Problem (and the interactive demo of information entropy in effect.
  • Playing with a deck of cards is also a fun way to think about information entropy. What’s the measure of entropy in a 52-card deck? What’s the entropy of just the suits?
  • Simulations and bioinformatics are giving us increasing syntropic power over previously chaotic (read “highly entropic”) systems. Chaos theory could just as well be “Information Entropy Theory.”
  • A Letter to Roger Ebert Concerning a Misconception About Evolution

    Posted on 10th September 2009 by ideonexus in Ionian Enchantment

    Dear Roger Ebert,

    In your recent review of “Extract,” you made the comment about the film “Idiocracy” that “those Idiots had the benefit of a few hundred years during which to refute Darwin by evolving less intelligence.” I know that you are a man who appreciates science, and thought you should know that your statement reflects a common misunderstanding of evolution through natural selection: that species are always evolving to better, more advanced states. In fact, there are many examples of animals “de-evolving” to previous states, such as whales, which are descendent of land mammals, but gave up their legs and returned to the seas. Many species of whale still have the remnants of tiny hip bones floating deep inside them and the remnants of finger bones inside their fins.

    Vestigial Hip Bones in Whales
    Vestigial Hip Bones in Whales
    Credit: Moi

    The same is true of intelligence. Our big brains have conferred a magnificent survival advantage on us, but they come at a huge cost in energy to fuel them. The Indonesian “hobbit” fossils, homo floresiensis, discovered in recent years tell the story of a species of human that, once geographically isolated on an island with limited resources, adapted by shrinking in stature, including atrophying of the brain. The sea squirt is born with a brain, which it uses to navigate the world until it finds a suitable spot to plant itself, at which point it promptly .

    I believe the evidence, in the form of increasing average IQs and other test measures, shows that we are growing more intelligent as a species. The sophisticated cultural environment we have constructed makes increasing demands on our average intelligence in order to survive and be successful; however, the sea squirt and homo floresiensis are cautionary tales that we must remain ever-vigilant. As the evolutionary biologist J. B. S. Haldane said, “The ancestors of oysters and barnacles had heads. Snakes have lost their limbs and ostriches and penguins their power of flight. Man may just as easily lose his intelligence.”

    Sincerely,

    Ryan Somma

    http://ideonexus.com

    PS – Thank you so much for your recent blog-post about trivia. You so eloquently expressed ideas I have been trying to articulate for years on the subject.

    Understanding the Animal Side of Human Nature

    Posted on 16th July 2009 by ideonexus in Ionian Enchantment

    Grrrrr!

    Grrrrr!
    Credit: Brian Scott

    Colin Powell’s appearance on State of the Union recently stirred up many healthy debates on Iraq, Obama’s presidency, and Sotomayor, but I found his thoughts on American families most interesting, especially the following:

    And I’m kind of a simple guy on things like this, John. I watch National Geographic and Animal Planet, and I love to watch lion shows or tiger shows, where a cub is born. And there is the mother and the father.

    The father may be away at a distance, but he’s providing protection for the family. And the family unit knows exactly how much a cub is able to do at what age. And until you‚re 4 months old, you never leave the mom. And then when you’re 6 months old, you can go out a little way, but you’ll get smacked back if you ever exceed the limits of which you’re capable of managing.

    Are we the only mammal who thinks we don’t have to follow these rules? That we don’t have to pass on a thousand previous generations of experience? That’s not acceptable.

    Powell is a powerfully persuasive speaker with a great deal of integrity. Earlier he had mentioned children’s need to belong to a group, be it a family or a gang. I admire his comparing human families to a lion pride and specifically referring to us as “mammals,” who are subject to the same basic needs and instincts.

    It’s dangerous when humans try to distinguish ourselves from the animal kingdom, as somehow above it. Philosophers and theologians have spent millennia trying to find a solid argument for why we are not animals, but this line of thought abandons all we may learn by studying our animal nature. Powell has deep insights into what motivates us by watching Animal Planet, and we have much to learn about ourselves by observing our primate relatives on the evolutionary tree.

    Powell’s example of the lion pride resembles hierarchies found in ape and monkey societies. In 1925, the London Zoo put together a baboon exhibit, bringing 99 male baboons and 36 females into an enclosure where each had about 60 square feet of personal space. Six years later, 35 of the males and five of the females were still alive. The remaining females were removed for their own safety, as fighting among the male baboons had killed off much of the zoo’s population.

    The incident was seen as reinforcing the idea of animal savagery, that such violence was in the baboons’ nature and humans were above such baser instincts, but if such behavior were to occur in the wild, baboons would quickly be driven to extinction. Eventually, a more rational explanation came to light, as Carl Sagan and Ann Druyan elucidate:

    What had gone wrong on Monkey Hill? First, almost all of the baboons introduced to the “colony” were unknown to one another. There was no long-term mutual habituation, no prior establishment of dominance hierarchies, no common understanding in these harem-obsessed males of who was to have many females and who none at all. No kinship-based female dominance hierarchy had been established.

    Baboons in the wild are born into a cultural context, where there are adults with social relationships already worked out. The London Zoo’s baboons were all thrown in together at random, stripped of their social network, and forced to work out a completely new social hierarchy on the spot, resulting in unconscionable acts of mortal violence. As with Powell’s lion pride example, baboon children are part of a culture, a social network and they learn their place in it, the way it works, and how they may rise in status within it.


    Piled baboons

    Piled baboons
    Credit: Tambako the Jaguar

    There are examples of patriarchal and matriarchal social hierarchies in primate societies, each conferring the advantage of stability on the whole. Rather than look down on the animal kingdom as something meant for subjugation, we can see the importance of family and community for providing social stability from which we all benefit.

    Fun With Tick Clockwork

    Posted on 9th July 2009 by ideonexus in Ionian Enchantment

    Dermacentor variabilis, female

    Dermacentor variabilis, female
    Credit: National Tick Collection

    Vicky and I went for a short hike in Chesapeake’s Northwest River Park last weekend, a lovely site filled with marshland and waterways for canoeing, camping, … and ticks. Hot summer days combined with the humidity of the wetlands climate equals lots and lots of ticks, and this hike was no exception.

    Except for Vicky. While I had to stop every so many hundred yards to scan my legs for the little bloodsuckers, of which I was literally finding dozens, Vicky found maybe four on her the whole trip. What gives?

    Carl Sagan gives a brief description of the life of a tick in his book Shadows of Forgotten Ancestors, and the small set of instinctual rules that command its life. Lacking eyes, ticks must find each other to mate by detecting the pheromones C6H3 OHCl2. Then:

    After mating, the female climbs up a bush or shrub and out onto a twig or leaf. How does she know which way is up? Her skin can sense the direction from which light is coming, even if she cannot generate an optical image of her surroundings. Poised out on the leaf or twig, exposed to the elements, she waits. Conception has not yet occurred. The sperm cells within her are neatly encapsulated; they’ve been put in long-term storage. She may wait for months or even years without eating. She is very patient.

    Makes the tick sound like a tiny clockwork machine, doesn’t it? It’s possible to have some fun with this instinctual behavior too. As when Vicky and I put a blood-engorged tick in a sealed vial, where it soon laid a bazillion eggs, which hatched a few months later.


    Baby Ticks

    Baby Ticks
    Credit: Vicky

    The baby ticks all climbed to the top of the vial. When the vial was turned over, they all climbed to the highest point again. Over and over again, until they eventually stopped moving (Don’t tell PETA). It was like some twisted version of an hourglass, appropriate for some Tim Burton film. Try this sometime, it makes a great conversational piece when you have guests over that you’re not very fond of.

    Eventually, the right stimulus comes along the forest path, triggering the tick to drop, hopefully (for her), onto something full of blood:

    What she’s waiting for is a smell, a whiff of another specific molecule, perhaps butyric acid, which can be written C2H7COOH. Many mammals, including humans, give off butyric acid from their skin and sexual parts. A small cloud of the stuff follows them around like cheap perfume. It’s a sex attractant for mammals. But ticks use it to find food for prosepctive mothers.

    Here’s the clue as to why I was getting bum-rushed with blood-sucking arachnids, while Vicky was passing through the forest virtually untouched, butyric acid. Vicky had showered that morning and put on fresh clothes, while I figured I’d shower after the hike, and put on my workout clothes from the day before. I even wore the same icky socks. Vicky was virtually clean of butyric acid, while I was fairly drenched in the stuff.

    So while I like to joke that the ticks preferred me because I was sweeter, in reality, they wanted me because I was stinkier! (Why am I sharing this with you?)

    In fact, this basic instinctual set of commands can cause the arachnid to exhibit some buggy (in a software-metaphor sense of the word) behavior, as with the stimulus to trigger her blood-drinking response:

    It’s not the taste of the blood that attracts her, but the warmth. If she drops onto a butyric acid-scented toy balloon filled with warm water, she will readily puncture it and, an inept Dracula, gorge herself on tap water.

    I think I have plans for some future fun with stinky socks and warm-water filled balloons in my future. : )

    Creative Commons License