MCB128: AI in Molecular Biology (Spring 2026)

(Under construction)

(Now hiring TFs Fall 2026)

• Motivation for the logistic function
  • The noisy LED display
  • Solution

block 0 :

Motivation for the logistic function

The logistic function appears in problems where there is a binary decision to make. Here you will workout a problem (based on MacKays’s exercise 39.5) that like a binary neuron, also uses a logistic function.

The noisy LED display

In a LED display each number corresponds to a pattern of on(+1) or off(-1) for the 7 different elements that compose the display. For instance, the patterns for numbers 2 and 3 are:

\[\mathbf{c_2} = (+1,-1,+1,+1,+1,-1,+1)\] \[\mathbf{c_3} = (+1,-1,+1,+1,-1,+1,+1)\]

Imagine you have a LED display that is not working properly. This defective LED is such that, for a given number the LED wants to display:

• Each Element adopts their correct state with probability \(1-f\).

• Each Element adopts the incorrect state with probability \(f\),

The LED is allowed to display ONLY a number “2” or a number “3”. And it does so by emitting a patter \(\mathbf{p}=(p1,p2,p3,p4,p5,p6,p7)\), where \(p_i = +1, -1\)

Calculate the posterior probability that the intended number was a “2”, given the pattern \(\mathbf{p}\) you observe in the LED, that is,

\[P(n=2\mid \mathbf{p}).\]

Show that you can express that posterior probability as a logistic function,

\[P(n=2\mid \mathbf{p}) = \frac{1}{1+e^{-(\mathbf{w}\mathbf{p} + w_0)}}\]

for some weights \(\mathbf{w}\), and some constant \(\theta\).

You can assume that the prior probabilities for either number, \(P_2\) and \(P_3\), are given.

Hint: \(x^y = e^{y\log x}\) for any two real numbers \(x, y\).

Solution

The probability that we can calculate is \(P(\mathbf{p}\mid 2)\), that is the probability that observing a particular pattern \(\mathbf{p}\), given that the LED tried to emit a “2”,

\[P(\mathbf{p}\mid 2) = (1-f)^{C}\, f^{I},\]

where \(C\) is the number of correct elements and \(I\) is the number of incorrect elements.

Using the relations

\[\begin{aligned} 7 &= C + I\\ \mathbf{c_2}\cdot\mathbf{p} &= C - I\\ \end{aligned}\]

We can write

\[P(\mathbf{p}\mid 2) = (1-f)^{\frac{1}{2}(7+\mathbf{c_2}\cdot\mathbf{p})}\, f^{\frac{1}{2}(7-\mathbf{c_2}\cdot\mathbf{p})},\]

Then using the hint above, we can rewrite,

\[\begin{aligned} P(\mathbf{p}\mid 2) &= e^{ \frac{1}{2}(7+\mathbf{c_2}\cdot\mathbf{p})\,\log(1-f) + \frac{1}{2}(7-\mathbf{c_2}\cdot\mathbf{p})\,\log(f)}\\ &= e^{ \frac{7}{2}\log f(1-f) + \frac{1}{2} \log(\frac{1-f}{f}) \mathbf{c_2}\cdot\mathbf{p}}\\ \end{aligned}\]

The quantity we have been asked to calculate is not \(P(\mathbf{p}\mid 2)\), but instead, given that we have seen a pattern \(\mathbf{p}\), what is the probability that the pattern was generated with a “2” in mind. That is the posterior probability \(P(2\mid \mathbf{p})\), which using Bayes theorem is given as a function of \(P(\mathbf{p}\mid 2)\) as

\[P(2\mid \mathbf{p}) = \frac{P(\mathbf{p}\mid 2) P(2)}{P(\mathbf{p})},\]

where \(P(2)\) is a prior probability.

In the general case in which the LED can produce any of the 10 digits (from 0 to 9), then we have by marginalization

\[\begin{aligned} P(\mathbf{p}) &= P(\mathbf{p}\mid 0) P(0) + P(\mathbf{p}\mid 1) P(1) + P(\mathbf{p}\mid 2) P(2) + \ldots + P(\mathbf{p}\mid 9) P(9).\\ \end{aligned}\]

Resulting in the general solution,

\[P(2\mid \mathbf{p}) = \frac{e^{\mathbf{a_2}\,\mathbf{p}}\, P(2)}{\sum_{n=0}^{9} e^{\mathbf{a_n}\,\mathbf{p}}\, P(n)},\]

with

\[a_n = \frac{1}{2} \log{\frac{1-f}{f}} \mathbf{c_n}\mathbf{p}.\]

Notice that, the normalization condition is \(\sum_{n=0}^9 P(n\mid \mathbf{p}) = 1\).

For our particular problem, where we want to distinguish only between the pattern being generated by a “2” or a “3”, that results in

\[P(2\mid \mathbf{p}) = \frac{e^{\mathbf{a_2}\,\mathbf{p}}\, P(2)}{e^{\mathbf{a_2}\,\mathbf{p}}\, P(2) + e^{\mathbf{a_3}\,\mathbf{p}}\, P(3)},\]

where here the normalization condition is

\[P(2\mid \mathbf{p}) + P(3\mid \mathbf{p}) = 1.\]

The posterior probability \(P(2\mid \mathbf{p})\) can be re-written as

\[P(2\mid \mathbf{p}) = \frac{1}{1 + e^{-\left[\mathbf{a_2}-\mathbf{a_3}\right]\,\mathbf{p}}\, \frac{P(3)}{P(2)}}.\]

We can define the weights

\[\mathbf{w} = \mathbf{a_2}-\mathbf{a_3}.\]

We can also parameterize the priors as

\[\frac{P(3)}{P(2)}=e^{-w_0}.\]

For instance, \(w_0 = 0\) for \(P(2) = P(3) = 1/2\).

Then, we have the expression we wanted to obtain of \(P(2\mid \mathbf{p})\) as a logistic linear function

\[P(2\mid \mathbf{p}) = \frac{1}{1 - e^{-(\mathbf{w}\,\mathbf{p} + w_0)}},\]

with weights,

\[\begin{aligned} \mathbf{w} &= \mathbf{a_2} - \mathbf{a_3} \\ &=\frac{1}{2}\,\log \frac{1-f}{f}\left[\mathbf{c_2} - \mathbf{c_3}\right]\\ \end{aligned}\]

or

\[\mathbf{w} = \log \frac{1-f}{f}\left(0, 0, 0, 0, +1, -1, 0\right).\]

Here is the code associated to this Section.