# Tag Info

6

With some algebraic manipulation (as pointed out in @orlp's answer), we can deduce the following: $$f(x) = x \tanh(\log(1+e^x)) \tag{1}$$ $$= x\frac{(1+e^x)^2 - 1}{(1+e^x)^2 + 1} = x\frac{e^{2x} + 2e^x}{e^{2x} + 2e^x + 2}\tag{2}$$ $$= x - \frac{2x}{(1 + e^x)^2 + 1} \tag{3}$$ Expression $(3)$ works great when $x$ is negative with very little loss of ...

4

OP points to a particular implementation of the mish activation function for accuracy specifications, so I had to characterize this first. That implementation uses single precision (float), and is stable and accurate in the positive half-plane. In the negative half-plane, because it uses logf instead of log1pf, relative error quickly grows a $x\to-\infty$. ...

3

There's no need to perform the logarithm. If you let $p = 1+\exp(x)$ then we have $f(x) = x\cdot\dfrac{p^2-1}{p^2+1}$ or alternatively $f(x) = x - \dfrac{2x}{p^2+1}$.

2

The context here is computer vision and the activation function for training neural nets. Chances are this code is going to be executed on a GPU. While performance is going to depend on the distribution of typical inputs, generally speaking it is important to avoid branches in GPU code. Warp divergence can significantly degrade performance of your code. ...

2

My impression is that someone wanted to multiply x by a function f(x) that goes smoothly from 0 to 1, and experimented until they found an expression using elementary functions that did this, with no mathematical reason behind the choice of functions. After choosing a parameter t, let $p_t(x) = 1/2 + (3 / 4t)x - x^3 / (4t^3)$, then $p_t(0) = 1/2$, $p_t(t) =... 2 Many people can afford to buy a computer with a powerful processor and say 128GB of RAM. Maybe the same cost as a small car. That's enough to store one number with 300 billion digits. It's enough to store 300,000 numbers of a million digits. But it is also enough to store 16 billion double precision floating point numbers. There are problems where 300,000 ... 2 I'll give you a simple example. Suppose you have some floating-point numbers to add together. We'll assume they're all non-negative so that cancellation isn't an issue. For the purpose of this example, I'm going to use decimal with four significant digits of precision just to illustrate the point. The numbers are: 1.000e-4 1.000e-4 1.000e-4 1.000e-4 1.... 2 Although, the question is a bit old, but it may help people coming here for similar question. A vital detail was missed out in the article that you referred to and it is that the standard chose to interpret an all 0s exponent to be equivalent to '-126' and not '-127'. One place, where I found a nice explanation (and an explicit statement about this truth) ... 1 You are not doing arithmetic on arbitrary infinitely precise numbers. You are doing arithmetic on the subset of number representable in your computer's native floating point format. Moreover, the values you see printed out are not infinitely precise representations of the actual values encoded in the computer. They are decimal numbers (of a maximum precision)... 1 There is one method to compare floating point numbers for equality, which is both very simple and correct: You use the equality (==) operator. There is another method to compare whether floating point numbers are identical, which unlike the equality operator would find that -0 and +0 are not the same, and that NaNs with the same bit pattern are the same: Use ... 1 Take x=1+u, y=1+2u, z=2+4u, where u is the value of the lowest mantissa bit. x+y = 2+3u gets rounded up to 2+4u, so x+y<z is false. z-y = 2+4u - (1+2u) = 1+2u, so x < z-y is true. That’s a counter example for the first case. With x=1+3u you get a counter example for the second case. 1 Consider the case that x and y hold the largest representable value for the current type, and z is positive infinity. By the rules of floating point arithmetics, x + y is now infinity, which isn't less than infinity. On the other hand, z - y is also infinity, which the largest representable value is less than. Another example would be when x is very small ... 1 First of all, the IEEE 745 16-bit format has: 1 bit for the sign. 5 bits (not 4) for the biased exponent. The bias is$15$. 10 bits for the mantissa. The leading$1$is implicit (not explicit). The calculation is incorrect since$(29.375)_2 = (11101.011)_2 = (1.1101011)_2 \cdot 2^4$(and not$2^3$). Then: The sign bit is$1$because the number is ... 1 (Disclaimer: Sorry this answer got a bit long, but I had the same problem and also didn't find an easy explanation on IEEE754 on the internet so I'm posting this. This is answering the question of the OP but also a bit more than that.) I'll explain this with an example. If you consider these three numbers and their binary values: 0.06 = 0.000011110101110...... 1 It is 0.100000, because normalised values are only allowed to start with 01 or 10. The exponent is 1000 (-8 in decimal) because that is the smallest possible value that can be represented using two's compliment in 4 bits. The value of the exponent means that the floating point will be moved 8 values to the left, thus making the result smaller. 1 Both encodings for the exponent are possible, and they are called Excess-$2^{n-1}$and Excess-$2^{n-1}-1$- in particular, you might meet such number encodings as Excess-127, Excess-128, Excess-1023 and so on. The IEEE-754 standard happened to choose the Excess-127 and the Excess-1023 encodings for exponents, so these encodings became more customary. Your ... 1 I don't know how old your book is, but some architectures (I know of the early SPARCs, there may be others) only had 32 bit floating point registers. Double precision instructions used pairs of registers for storage. This became much less common on general-purpose hardware when 64-bit integer registers became the norm. If anything, we have the converse ... 1 After reading further, I found the answer. The answer is that the$x$given in the question is not in$\mathbb{R}(5, 3)$. The way I figured this out is as follows: On the next page, the author states that $$\min_{x \in \mathbb{R}(t, s)} |x| = 2^{-2^s}$$ This is consistent with$(.100 \cdots 0) \cdot2^{-(11\cdots 1)}$being the least positive number in$\...

1

There are a couple of different things happening in that question. Is it due to the limit of binary system to express floating-point numbers? Loss of precision isn't due to the use of binary, it is due to keeping the storage size constant. It also happens if you work with, say, 8-digit decimal numbers, also if you do it with pen and paper. Eventually you ...

1

I'm going to start with this famous quote from James Wilkinson's 1970 Turing Award Lecture, Some Comments from a Numerical Analyst. In the early days of the computer revolution computer designers and numerical analysts worked closely together and indeed were often the same people. Now there is a regrettable tendency for numerical analysts to opt out ...

1

There is more than one correct approach to convert the number. The former is a correct algorithm for the problem. This kind of base conversion would usually be implemented in a library for the programming language you are using. There might be many such implementations out there, and I don't know what all of them are doing. I would not recommend the ...

1

For example, C and C++ don’t adhere to it. They don’t define the details of floating point arithmetic. An implementation may adhere to that standard, and there is a standard way to inform the user about it. I used one compiler with ieee compliant 32 and 64 bit floating point, and totally non-compliant 128 bit fp with roughly 105 bit mantissa.

1

There is a huge discrepancy in your notes. Either you assume that you are using base 2, in which case there should be no beta introduced, but just the number 2 used. Or you don't assume that you are using base 2, but for example base 10, then the number before the decimal point can't be forced to be 1 (or you could only represent 1 ≤ x < 2, 10 ≤ x < 20,...

1

There is definitely an inconsistency in what you've written here. My guess is that $t$ doesn't mean what you think it means. You are using it here to describe the size of the significand field, that is, the number of digits after the radix point. (It's not a decimal point in general!) My guess is that your lecturer is using it to refer to the precision. The ...

1

I don't know exactly where you're trying to get to with this, but I think you probably want to Taylor expand the division: $$\frac{1}{1 - \delta_2} = 1 + \delta_2 + \delta_2^2 + \delta_2^3 + \cdots$$ Then, for example, if $\epsilon$ is the machine epsilon and $\delta_2 > 0$: $$1 + \delta_2 \le \frac{1}{1 - \delta_2} \le 1 + \delta_2 + \epsilon$$ That ...

Only top voted, non community-wiki answers of a minimum length are eligible