Formulas¶
$N(w; h)$ - number of ways to tile $w\times h$ rectangle (including symmetric solutions)
$T(w; h) = \begin{cases} 1, & N(w; h) \geq 1 \\ 0, & \text{else} \end{cases}$ - tileability function, $1$ if tiles rectangle, $0$ otherwise
$A(w; h) = \left(N(w; h)\right)^{\frac{1}{wh}}$ - average number of ways to tile cell in $w\times h$ rectangle (including symmetric solutions)
$G(T; x; y) = \sum_{w=1}^{\infty}\sum _{h=1}^{\infty}T(w; h)x^wy^h$ - bivariate generating function of $T(w; h)$
$G(A; x; y) = \sum_{w=1}^{\infty}\sum _{h=1}^{\infty}A(w; h)x^wy^h$ - bivariate generating function of $A(w; h)$
$N(1; n) = T(1; n) = 0, \qquad n \geq 1 \tag{1}$
$N(2; n) = T(2; n) = 0, \qquad n \geq 1 \tag{2}$
$N(3; n) = 2 \times N(3; n - 4), \qquad n \geq 5 \tag{3}$
$N(4; n) = 2 \times N(4; n - 3), \qquad n \geq 4 \tag{4}$
$N(5; n) = T(5; n) = 0, \qquad n \geq 1 \tag{5}$
$N(6; n) = 4 \times N(6; n - 4), \qquad n \geq 5 \tag{6}$
$N(7; n) = 256 \times N(7; n - 12), \qquad n \geq 13 \tag{7}$
$N(8; n) = 4 \times N(8; n - 3), \qquad n \geq 4 \tag{8}$
$N(9; n) = 8 \times N(9; n - 4), \qquad n \geq 5 \tag{9}$
$N(10; n) = 3072 \times N(10; n - 12), \qquad n \geq 13 \tag{10}$
$N(11; n) = 6144 \times N(11; n - 12), \qquad n \geq 13 \tag{11}$
$N(12; n) = 8 \times N(12; n - 3) + 16 \times N(12; n - 4), \qquad n \geq 5 \tag{12}$
$N(13; n) = 32768 \times N(13; n - 12), \qquad n \geq 13 \tag{13}$
$N(n; m) = T(n; m) = 0, \qquad 3\nmid n,3\nmid m,4\nmid n,4\nmid m \tag{14}$
First, note that J hexominoes always come in following pairs, because there is only two ways to fill concave area:
Second, assume J hexomino tiles $n\times m$ rectangles for $3\nmid n,3\nmid m,4\nmid n,4\nmid m$.
Place numbers in rectangles' cells according to function $F(x,y)\equiv 1+12\left(\left\lfloor\frac{x}{4}\right\rfloor - \left\lfloor\frac{x-1}{4}\right\rfloor + \left\lfloor\frac{y}{4}\right\rfloor - \left\lfloor\frac{y-1}{4}\right\rfloor\right)\pmod{24}$, where $x$ and $y$ are cells' coordinates (zero-based). On the one hand, each J hexomino pair, no matter how placed, covers sum congruent to $0\pmod{24}$. Then sum covered by all hexominoes is also congruent to $0\pmod{24}$. On the other hand, rectangle covers sum congruent to $\sum_{x=0}^{n-1}\sum_{y=0}^{m-1}F(x,y)$, which is not congruent to $0\pmod{24}$ for $3\nmid n,3\nmid m,4\nmid n,4\nmid m$. Contradiction, as hexomino tiles this rectangle and thus sum covered by all hexominoes should be equal to sum covered by rectangle. Thus only assumption we made is false - J hexomino doesn't tile $n\times m$ rectangles for $3\nmid n,3\nmid m,4\nmid n,4\nmid m$. Q.E.D.