Sum of its parts
Jane Street posted a very interesting puzzle on Hugging Face:
Today I went on a hike and found a pile of tensors hidden underneath a neolithic burial mound!
I sent it over to the local neural plumber, and they managed to cobble together this:
Anyway, I’m not sure what it does yet, but it must have been important to this past civilization. Maybe start by looking at the last two layers.
It includes an interactive playground widget, with the sample input
vegetable dog, to which the model replies
0.
This ended up being one of the most fascinating puzzles I’ve ever worked on. I encourage you to try to solve it on your own, without spoilers, first.
This post won’t outright spoil the answer. It will analyze a tiny piece of the full model which has very interesting properties. But it will spoil much of how the model is structured, and let one easily extrapolate the kind of operation to look for in the rest.
Assets if you’re interested in exploring the full puzzle on your own,
but wary of executing pickled code.
Contains instructions on reconstructing the model from a supplied
safetensors file.
Spoils the feature vector encoder, by
providing disassembly and reconstruction. I consider this relatively
minor.
The full model is an alternating series of Linear and
ReLU layers.
I’ve provided a safetensors file containing the weights and biases for each linear layer.
The feature vector encoder is set at model._call_impl,
and is equivalent to:
lambda x: model.forward(torch.Tensor(list(map(ord, str(x)[:55].ljust(55, '\x00')))))Disassembly comparison
from dis import dis
import torch
from typing import Any
model: Any # spoofdis(model._call_impl) 2 0 RESUME 0
2 LOAD_GLOBAL 0 (model)
14 LOAD_METHOD 1 (forward)
36 LOAD_GLOBAL 4 (torch)
48 LOAD_METHOD 3 (Tensor)
70 LOAD_GLOBAL 9 (NULL + list)
82 LOAD_GLOBAL 11 (NULL + map)
94 LOAD_GLOBAL 12 (ord)
106 LOAD_GLOBAL 15 (NULL + str)
118 LOAD_FAST 0 (x)
120 PRECALL 1
124 CALL 1
134 LOAD_CONST 0 (None)
136 LOAD_CONST 1 (55)
138 BUILD_SLICE 2
140 BINARY_SUBSCR
150 LOAD_METHOD 8 (ljust)
172 LOAD_CONST 1 (55)
174 LOAD_CONST 2 ('\x00')
176 PRECALL 2
180 CALL 2
190 PRECALL 2
194 CALL 2
204 PRECALL 1
208 CALL 1
218 PRECALL 1
222 CALL 1
232 PRECALL 1
236 CALL 1
246 RETURN_VALUEdis(lambda x: model.forward(torch.Tensor(list(map(ord, str(x)[:55].ljust(55, '\x00')))))) 1 0 RESUME 0
2 LOAD_GLOBAL 0 (model)
14 LOAD_METHOD 1 (forward)
36 LOAD_GLOBAL 4 (torch)
48 LOAD_METHOD 3 (Tensor)
70 LOAD_GLOBAL 9 (NULL + list)
82 LOAD_GLOBAL 11 (NULL + map)
94 LOAD_GLOBAL 12 (ord)
106 LOAD_GLOBAL 15 (NULL + str)
118 LOAD_FAST 0 (x)
120 PRECALL 1
124 CALL 1
134 LOAD_CONST 0 (None)
136 LOAD_CONST 1 (55)
138 BUILD_SLICE 2
140 BINARY_SUBSCR
150 LOAD_METHOD 8 (ljust)
172 LOAD_CONST 1 (55)
174 LOAD_CONST 2 ('\x00')
176 PRECALL 2
180 CALL 2
190 PRECALL 2
194 CALL 2
204 PRECALL 1
208 CALL 1
218 PRECALL 1
222 CALL 1
232 PRECALL 1
236 CALL 1
246 RETURN_VALUE(This is an extremely weird approach to encoding string input! This is what first seriously piqued my curiosity about the puzzle.)
Under strace in a sandbox, there didn’t appear to be
anything untoward happening during loading, or running forward
inference.
The model can be fully reconstructed using the following code:
from functools import cache
from torch import Tensor
from torch.nn import Linear, Module, Parameter, ReLU, Sequential
from typing import Iterable
@cache
def get_network_state() -> dict[str, Tensor]:
from safetensors.torch import load_file
state = load_file('files/model_3_11.safetensors')
return state
@cache
def get_network() -> Sequential:
state = get_network_state()
def layers() -> Iterable[Module]:
for i in range(0, 5442, 2):
weights = state[f'{i}.weight']
bias = state[f'{i}.bias']
lin = Linear(
in_features=weights.shape[1],
out_features=weights.shape[0],
)
lin.weight = Parameter(weights, requires_grad=False)
lin.bias = Parameter(bias, requires_grad=False)
yield lin
yield ReLU()
model = Sequential(*layers()).eval()
model.to("cuda")
return modeldisplay(get_network())Sequential(
(0): Linear(in_features=55, out_features=224, bias=True)
(1): ReLU()
(2): Linear(in_features=224, out_features=232, bias=True)
(3): ReLU()
(4): Linear(in_features=232, out_features=64, bias=True)
(5): ReLU()
(6): Linear(in_features=64, out_features=208, bias=True)
(7): ReLU()
(8): Linear(in_features=208, out_features=200, bias=True)
(9): ReLU()
(10): Linear(in_features=200, out_features=212, bias=True)
(11): ReLU()
(12): Linear(in_features=212, out_features=204, bias=True)
(13): ReLU()
(14): Linear(in_features=204, out_features=216, bias=True)
(15): ReLU()
(16): Linear(in_features=216, out_features=208, bias=True)
(17): ReLU()
(18): Linear(in_features=208, out_features=220, bias=True)
(19): ReLU()
(20): Linear(in_features=220, out_features=212, bias=True)
(21): ReLU()
(22): Linear(in_features=212, out_features=224, bias=True)
(23): ReLU()
(24): Linear(in_features=224, out_features=216, bias=True)
(25): ReLU()
(26): Linear(in_features=216, out_features=228, bias=True)
(27): ReLU()
(28): Linear(in_features=228, out_features=220, bias=True)
(29): ReLU()
(30): Linear(in_features=220, out_features=232, bias=True)
(31): ReLU()
(32): Linear(in_features=232, out_features=224, bias=True)
(33): ReLU()
(34): Linear(in_features=224, out_features=336, bias=True)
(35): ReLU()
(36): Linear(in_features=336, out_features=296, bias=True)
(37): ReLU()
(38): Linear(in_features=296, out_features=340, bias=True)
(39): ReLU()
(40): Linear(in_features=340, out_features=332, bias=True)
(41): ReLU()
(42): Linear(in_features=332, out_features=375, bias=True)
(43): ReLU()
(44): Linear(in_features=375, out_features=399, bias=True)
(45): ReLU()
(46): Linear(in_features=399, out_features=410, bias=True)
(47): ReLU()
(48): Linear(in_features=410, out_features=402, bias=True)
(49): ReLU()
(50): Linear(in_features=402, out_features=412, bias=True)
(51): ReLU()
(52): Linear(in_features=412, out_features=404, bias=True)
(53): ReLU()
(54): Linear(in_features=404, out_features=412, bias=True)
(55): ReLU()
(56): Linear(in_features=412, out_features=404, bias=True)
(57): ReLU()
(58): Linear(in_features=404, out_features=408, bias=True)
(59): ReLU()
(60): Linear(in_features=408, out_features=400, bias=True)
(61): ReLU()
(62): Linear(in_features=400, out_features=352, bias=True)
(63): ReLU()
(64): Linear(in_features=352, out_features=288, bias=True)
(65): ReLU()
(66): Linear(in_features=288, out_features=288, bias=True)
(67): ReLU()
(68): Linear(in_features=288, out_features=256, bias=True)
(69): ReLU()
(70): Linear(in_features=256, out_features=319, bias=True)
(71): ReLU()
(72): Linear(in_features=319, out_features=288, bias=True)
(73): ReLU()
(74): Linear(in_features=288, out_features=318, bias=True)
(75): ReLU()
(76): Linear(in_features=318, out_features=288, bias=True)
(77): ReLU()
(78): Linear(in_features=288, out_features=316, bias=True)
(79): ReLU()
(80): Linear(in_features=316, out_features=288, bias=True)
(81): ReLU()
(82): Linear(in_features=288, out_features=312, bias=True)
(83): ReLU()
(84): Linear(in_features=312, out_features=288, bias=True)
(85): ReLU()
(86): Linear(in_features=288, out_features=304, bias=True)
(87): ReLU()
(88): Linear(in_features=304, out_features=288, bias=True)
(89): ReLU()
(90): Linear(in_features=288, out_features=256, bias=True)
(91): ReLU()
(92): Linear(in_features=256, out_features=288, bias=True)
(93): ReLU()
(94): Linear(in_features=288, out_features=256, bias=True)
(95): ReLU()
(96): Linear(in_features=256, out_features=319, bias=True)
(97): ReLU()
(98): Linear(in_features=319, out_features=288, bias=True)
(99): ReLU()
(100): Linear(in_features=288, out_features=318, bias=True)
(101): ReLU()
(102): Linear(in_features=318, out_features=288, bias=True)
(103): ReLU()
(104): Linear(in_features=288, out_features=316, bias=True)
(105): ReLU()
(106): Linear(in_features=316, out_features=288, bias=True)
(107): ReLU()
(108): Linear(in_features=288, out_features=312, bias=True)
(109): ReLU()
(110): Linear(in_features=312, out_features=288, bias=True)
(111): ReLU()
(112): Linear(in_features=288, out_features=304, bias=True)
(113): ReLU()
(114): Linear(in_features=304, out_features=288, bias=True)
(115): ReLU()
(116): Linear(in_features=288, out_features=256, bias=True)
(117): ReLU()
(118): Linear(in_features=256, out_features=336, bias=True)
(119): ReLU()
(120): Linear(in_features=336, out_features=296, bias=True)
(121): ReLU()
(122): Linear(in_features=296, out_features=340, bias=True)
(123): ReLU()
(124): Linear(in_features=340, out_features=332, bias=True)
(125): ReLU()
(126): Linear(in_features=332, out_features=375, bias=True)
(127): ReLU()
(128): Linear(in_features=375, out_features=399, bias=True)
(129): ReLU()
(130): Linear(in_features=399, out_features=410, bias=True)
(131): ReLU()
(132): Linear(in_features=410, out_features=402, bias=True)
(133): ReLU()
(134): Linear(in_features=402, out_features=412, bias=True)
(135): ReLU()
(136): Linear(in_features=412, out_features=404, bias=True)
(137): ReLU()
(138): Linear(in_features=404, out_features=412, bias=True)
(139): ReLU()
(140): Linear(in_features=412, out_features=404, bias=True)
(141): ReLU()
(142): Linear(in_features=404, out_features=408, bias=True)
(143): ReLU()
(144): Linear(in_features=408, out_features=400, bias=True)
(145): ReLU()
(146): Linear(in_features=400, out_features=352, bias=True)
(147): ReLU()
(148): Linear(in_features=352, out_features=288, bias=True)
(149): ReLU()
(150): Linear(in_features=288, out_features=288, bias=True)
(151): ReLU()
(152): Linear(in_features=288, out_features=256, bias=True)
(153): ReLU()
(154): Linear(in_features=256, out_features=319, bias=True)
(155): ReLU()
(156): Linear(in_features=319, out_features=288, bias=True)
(157): ReLU()
(158): Linear(in_features=288, out_features=318, bias=True)
(159): ReLU()
(160): Linear(in_features=318, out_features=288, bias=True)
(161): ReLU()
(162): Linear(in_features=288, out_features=316, bias=True)
(163): ReLU()
(164): Linear(in_features=316, out_features=288, bias=True)
(165): ReLU()
(166): Linear(in_features=288, out_features=312, bias=True)
(167): ReLU()
(168): Linear(in_features=312, out_features=288, bias=True)
(169): ReLU()
(170): Linear(in_features=288, out_features=304, bias=True)
(171): ReLU()
(172): Linear(in_features=304, out_features=288, bias=True)
(173): ReLU()
(174): Linear(in_features=288, out_features=256, bias=True)
(175): ReLU()
(176): Linear(in_features=256, out_features=288, bias=True)
(177): ReLU()
(178): Linear(in_features=288, out_features=256, bias=True)
(179): ReLU()
(180): Linear(in_features=256, out_features=319, bias=True)
(181): ReLU()
(182): Linear(in_features=319, out_features=288, bias=True)
(183): ReLU()
(184): Linear(in_features=288, out_features=318, bias=True)
(185): ReLU()
(186): Linear(in_features=318, out_features=288, bias=True)
(187): ReLU()
(188): Linear(in_features=288, out_features=316, bias=True)
(189): ReLU()
(190): Linear(in_features=316, out_features=288, bias=True)
(191): ReLU()
(192): Linear(in_features=288, out_features=312, bias=True)
(193): ReLU()
(194): Linear(in_features=312, out_features=288, bias=True)
(195): ReLU()
(196): Linear(in_features=288, out_features=304, bias=True)
(197): ReLU()
(198): Linear(in_features=304, out_features=288, bias=True)
(199): ReLU()
(200): Linear(in_features=288, out_features=256, bias=True)
(201): ReLU()
(202): Linear(in_features=256, out_features=336, bias=True)
(203): ReLU()
(204): Linear(in_features=336, out_features=296, bias=True)
(205): ReLU()
(206): Linear(in_features=296, out_features=340, bias=True)
(207): ReLU()
(208): Linear(in_features=340, out_features=332, bias=True)
(209): ReLU()
(210): Linear(in_features=332, out_features=375, bias=True)
(211): ReLU()
(212): Linear(in_features=375, out_features=399, bias=True)
(213): ReLU()
(214): Linear(in_features=399, out_features=410, bias=True)
(215): ReLU()
(216): Linear(in_features=410, out_features=402, bias=True)
(217): ReLU()
(218): Linear(in_features=402, out_features=412, bias=True)
(219): ReLU()
(220): Linear(in_features=412, out_features=404, bias=True)
(221): ReLU()
(222): Linear(in_features=404, out_features=412, bias=True)
(223): ReLU()
(224): Linear(in_features=412, out_features=404, bias=True)
(225): ReLU()
(226): Linear(in_features=404, out_features=408, bias=True)
(227): ReLU()
(228): Linear(in_features=408, out_features=400, bias=True)
(229): ReLU()
(230): Linear(in_features=400, out_features=352, bias=True)
(231): ReLU()
(232): Linear(in_features=352, out_features=288, bias=True)
(233): ReLU()
(234): Linear(in_features=288, out_features=288, bias=True)
(235): ReLU()
(236): Linear(in_features=288, out_features=256, bias=True)
(237): ReLU()
(238): Linear(in_features=256, out_features=319, bias=True)
(239): ReLU()
(240): Linear(in_features=319, out_features=288, bias=True)
(241): ReLU()
(242): Linear(in_features=288, out_features=318, bias=True)
(243): ReLU()
(244): Linear(in_features=318, out_features=288, bias=True)
(245): ReLU()
(246): Linear(in_features=288, out_features=316, bias=True)
(247): ReLU()
(248): Linear(in_features=316, out_features=288, bias=True)
(249): ReLU()
(250): Linear(in_features=288, out_features=312, bias=True)
(251): ReLU()
(252): Linear(in_features=312, out_features=288, bias=True)
(253): ReLU()
(254): Linear(in_features=288, out_features=304, bias=True)
(255): ReLU()
(256): Linear(in_features=304, out_features=288, bias=True)
(257): ReLU()
(258): Linear(in_features=288, out_features=256, bias=True)
(259): ReLU()
(260): Linear(in_features=256, out_features=288, bias=True)
(261): ReLU()
(262): Linear(in_features=288, out_features=256, bias=True)
(263): ReLU()
(264): Linear(in_features=256, out_features=319, bias=True)
(265): ReLU()
(266): Linear(in_features=319, out_features=288, bias=True)
(267): ReLU()
(268): Linear(in_features=288, out_features=318, bias=True)
(269): ReLU()
(270): Linear(in_features=318, out_features=288, bias=True)
(271): ReLU()
(272): Linear(in_features=288, out_features=316, bias=True)
(273): ReLU()
(274): Linear(in_features=316, out_features=288, bias=True)
(275): ReLU()
(276): Linear(in_features=288, out_features=312, bias=True)
(277): ReLU()
(278): Linear(in_features=312, out_features=288, bias=True)
(279): ReLU()
(280): Linear(in_features=288, out_features=304, bias=True)
(281): ReLU()
(282): Linear(in_features=304, out_features=288, bias=True)
(283): ReLU()
(284): Linear(in_features=288, out_features=256, bias=True)
(285): ReLU()
(286): Linear(in_features=256, out_features=336, bias=True)
(287): ReLU()
(288): Linear(in_features=336, out_features=296, bias=True)
(289): ReLU()
(290): Linear(in_features=296, out_features=340, bias=True)
(291): ReLU()
(292): Linear(in_features=340, out_features=332, bias=True)
(293): ReLU()
(294): Linear(in_features=332, out_features=375, bias=True)
(295): ReLU()
(296): Linear(in_features=375, out_features=399, bias=True)
(297): ReLU()
(298): Linear(in_features=399, out_features=410, bias=True)
(299): ReLU()
(300): Linear(in_features=410, out_features=402, bias=True)
(301): ReLU()
(302): Linear(in_features=402, out_features=412, bias=True)
(303): ReLU()
(304): Linear(in_features=412, out_features=404, bias=True)
(305): ReLU()
(306): Linear(in_features=404, out_features=412, bias=True)
(307): ReLU()
(308): Linear(in_features=412, out_features=404, bias=True)
(309): ReLU()
(310): Linear(in_features=404, out_features=408, bias=True)
(311): ReLU()
(312): Linear(in_features=408, out_features=400, bias=True)
(313): ReLU()
(314): Linear(in_features=400, out_features=352, bias=True)
(315): ReLU()
(316): Linear(in_features=352, out_features=288, bias=True)
(317): ReLU()
(318): Linear(in_features=288, out_features=288, bias=True)
(319): ReLU()
(320): Linear(in_features=288, out_features=256, bias=True)
(321): ReLU()
(322): Linear(in_features=256, out_features=319, bias=True)
(323): ReLU()
(324): Linear(in_features=319, out_features=288, bias=True)
(325): ReLU()
(326): Linear(in_features=288, out_features=318, bias=True)
(327): ReLU()
(328): Linear(in_features=318, out_features=288, bias=True)
(329): ReLU()
(330): Linear(in_features=288, out_features=316, bias=True)
(331): ReLU()
(332): Linear(in_features=316, out_features=288, bias=True)
(333): ReLU()
(334): Linear(in_features=288, out_features=312, bias=True)
(335): ReLU()
(336): Linear(in_features=312, out_features=288, bias=True)
(337): ReLU()
(338): Linear(in_features=288, out_features=304, bias=True)
(339): ReLU()
(340): Linear(in_features=304, out_features=288, bias=True)
(341): ReLU()
(342): Linear(in_features=288, out_features=256, bias=True)
(343): ReLU()
(344): Linear(in_features=256, out_features=288, bias=True)
(345): ReLU()
(346): Linear(in_features=288, out_features=256, bias=True)
(347): ReLU()
(348): Linear(in_features=256, out_features=319, bias=True)
(349): ReLU()
(350): Linear(in_features=319, out_features=288, bias=True)
(351): ReLU()
(352): Linear(in_features=288, out_features=318, bias=True)
(353): ReLU()
(354): Linear(in_features=318, out_features=288, bias=True)
(355): ReLU()
(356): Linear(in_features=288, out_features=316, bias=True)
(357): ReLU()
(358): Linear(in_features=316, out_features=288, bias=True)
(359): ReLU()
(360): Linear(in_features=288, out_features=312, bias=True)
(361): ReLU()
(362): Linear(in_features=312, out_features=288, bias=True)
(363): ReLU()
(364): Linear(in_features=288, out_features=304, bias=True)
(365): ReLU()
(366): Linear(in_features=304, out_features=288, bias=True)
(367): ReLU()
(368): Linear(in_features=288, out_features=256, bias=True)
(369): ReLU()
(370): Linear(in_features=256, out_features=336, bias=True)
(371): ReLU()
(372): Linear(in_features=336, out_features=296, bias=True)
(373): ReLU()
(374): Linear(in_features=296, out_features=340, bias=True)
(375): ReLU()
(376): Linear(in_features=340, out_features=332, bias=True)
(377): ReLU()
(378): Linear(in_features=332, out_features=375, bias=True)
(379): ReLU()
(380): Linear(in_features=375, out_features=399, bias=True)
(381): ReLU()
(382): Linear(in_features=399, out_features=410, bias=True)
(383): ReLU()
(384): Linear(in_features=410, out_features=402, bias=True)
(385): ReLU()
(386): Linear(in_features=402, out_features=412, bias=True)
(387): ReLU()
(388): Linear(in_features=412, out_features=404, bias=True)
(389): ReLU()
(390): Linear(in_features=404, out_features=412, bias=True)
(391): ReLU()
(392): Linear(in_features=412, out_features=404, bias=True)
(393): ReLU()
(394): Linear(in_features=404, out_features=408, bias=True)
(395): ReLU()
(396): Linear(in_features=408, out_features=400, bias=True)
(397): ReLU()
(398): Linear(in_features=400, out_features=352, bias=True)
(399): ReLU()
(400): Linear(in_features=352, out_features=288, bias=True)
(401): ReLU()
(402): Linear(in_features=288, out_features=288, bias=True)
(403): ReLU()
(404): Linear(in_features=288, out_features=256, bias=True)
(405): ReLU()
(406): Linear(in_features=256, out_features=319, bias=True)
(407): ReLU()
(408): Linear(in_features=319, out_features=288, bias=True)
(409): ReLU()
(410): Linear(in_features=288, out_features=318, bias=True)
(411): ReLU()
(412): Linear(in_features=318, out_features=288, bias=True)
(413): ReLU()
(414): Linear(in_features=288, out_features=316, bias=True)
(415): ReLU()
(416): Linear(in_features=316, out_features=288, bias=True)
(417): ReLU()
(418): Linear(in_features=288, out_features=312, bias=True)
(419): ReLU()
(420): Linear(in_features=312, out_features=288, bias=True)
(421): ReLU()
(422): Linear(in_features=288, out_features=304, bias=True)
(423): ReLU()
(424): Linear(in_features=304, out_features=288, bias=True)
(425): ReLU()
(426): Linear(in_features=288, out_features=256, bias=True)
(427): ReLU()
(428): Linear(in_features=256, out_features=288, bias=True)
(429): ReLU()
(430): Linear(in_features=288, out_features=256, bias=True)
(431): ReLU()
(432): Linear(in_features=256, out_features=319, bias=True)
(433): ReLU()
(434): Linear(in_features=319, out_features=288, bias=True)
(435): ReLU()
(436): Linear(in_features=288, out_features=318, bias=True)
(437): ReLU()
(438): Linear(in_features=318, out_features=288, bias=True)
(439): ReLU()
(440): Linear(in_features=288, out_features=316, bias=True)
(441): ReLU()
(442): Linear(in_features=316, out_features=288, bias=True)
(443): ReLU()
(444): Linear(in_features=288, out_features=312, bias=True)
(445): ReLU()
(446): Linear(in_features=312, out_features=288, bias=True)
(447): ReLU()
(448): Linear(in_features=288, out_features=304, bias=True)
(449): ReLU()
(450): Linear(in_features=304, out_features=288, bias=True)
(451): ReLU()
(452): Linear(in_features=288, out_features=256, bias=True)
(453): ReLU()
(454): Linear(in_features=256, out_features=336, bias=True)
(455): ReLU()
(456): Linear(in_features=336, out_features=296, bias=True)
(457): ReLU()
(458): Linear(in_features=296, out_features=340, bias=True)
(459): ReLU()
(460): Linear(in_features=340, out_features=332, bias=True)
(461): ReLU()
(462): Linear(in_features=332, out_features=375, bias=True)
(463): ReLU()
(464): Linear(in_features=375, out_features=399, bias=True)
(465): ReLU()
(466): Linear(in_features=399, out_features=410, bias=True)
(467): ReLU()
(468): Linear(in_features=410, out_features=402, bias=True)
(469): ReLU()
(470): Linear(in_features=402, out_features=412, bias=True)
(471): ReLU()
(472): Linear(in_features=412, out_features=404, bias=True)
(473): ReLU()
(474): Linear(in_features=404, out_features=412, bias=True)
(475): ReLU()
(476): Linear(in_features=412, out_features=404, bias=True)
(477): ReLU()
(478): Linear(in_features=404, out_features=408, bias=True)
(479): ReLU()
(480): Linear(in_features=408, out_features=400, bias=True)
(481): ReLU()
(482): Linear(in_features=400, out_features=352, bias=True)
(483): ReLU()
(484): Linear(in_features=352, out_features=288, bias=True)
(485): ReLU()
(486): Linear(in_features=288, out_features=288, bias=True)
(487): ReLU()
(488): Linear(in_features=288, out_features=256, bias=True)
(489): ReLU()
(490): Linear(in_features=256, out_features=319, bias=True)
(491): ReLU()
(492): Linear(in_features=319, out_features=288, bias=True)
(493): ReLU()
(494): Linear(in_features=288, out_features=318, bias=True)
(495): ReLU()
(496): Linear(in_features=318, out_features=288, bias=True)
(497): ReLU()
(498): Linear(in_features=288, out_features=316, bias=True)
(499): ReLU()
(500): Linear(in_features=316, out_features=288, bias=True)
(501): ReLU()
(502): Linear(in_features=288, out_features=312, bias=True)
(503): ReLU()
(504): Linear(in_features=312, out_features=288, bias=True)
(505): ReLU()
(506): Linear(in_features=288, out_features=304, bias=True)
(507): ReLU()
(508): Linear(in_features=304, out_features=288, bias=True)
(509): ReLU()
(510): Linear(in_features=288, out_features=256, bias=True)
(511): ReLU()
(512): Linear(in_features=256, out_features=288, bias=True)
(513): ReLU()
(514): Linear(in_features=288, out_features=256, bias=True)
(515): ReLU()
(516): Linear(in_features=256, out_features=319, bias=True)
(517): ReLU()
(518): Linear(in_features=319, out_features=288, bias=True)
(519): ReLU()
(520): Linear(in_features=288, out_features=318, bias=True)
(521): ReLU()
(522): Linear(in_features=318, out_features=288, bias=True)
(523): ReLU()
(524): Linear(in_features=288, out_features=316, bias=True)
(525): ReLU()
(526): Linear(in_features=316, out_features=288, bias=True)
(527): ReLU()
(528): Linear(in_features=288, out_features=312, bias=True)
(529): ReLU()
(530): Linear(in_features=312, out_features=288, bias=True)
(531): ReLU()
(532): Linear(in_features=288, out_features=304, bias=True)
(533): ReLU()
(534): Linear(in_features=304, out_features=288, bias=True)
(535): ReLU()
(536): Linear(in_features=288, out_features=256, bias=True)
(537): ReLU()
(538): Linear(in_features=256, out_features=336, bias=True)
(539): ReLU()
(540): Linear(in_features=336, out_features=296, bias=True)
(541): ReLU()
(542): Linear(in_features=296, out_features=340, bias=True)
(543): ReLU()
(544): Linear(in_features=340, out_features=332, bias=True)
(545): ReLU()
(546): Linear(in_features=332, out_features=375, bias=True)
(547): ReLU()
(548): Linear(in_features=375, out_features=399, bias=True)
(549): ReLU()
(550): Linear(in_features=399, out_features=410, bias=True)
(551): ReLU()
(552): Linear(in_features=410, out_features=402, bias=True)
(553): ReLU()
(554): Linear(in_features=402, out_features=412, bias=True)
(555): ReLU()
(556): Linear(in_features=412, out_features=404, bias=True)
(557): ReLU()
(558): Linear(in_features=404, out_features=412, bias=True)
(559): ReLU()
(560): Linear(in_features=412, out_features=404, bias=True)
(561): ReLU()
(562): Linear(in_features=404, out_features=408, bias=True)
(563): ReLU()
(564): Linear(in_features=408, out_features=400, bias=True)
(565): ReLU()
(566): Linear(in_features=400, out_features=352, bias=True)
(567): ReLU()
(568): Linear(in_features=352, out_features=288, bias=True)
(569): ReLU()
(570): Linear(in_features=288, out_features=288, bias=True)
(571): ReLU()
(572): Linear(in_features=288, out_features=256, bias=True)
(573): ReLU()
(574): Linear(in_features=256, out_features=319, bias=True)
(575): ReLU()
(576): Linear(in_features=319, out_features=288, bias=True)
(577): ReLU()
(578): Linear(in_features=288, out_features=318, bias=True)
(579): ReLU()
(580): Linear(in_features=318, out_features=288, bias=True)
(581): ReLU()
(582): Linear(in_features=288, out_features=316, bias=True)
(583): ReLU()
(584): Linear(in_features=316, out_features=288, bias=True)
(585): ReLU()
(586): Linear(in_features=288, out_features=312, bias=True)
(587): ReLU()
(588): Linear(in_features=312, out_features=288, bias=True)
(589): ReLU()
(590): Linear(in_features=288, out_features=304, bias=True)
(591): ReLU()
(592): Linear(in_features=304, out_features=288, bias=True)
(593): ReLU()
(594): Linear(in_features=288, out_features=256, bias=True)
(595): ReLU()
(596): Linear(in_features=256, out_features=288, bias=True)
(597): ReLU()
(598): Linear(in_features=288, out_features=256, bias=True)
(599): ReLU()
(600): Linear(in_features=256, out_features=319, bias=True)
(601): ReLU()
(602): Linear(in_features=319, out_features=288, bias=True)
(603): ReLU()
(604): Linear(in_features=288, out_features=318, bias=True)
(605): ReLU()
(606): Linear(in_features=318, out_features=288, bias=True)
(607): ReLU()
(608): Linear(in_features=288, out_features=316, bias=True)
(609): ReLU()
(610): Linear(in_features=316, out_features=288, bias=True)
(611): ReLU()
(612): Linear(in_features=288, out_features=312, bias=True)
(613): ReLU()
(614): Linear(in_features=312, out_features=288, bias=True)
(615): ReLU()
(616): Linear(in_features=288, out_features=304, bias=True)
(617): ReLU()
(618): Linear(in_features=304, out_features=288, bias=True)
(619): ReLU()
(620): Linear(in_features=288, out_features=256, bias=True)
(621): ReLU()
(622): Linear(in_features=256, out_features=336, bias=True)
(623): ReLU()
(624): Linear(in_features=336, out_features=296, bias=True)
(625): ReLU()
(626): Linear(in_features=296, out_features=340, bias=True)
(627): ReLU()
(628): Linear(in_features=340, out_features=332, bias=True)
(629): ReLU()
(630): Linear(in_features=332, out_features=375, bias=True)
(631): ReLU()
(632): Linear(in_features=375, out_features=399, bias=True)
(633): ReLU()
(634): Linear(in_features=399, out_features=410, bias=True)
(635): ReLU()
(636): Linear(in_features=410, out_features=402, bias=True)
(637): ReLU()
(638): Linear(in_features=402, out_features=412, bias=True)
(639): ReLU()
(640): Linear(in_features=412, out_features=404, bias=True)
(641): ReLU()
(642): Linear(in_features=404, out_features=412, bias=True)
(643): ReLU()
(644): Linear(in_features=412, out_features=404, bias=True)
(645): ReLU()
(646): Linear(in_features=404, out_features=408, bias=True)
(647): ReLU()
(648): Linear(in_features=408, out_features=400, bias=True)
(649): ReLU()
(650): Linear(in_features=400, out_features=352, bias=True)
(651): ReLU()
(652): Linear(in_features=352, out_features=288, bias=True)
(653): ReLU()
(654): Linear(in_features=288, out_features=288, bias=True)
(655): ReLU()
(656): Linear(in_features=288, out_features=256, bias=True)
(657): ReLU()
(658): Linear(in_features=256, out_features=319, bias=True)
(659): ReLU()
(660): Linear(in_features=319, out_features=288, bias=True)
(661): ReLU()
(662): Linear(in_features=288, out_features=318, bias=True)
(663): ReLU()
(664): Linear(in_features=318, out_features=288, bias=True)
(665): ReLU()
(666): Linear(in_features=288, out_features=316, bias=True)
(667): ReLU()
(668): Linear(in_features=316, out_features=288, bias=True)
(669): ReLU()
(670): Linear(in_features=288, out_features=312, bias=True)
(671): ReLU()
(672): Linear(in_features=312, out_features=288, bias=True)
(673): ReLU()
(674): Linear(in_features=288, out_features=304, bias=True)
(675): ReLU()
(676): Linear(in_features=304, out_features=288, bias=True)
(677): ReLU()
(678): Linear(in_features=288, out_features=256, bias=True)
(679): ReLU()
(680): Linear(in_features=256, out_features=288, bias=True)
(681): ReLU()
(682): Linear(in_features=288, out_features=256, bias=True)
(683): ReLU()
(684): Linear(in_features=256, out_features=319, bias=True)
(685): ReLU()
(686): Linear(in_features=319, out_features=288, bias=True)
(687): ReLU()
(688): Linear(in_features=288, out_features=318, bias=True)
(689): ReLU()
(690): Linear(in_features=318, out_features=288, bias=True)
(691): ReLU()
(692): Linear(in_features=288, out_features=316, bias=True)
(693): ReLU()
(694): Linear(in_features=316, out_features=288, bias=True)
(695): ReLU()
(696): Linear(in_features=288, out_features=312, bias=True)
(697): ReLU()
(698): Linear(in_features=312, out_features=288, bias=True)
(699): ReLU()
(700): Linear(in_features=288, out_features=304, bias=True)
(701): ReLU()
(702): Linear(in_features=304, out_features=288, bias=True)
(703): ReLU()
(704): Linear(in_features=288, out_features=256, bias=True)
(705): ReLU()
(706): Linear(in_features=256, out_features=336, bias=True)
(707): ReLU()
(708): Linear(in_features=336, out_features=296, bias=True)
(709): ReLU()
(710): Linear(in_features=296, out_features=340, bias=True)
(711): ReLU()
(712): Linear(in_features=340, out_features=332, bias=True)
(713): ReLU()
(714): Linear(in_features=332, out_features=375, bias=True)
(715): ReLU()
(716): Linear(in_features=375, out_features=399, bias=True)
(717): ReLU()
(718): Linear(in_features=399, out_features=410, bias=True)
(719): ReLU()
(720): Linear(in_features=410, out_features=402, bias=True)
(721): ReLU()
(722): Linear(in_features=402, out_features=412, bias=True)
(723): ReLU()
(724): Linear(in_features=412, out_features=404, bias=True)
(725): ReLU()
(726): Linear(in_features=404, out_features=412, bias=True)
(727): ReLU()
(728): Linear(in_features=412, out_features=404, bias=True)
(729): ReLU()
(730): Linear(in_features=404, out_features=408, bias=True)
(731): ReLU()
(732): Linear(in_features=408, out_features=400, bias=True)
(733): ReLU()
(734): Linear(in_features=400, out_features=352, bias=True)
(735): ReLU()
(736): Linear(in_features=352, out_features=288, bias=True)
(737): ReLU()
(738): Linear(in_features=288, out_features=288, bias=True)
(739): ReLU()
(740): Linear(in_features=288, out_features=256, bias=True)
(741): ReLU()
(742): Linear(in_features=256, out_features=319, bias=True)
(743): ReLU()
(744): Linear(in_features=319, out_features=288, bias=True)
(745): ReLU()
(746): Linear(in_features=288, out_features=318, bias=True)
(747): ReLU()
(748): Linear(in_features=318, out_features=288, bias=True)
(749): ReLU()
(750): Linear(in_features=288, out_features=316, bias=True)
(751): ReLU()
(752): Linear(in_features=316, out_features=288, bias=True)
(753): ReLU()
(754): Linear(in_features=288, out_features=312, bias=True)
(755): ReLU()
(756): Linear(in_features=312, out_features=288, bias=True)
(757): ReLU()
(758): Linear(in_features=288, out_features=304, bias=True)
(759): ReLU()
(760): Linear(in_features=304, out_features=288, bias=True)
(761): ReLU()
(762): Linear(in_features=288, out_features=256, bias=True)
(763): ReLU()
(764): Linear(in_features=256, out_features=288, bias=True)
(765): ReLU()
(766): Linear(in_features=288, out_features=256, bias=True)
(767): ReLU()
(768): Linear(in_features=256, out_features=319, bias=True)
(769): ReLU()
(770): Linear(in_features=319, out_features=288, bias=True)
(771): ReLU()
(772): Linear(in_features=288, out_features=318, bias=True)
(773): ReLU()
(774): Linear(in_features=318, out_features=288, bias=True)
(775): ReLU()
(776): Linear(in_features=288, out_features=316, bias=True)
(777): ReLU()
(778): Linear(in_features=316, out_features=288, bias=True)
(779): ReLU()
(780): Linear(in_features=288, out_features=312, bias=True)
(781): ReLU()
(782): Linear(in_features=312, out_features=288, bias=True)
(783): ReLU()
(784): Linear(in_features=288, out_features=304, bias=True)
(785): ReLU()
(786): Linear(in_features=304, out_features=288, bias=True)
(787): ReLU()
(788): Linear(in_features=288, out_features=256, bias=True)
(789): ReLU()
(790): Linear(in_features=256, out_features=336, bias=True)
(791): ReLU()
(792): Linear(in_features=336, out_features=296, bias=True)
(793): ReLU()
(794): Linear(in_features=296, out_features=340, bias=True)
(795): ReLU()
(796): Linear(in_features=340, out_features=332, bias=True)
(797): ReLU()
(798): Linear(in_features=332, out_features=375, bias=True)
(799): ReLU()
(800): Linear(in_features=375, out_features=399, bias=True)
(801): ReLU()
(802): Linear(in_features=399, out_features=410, bias=True)
(803): ReLU()
(804): Linear(in_features=410, out_features=402, bias=True)
(805): ReLU()
(806): Linear(in_features=402, out_features=412, bias=True)
(807): ReLU()
(808): Linear(in_features=412, out_features=404, bias=True)
(809): ReLU()
(810): Linear(in_features=404, out_features=412, bias=True)
(811): ReLU()
(812): Linear(in_features=412, out_features=404, bias=True)
(813): ReLU()
(814): Linear(in_features=404, out_features=408, bias=True)
(815): ReLU()
(816): Linear(in_features=408, out_features=400, bias=True)
(817): ReLU()
(818): Linear(in_features=400, out_features=352, bias=True)
(819): ReLU()
(820): Linear(in_features=352, out_features=288, bias=True)
(821): ReLU()
(822): Linear(in_features=288, out_features=288, bias=True)
(823): ReLU()
(824): Linear(in_features=288, out_features=256, bias=True)
(825): ReLU()
(826): Linear(in_features=256, out_features=319, bias=True)
(827): ReLU()
(828): Linear(in_features=319, out_features=288, bias=True)
(829): ReLU()
(830): Linear(in_features=288, out_features=318, bias=True)
(831): ReLU()
(832): Linear(in_features=318, out_features=288, bias=True)
(833): ReLU()
(834): Linear(in_features=288, out_features=316, bias=True)
(835): ReLU()
(836): Linear(in_features=316, out_features=288, bias=True)
(837): ReLU()
(838): Linear(in_features=288, out_features=312, bias=True)
(839): ReLU()
(840): Linear(in_features=312, out_features=288, bias=True)
(841): ReLU()
(842): Linear(in_features=288, out_features=304, bias=True)
(843): ReLU()
(844): Linear(in_features=304, out_features=288, bias=True)
(845): ReLU()
(846): Linear(in_features=288, out_features=256, bias=True)
(847): ReLU()
(848): Linear(in_features=256, out_features=288, bias=True)
(849): ReLU()
(850): Linear(in_features=288, out_features=256, bias=True)
(851): ReLU()
(852): Linear(in_features=256, out_features=319, bias=True)
(853): ReLU()
(854): Linear(in_features=319, out_features=288, bias=True)
(855): ReLU()
(856): Linear(in_features=288, out_features=318, bias=True)
(857): ReLU()
(858): Linear(in_features=318, out_features=288, bias=True)
(859): ReLU()
(860): Linear(in_features=288, out_features=316, bias=True)
(861): ReLU()
(862): Linear(in_features=316, out_features=288, bias=True)
(863): ReLU()
(864): Linear(in_features=288, out_features=312, bias=True)
(865): ReLU()
(866): Linear(in_features=312, out_features=288, bias=True)
(867): ReLU()
(868): Linear(in_features=288, out_features=304, bias=True)
(869): ReLU()
(870): Linear(in_features=304, out_features=288, bias=True)
(871): ReLU()
(872): Linear(in_features=288, out_features=256, bias=True)
(873): ReLU()
(874): Linear(in_features=256, out_features=336, bias=True)
(875): ReLU()
(876): Linear(in_features=336, out_features=296, bias=True)
(877): ReLU()
(878): Linear(in_features=296, out_features=340, bias=True)
(879): ReLU()
(880): Linear(in_features=340, out_features=332, bias=True)
(881): ReLU()
(882): Linear(in_features=332, out_features=375, bias=True)
(883): ReLU()
(884): Linear(in_features=375, out_features=399, bias=True)
(885): ReLU()
(886): Linear(in_features=399, out_features=410, bias=True)
(887): ReLU()
(888): Linear(in_features=410, out_features=402, bias=True)
(889): ReLU()
(890): Linear(in_features=402, out_features=412, bias=True)
(891): ReLU()
(892): Linear(in_features=412, out_features=404, bias=True)
(893): ReLU()
(894): Linear(in_features=404, out_features=412, bias=True)
(895): ReLU()
(896): Linear(in_features=412, out_features=404, bias=True)
(897): ReLU()
(898): Linear(in_features=404, out_features=408, bias=True)
(899): ReLU()
(900): Linear(in_features=408, out_features=400, bias=True)
(901): ReLU()
(902): Linear(in_features=400, out_features=352, bias=True)
(903): ReLU()
(904): Linear(in_features=352, out_features=288, bias=True)
(905): ReLU()
(906): Linear(in_features=288, out_features=288, bias=True)
(907): ReLU()
(908): Linear(in_features=288, out_features=256, bias=True)
(909): ReLU()
(910): Linear(in_features=256, out_features=319, bias=True)
(911): ReLU()
(912): Linear(in_features=319, out_features=288, bias=True)
(913): ReLU()
(914): Linear(in_features=288, out_features=318, bias=True)
(915): ReLU()
(916): Linear(in_features=318, out_features=288, bias=True)
(917): ReLU()
(918): Linear(in_features=288, out_features=316, bias=True)
(919): ReLU()
(920): Linear(in_features=316, out_features=288, bias=True)
(921): ReLU()
(922): Linear(in_features=288, out_features=312, bias=True)
(923): ReLU()
(924): Linear(in_features=312, out_features=288, bias=True)
(925): ReLU()
(926): Linear(in_features=288, out_features=304, bias=True)
(927): ReLU()
(928): Linear(in_features=304, out_features=288, bias=True)
(929): ReLU()
(930): Linear(in_features=288, out_features=256, bias=True)
(931): ReLU()
(932): Linear(in_features=256, out_features=288, bias=True)
(933): ReLU()
(934): Linear(in_features=288, out_features=256, bias=True)
(935): ReLU()
(936): Linear(in_features=256, out_features=319, bias=True)
(937): ReLU()
(938): Linear(in_features=319, out_features=288, bias=True)
(939): ReLU()
(940): Linear(in_features=288, out_features=318, bias=True)
(941): ReLU()
(942): Linear(in_features=318, out_features=288, bias=True)
(943): ReLU()
(944): Linear(in_features=288, out_features=316, bias=True)
(945): ReLU()
(946): Linear(in_features=316, out_features=288, bias=True)
(947): ReLU()
(948): Linear(in_features=288, out_features=312, bias=True)
(949): ReLU()
(950): Linear(in_features=312, out_features=288, bias=True)
(951): ReLU()
(952): Linear(in_features=288, out_features=304, bias=True)
(953): ReLU()
(954): Linear(in_features=304, out_features=288, bias=True)
(955): ReLU()
(956): Linear(in_features=288, out_features=256, bias=True)
(957): ReLU()
(958): Linear(in_features=256, out_features=336, bias=True)
(959): ReLU()
(960): Linear(in_features=336, out_features=296, bias=True)
(961): ReLU()
(962): Linear(in_features=296, out_features=340, bias=True)
(963): ReLU()
(964): Linear(in_features=340, out_features=332, bias=True)
(965): ReLU()
(966): Linear(in_features=332, out_features=375, bias=True)
(967): ReLU()
(968): Linear(in_features=375, out_features=399, bias=True)
(969): ReLU()
(970): Linear(in_features=399, out_features=410, bias=True)
(971): ReLU()
(972): Linear(in_features=410, out_features=402, bias=True)
(973): ReLU()
(974): Linear(in_features=402, out_features=412, bias=True)
(975): ReLU()
(976): Linear(in_features=412, out_features=404, bias=True)
(977): ReLU()
(978): Linear(in_features=404, out_features=412, bias=True)
(979): ReLU()
(980): Linear(in_features=412, out_features=404, bias=True)
(981): ReLU()
(982): Linear(in_features=404, out_features=408, bias=True)
(983): ReLU()
(984): Linear(in_features=408, out_features=400, bias=True)
(985): ReLU()
(986): Linear(in_features=400, out_features=352, bias=True)
(987): ReLU()
(988): Linear(in_features=352, out_features=288, bias=True)
(989): ReLU()
(990): Linear(in_features=288, out_features=288, bias=True)
(991): ReLU()
(992): Linear(in_features=288, out_features=256, bias=True)
(993): ReLU()
(994): Linear(in_features=256, out_features=319, bias=True)
(995): ReLU()
(996): Linear(in_features=319, out_features=288, bias=True)
(997): ReLU()
(998): Linear(in_features=288, out_features=318, bias=True)
(999): ReLU()
(1000): Linear(in_features=318, out_features=288, bias=True)
(1001): ReLU()
(1002): Linear(in_features=288, out_features=316, bias=True)
(1003): ReLU()
(1004): Linear(in_features=316, out_features=288, bias=True)
(1005): ReLU()
(1006): Linear(in_features=288, out_features=312, bias=True)
(1007): ReLU()
(1008): Linear(in_features=312, out_features=288, bias=True)
(1009): ReLU()
(1010): Linear(in_features=288, out_features=304, bias=True)
(1011): ReLU()
(1012): Linear(in_features=304, out_features=288, bias=True)
(1013): ReLU()
(1014): Linear(in_features=288, out_features=256, bias=True)
(1015): ReLU()
(1016): Linear(in_features=256, out_features=288, bias=True)
(1017): ReLU()
(1018): Linear(in_features=288, out_features=256, bias=True)
(1019): ReLU()
(1020): Linear(in_features=256, out_features=319, bias=True)
(1021): ReLU()
(1022): Linear(in_features=319, out_features=288, bias=True)
(1023): ReLU()
(1024): Linear(in_features=288, out_features=318, bias=True)
(1025): ReLU()
(1026): Linear(in_features=318, out_features=288, bias=True)
(1027): ReLU()
(1028): Linear(in_features=288, out_features=316, bias=True)
(1029): ReLU()
(1030): Linear(in_features=316, out_features=288, bias=True)
(1031): ReLU()
(1032): Linear(in_features=288, out_features=312, bias=True)
(1033): ReLU()
(1034): Linear(in_features=312, out_features=288, bias=True)
(1035): ReLU()
(1036): Linear(in_features=288, out_features=304, bias=True)
(1037): ReLU()
(1038): Linear(in_features=304, out_features=288, bias=True)
(1039): ReLU()
(1040): Linear(in_features=288, out_features=256, bias=True)
(1041): ReLU()
(1042): Linear(in_features=256, out_features=336, bias=True)
(1043): ReLU()
(1044): Linear(in_features=336, out_features=296, bias=True)
(1045): ReLU()
(1046): Linear(in_features=296, out_features=340, bias=True)
(1047): ReLU()
(1048): Linear(in_features=340, out_features=332, bias=True)
(1049): ReLU()
(1050): Linear(in_features=332, out_features=375, bias=True)
(1051): ReLU()
(1052): Linear(in_features=375, out_features=399, bias=True)
(1053): ReLU()
(1054): Linear(in_features=399, out_features=410, bias=True)
(1055): ReLU()
(1056): Linear(in_features=410, out_features=402, bias=True)
(1057): ReLU()
(1058): Linear(in_features=402, out_features=412, bias=True)
(1059): ReLU()
(1060): Linear(in_features=412, out_features=404, bias=True)
(1061): ReLU()
(1062): Linear(in_features=404, out_features=412, bias=True)
(1063): ReLU()
(1064): Linear(in_features=412, out_features=404, bias=True)
(1065): ReLU()
(1066): Linear(in_features=404, out_features=408, bias=True)
(1067): ReLU()
(1068): Linear(in_features=408, out_features=400, bias=True)
(1069): ReLU()
(1070): Linear(in_features=400, out_features=352, bias=True)
(1071): ReLU()
(1072): Linear(in_features=352, out_features=288, bias=True)
(1073): ReLU()
(1074): Linear(in_features=288, out_features=288, bias=True)
(1075): ReLU()
(1076): Linear(in_features=288, out_features=256, bias=True)
(1077): ReLU()
(1078): Linear(in_features=256, out_features=319, bias=True)
(1079): ReLU()
(1080): Linear(in_features=319, out_features=288, bias=True)
(1081): ReLU()
(1082): Linear(in_features=288, out_features=318, bias=True)
(1083): ReLU()
(1084): Linear(in_features=318, out_features=288, bias=True)
(1085): ReLU()
(1086): Linear(in_features=288, out_features=316, bias=True)
(1087): ReLU()
(1088): Linear(in_features=316, out_features=288, bias=True)
(1089): ReLU()
(1090): Linear(in_features=288, out_features=312, bias=True)
(1091): ReLU()
(1092): Linear(in_features=312, out_features=288, bias=True)
(1093): ReLU()
(1094): Linear(in_features=288, out_features=304, bias=True)
(1095): ReLU()
(1096): Linear(in_features=304, out_features=288, bias=True)
(1097): ReLU()
(1098): Linear(in_features=288, out_features=256, bias=True)
(1099): ReLU()
(1100): Linear(in_features=256, out_features=288, bias=True)
(1101): ReLU()
(1102): Linear(in_features=288, out_features=256, bias=True)
(1103): ReLU()
(1104): Linear(in_features=256, out_features=319, bias=True)
(1105): ReLU()
(1106): Linear(in_features=319, out_features=288, bias=True)
(1107): ReLU()
(1108): Linear(in_features=288, out_features=318, bias=True)
(1109): ReLU()
(1110): Linear(in_features=318, out_features=288, bias=True)
(1111): ReLU()
(1112): Linear(in_features=288, out_features=316, bias=True)
(1113): ReLU()
(1114): Linear(in_features=316, out_features=288, bias=True)
(1115): ReLU()
(1116): Linear(in_features=288, out_features=312, bias=True)
(1117): ReLU()
(1118): Linear(in_features=312, out_features=288, bias=True)
(1119): ReLU()
(1120): Linear(in_features=288, out_features=304, bias=True)
(1121): ReLU()
(1122): Linear(in_features=304, out_features=288, bias=True)
(1123): ReLU()
(1124): Linear(in_features=288, out_features=256, bias=True)
(1125): ReLU()
(1126): Linear(in_features=256, out_features=336, bias=True)
(1127): ReLU()
(1128): Linear(in_features=336, out_features=296, bias=True)
(1129): ReLU()
(1130): Linear(in_features=296, out_features=340, bias=True)
(1131): ReLU()
(1132): Linear(in_features=340, out_features=332, bias=True)
(1133): ReLU()
(1134): Linear(in_features=332, out_features=375, bias=True)
(1135): ReLU()
(1136): Linear(in_features=375, out_features=399, bias=True)
(1137): ReLU()
(1138): Linear(in_features=399, out_features=410, bias=True)
(1139): ReLU()
(1140): Linear(in_features=410, out_features=402, bias=True)
(1141): ReLU()
(1142): Linear(in_features=402, out_features=412, bias=True)
(1143): ReLU()
(1144): Linear(in_features=412, out_features=404, bias=True)
(1145): ReLU()
(1146): Linear(in_features=404, out_features=412, bias=True)
(1147): ReLU()
(1148): Linear(in_features=412, out_features=404, bias=True)
(1149): ReLU()
(1150): Linear(in_features=404, out_features=408, bias=True)
(1151): ReLU()
(1152): Linear(in_features=408, out_features=400, bias=True)
(1153): ReLU()
(1154): Linear(in_features=400, out_features=352, bias=True)
(1155): ReLU()
(1156): Linear(in_features=352, out_features=288, bias=True)
(1157): ReLU()
(1158): Linear(in_features=288, out_features=288, bias=True)
(1159): ReLU()
(1160): Linear(in_features=288, out_features=256, bias=True)
(1161): ReLU()
(1162): Linear(in_features=256, out_features=319, bias=True)
(1163): ReLU()
(1164): Linear(in_features=319, out_features=288, bias=True)
(1165): ReLU()
(1166): Linear(in_features=288, out_features=318, bias=True)
(1167): ReLU()
(1168): Linear(in_features=318, out_features=288, bias=True)
(1169): ReLU()
(1170): Linear(in_features=288, out_features=316, bias=True)
(1171): ReLU()
(1172): Linear(in_features=316, out_features=288, bias=True)
(1173): ReLU()
(1174): Linear(in_features=288, out_features=312, bias=True)
(1175): ReLU()
(1176): Linear(in_features=312, out_features=288, bias=True)
(1177): ReLU()
(1178): Linear(in_features=288, out_features=304, bias=True)
(1179): ReLU()
(1180): Linear(in_features=304, out_features=288, bias=True)
(1181): ReLU()
(1182): Linear(in_features=288, out_features=256, bias=True)
(1183): ReLU()
(1184): Linear(in_features=256, out_features=288, bias=True)
(1185): ReLU()
(1186): Linear(in_features=288, out_features=256, bias=True)
(1187): ReLU()
(1188): Linear(in_features=256, out_features=319, bias=True)
(1189): ReLU()
(1190): Linear(in_features=319, out_features=288, bias=True)
(1191): ReLU()
(1192): Linear(in_features=288, out_features=318, bias=True)
(1193): ReLU()
(1194): Linear(in_features=318, out_features=288, bias=True)
(1195): ReLU()
(1196): Linear(in_features=288, out_features=316, bias=True)
(1197): ReLU()
(1198): Linear(in_features=316, out_features=288, bias=True)
(1199): ReLU()
(1200): Linear(in_features=288, out_features=312, bias=True)
(1201): ReLU()
(1202): Linear(in_features=312, out_features=288, bias=True)
(1203): ReLU()
(1204): Linear(in_features=288, out_features=304, bias=True)
(1205): ReLU()
(1206): Linear(in_features=304, out_features=288, bias=True)
(1207): ReLU()
(1208): Linear(in_features=288, out_features=256, bias=True)
(1209): ReLU()
(1210): Linear(in_features=256, out_features=336, bias=True)
(1211): ReLU()
(1212): Linear(in_features=336, out_features=296, bias=True)
(1213): ReLU()
(1214): Linear(in_features=296, out_features=340, bias=True)
(1215): ReLU()
(1216): Linear(in_features=340, out_features=332, bias=True)
(1217): ReLU()
(1218): Linear(in_features=332, out_features=375, bias=True)
(1219): ReLU()
(1220): Linear(in_features=375, out_features=399, bias=True)
(1221): ReLU()
(1222): Linear(in_features=399, out_features=410, bias=True)
(1223): ReLU()
(1224): Linear(in_features=410, out_features=402, bias=True)
(1225): ReLU()
(1226): Linear(in_features=402, out_features=412, bias=True)
(1227): ReLU()
(1228): Linear(in_features=412, out_features=404, bias=True)
(1229): ReLU()
(1230): Linear(in_features=404, out_features=412, bias=True)
(1231): ReLU()
(1232): Linear(in_features=412, out_features=404, bias=True)
(1233): ReLU()
(1234): Linear(in_features=404, out_features=408, bias=True)
(1235): ReLU()
(1236): Linear(in_features=408, out_features=400, bias=True)
(1237): ReLU()
(1238): Linear(in_features=400, out_features=352, bias=True)
(1239): ReLU()
(1240): Linear(in_features=352, out_features=288, bias=True)
(1241): ReLU()
(1242): Linear(in_features=288, out_features=288, bias=True)
(1243): ReLU()
(1244): Linear(in_features=288, out_features=256, bias=True)
(1245): ReLU()
(1246): Linear(in_features=256, out_features=319, bias=True)
(1247): ReLU()
(1248): Linear(in_features=319, out_features=288, bias=True)
(1249): ReLU()
(1250): Linear(in_features=288, out_features=318, bias=True)
(1251): ReLU()
(1252): Linear(in_features=318, out_features=288, bias=True)
(1253): ReLU()
(1254): Linear(in_features=288, out_features=316, bias=True)
(1255): ReLU()
(1256): Linear(in_features=316, out_features=288, bias=True)
(1257): ReLU()
(1258): Linear(in_features=288, out_features=312, bias=True)
(1259): ReLU()
(1260): Linear(in_features=312, out_features=288, bias=True)
(1261): ReLU()
(1262): Linear(in_features=288, out_features=304, bias=True)
(1263): ReLU()
(1264): Linear(in_features=304, out_features=288, bias=True)
(1265): ReLU()
(1266): Linear(in_features=288, out_features=256, bias=True)
(1267): ReLU()
(1268): Linear(in_features=256, out_features=288, bias=True)
(1269): ReLU()
(1270): Linear(in_features=288, out_features=256, bias=True)
(1271): ReLU()
(1272): Linear(in_features=256, out_features=319, bias=True)
(1273): ReLU()
(1274): Linear(in_features=319, out_features=288, bias=True)
(1275): ReLU()
(1276): Linear(in_features=288, out_features=318, bias=True)
(1277): ReLU()
(1278): Linear(in_features=318, out_features=288, bias=True)
(1279): ReLU()
(1280): Linear(in_features=288, out_features=316, bias=True)
(1281): ReLU()
(1282): Linear(in_features=316, out_features=288, bias=True)
(1283): ReLU()
(1284): Linear(in_features=288, out_features=312, bias=True)
(1285): ReLU()
(1286): Linear(in_features=312, out_features=288, bias=True)
(1287): ReLU()
(1288): Linear(in_features=288, out_features=304, bias=True)
(1289): ReLU()
(1290): Linear(in_features=304, out_features=288, bias=True)
(1291): ReLU()
(1292): Linear(in_features=288, out_features=256, bias=True)
(1293): ReLU()
(1294): Linear(in_features=256, out_features=336, bias=True)
(1295): ReLU()
(1296): Linear(in_features=336, out_features=296, bias=True)
(1297): ReLU()
(1298): Linear(in_features=296, out_features=340, bias=True)
(1299): ReLU()
(1300): Linear(in_features=340, out_features=332, bias=True)
(1301): ReLU()
(1302): Linear(in_features=332, out_features=375, bias=True)
(1303): ReLU()
(1304): Linear(in_features=375, out_features=399, bias=True)
(1305): ReLU()
(1306): Linear(in_features=399, out_features=410, bias=True)
(1307): ReLU()
(1308): Linear(in_features=410, out_features=402, bias=True)
(1309): ReLU()
(1310): Linear(in_features=402, out_features=412, bias=True)
(1311): ReLU()
(1312): Linear(in_features=412, out_features=404, bias=True)
(1313): ReLU()
(1314): Linear(in_features=404, out_features=412, bias=True)
(1315): ReLU()
(1316): Linear(in_features=412, out_features=404, bias=True)
(1317): ReLU()
(1318): Linear(in_features=404, out_features=408, bias=True)
(1319): ReLU()
(1320): Linear(in_features=408, out_features=400, bias=True)
(1321): ReLU()
(1322): Linear(in_features=400, out_features=352, bias=True)
(1323): ReLU()
(1324): Linear(in_features=352, out_features=288, bias=True)
(1325): ReLU()
(1326): Linear(in_features=288, out_features=288, bias=True)
(1327): ReLU()
(1328): Linear(in_features=288, out_features=256, bias=True)
(1329): ReLU()
(1330): Linear(in_features=256, out_features=319, bias=True)
(1331): ReLU()
(1332): Linear(in_features=319, out_features=288, bias=True)
(1333): ReLU()
(1334): Linear(in_features=288, out_features=318, bias=True)
(1335): ReLU()
(1336): Linear(in_features=318, out_features=288, bias=True)
(1337): ReLU()
(1338): Linear(in_features=288, out_features=316, bias=True)
(1339): ReLU()
(1340): Linear(in_features=316, out_features=288, bias=True)
(1341): ReLU()
(1342): Linear(in_features=288, out_features=312, bias=True)
(1343): ReLU()
(1344): Linear(in_features=312, out_features=288, bias=True)
(1345): ReLU()
(1346): Linear(in_features=288, out_features=304, bias=True)
(1347): ReLU()
(1348): Linear(in_features=304, out_features=288, bias=True)
(1349): ReLU()
(1350): Linear(in_features=288, out_features=256, bias=True)
(1351): ReLU()
(1352): Linear(in_features=256, out_features=288, bias=True)
(1353): ReLU()
(1354): Linear(in_features=288, out_features=256, bias=True)
(1355): ReLU()
(1356): Linear(in_features=256, out_features=319, bias=True)
(1357): ReLU()
(1358): Linear(in_features=319, out_features=288, bias=True)
(1359): ReLU()
(1360): Linear(in_features=288, out_features=318, bias=True)
(1361): ReLU()
(1362): Linear(in_features=318, out_features=288, bias=True)
(1363): ReLU()
(1364): Linear(in_features=288, out_features=316, bias=True)
(1365): ReLU()
(1366): Linear(in_features=316, out_features=288, bias=True)
(1367): ReLU()
(1368): Linear(in_features=288, out_features=312, bias=True)
(1369): ReLU()
(1370): Linear(in_features=312, out_features=288, bias=True)
(1371): ReLU()
(1372): Linear(in_features=288, out_features=304, bias=True)
(1373): ReLU()
(1374): Linear(in_features=304, out_features=288, bias=True)
(1375): ReLU()
(1376): Linear(in_features=288, out_features=256, bias=True)
(1377): ReLU()
(1378): Linear(in_features=256, out_features=336, bias=True)
(1379): ReLU()
(1380): Linear(in_features=336, out_features=296, bias=True)
(1381): ReLU()
(1382): Linear(in_features=296, out_features=340, bias=True)
(1383): ReLU()
(1384): Linear(in_features=340, out_features=332, bias=True)
(1385): ReLU()
(1386): Linear(in_features=332, out_features=375, bias=True)
(1387): ReLU()
(1388): Linear(in_features=375, out_features=399, bias=True)
(1389): ReLU()
(1390): Linear(in_features=399, out_features=410, bias=True)
(1391): ReLU()
(1392): Linear(in_features=410, out_features=402, bias=True)
(1393): ReLU()
(1394): Linear(in_features=402, out_features=412, bias=True)
(1395): ReLU()
(1396): Linear(in_features=412, out_features=404, bias=True)
(1397): ReLU()
(1398): Linear(in_features=404, out_features=412, bias=True)
(1399): ReLU()
(1400): Linear(in_features=412, out_features=404, bias=True)
(1401): ReLU()
(1402): Linear(in_features=404, out_features=408, bias=True)
(1403): ReLU()
(1404): Linear(in_features=408, out_features=400, bias=True)
(1405): ReLU()
(1406): Linear(in_features=400, out_features=352, bias=True)
(1407): ReLU()
(1408): Linear(in_features=352, out_features=288, bias=True)
(1409): ReLU()
(1410): Linear(in_features=288, out_features=288, bias=True)
(1411): ReLU()
(1412): Linear(in_features=288, out_features=256, bias=True)
(1413): ReLU()
(1414): Linear(in_features=256, out_features=319, bias=True)
(1415): ReLU()
(1416): Linear(in_features=319, out_features=288, bias=True)
(1417): ReLU()
(1418): Linear(in_features=288, out_features=318, bias=True)
(1419): ReLU()
(1420): Linear(in_features=318, out_features=288, bias=True)
(1421): ReLU()
(1422): Linear(in_features=288, out_features=316, bias=True)
(1423): ReLU()
(1424): Linear(in_features=316, out_features=288, bias=True)
(1425): ReLU()
(1426): Linear(in_features=288, out_features=312, bias=True)
(1427): ReLU()
(1428): Linear(in_features=312, out_features=288, bias=True)
(1429): ReLU()
(1430): Linear(in_features=288, out_features=304, bias=True)
(1431): ReLU()
(1432): Linear(in_features=304, out_features=288, bias=True)
(1433): ReLU()
(1434): Linear(in_features=288, out_features=256, bias=True)
(1435): ReLU()
(1436): Linear(in_features=256, out_features=288, bias=True)
(1437): ReLU()
(1438): Linear(in_features=288, out_features=256, bias=True)
(1439): ReLU()
(1440): Linear(in_features=256, out_features=319, bias=True)
(1441): ReLU()
(1442): Linear(in_features=319, out_features=288, bias=True)
(1443): ReLU()
(1444): Linear(in_features=288, out_features=318, bias=True)
(1445): ReLU()
(1446): Linear(in_features=318, out_features=288, bias=True)
(1447): ReLU()
(1448): Linear(in_features=288, out_features=316, bias=True)
(1449): ReLU()
(1450): Linear(in_features=316, out_features=288, bias=True)
(1451): ReLU()
(1452): Linear(in_features=288, out_features=312, bias=True)
(1453): ReLU()
(1454): Linear(in_features=312, out_features=288, bias=True)
(1455): ReLU()
(1456): Linear(in_features=288, out_features=304, bias=True)
(1457): ReLU()
(1458): Linear(in_features=304, out_features=288, bias=True)
(1459): ReLU()
(1460): Linear(in_features=288, out_features=256, bias=True)
(1461): ReLU()
(1462): Linear(in_features=256, out_features=336, bias=True)
(1463): ReLU()
(1464): Linear(in_features=336, out_features=296, bias=True)
(1465): ReLU()
(1466): Linear(in_features=296, out_features=340, bias=True)
(1467): ReLU()
(1468): Linear(in_features=340, out_features=332, bias=True)
(1469): ReLU()
(1470): Linear(in_features=332, out_features=375, bias=True)
(1471): ReLU()
(1472): Linear(in_features=375, out_features=399, bias=True)
(1473): ReLU()
(1474): Linear(in_features=399, out_features=410, bias=True)
(1475): ReLU()
(1476): Linear(in_features=410, out_features=402, bias=True)
(1477): ReLU()
(1478): Linear(in_features=402, out_features=412, bias=True)
(1479): ReLU()
(1480): Linear(in_features=412, out_features=404, bias=True)
(1481): ReLU()
(1482): Linear(in_features=404, out_features=412, bias=True)
(1483): ReLU()
(1484): Linear(in_features=412, out_features=404, bias=True)
(1485): ReLU()
(1486): Linear(in_features=404, out_features=408, bias=True)
(1487): ReLU()
(1488): Linear(in_features=408, out_features=400, bias=True)
(1489): ReLU()
(1490): Linear(in_features=400, out_features=352, bias=True)
(1491): ReLU()
(1492): Linear(in_features=352, out_features=288, bias=True)
(1493): ReLU()
(1494): Linear(in_features=288, out_features=288, bias=True)
(1495): ReLU()
(1496): Linear(in_features=288, out_features=256, bias=True)
(1497): ReLU()
(1498): Linear(in_features=256, out_features=319, bias=True)
(1499): ReLU()
(1500): Linear(in_features=319, out_features=288, bias=True)
(1501): ReLU()
(1502): Linear(in_features=288, out_features=318, bias=True)
(1503): ReLU()
(1504): Linear(in_features=318, out_features=288, bias=True)
(1505): ReLU()
(1506): Linear(in_features=288, out_features=316, bias=True)
(1507): ReLU()
(1508): Linear(in_features=316, out_features=288, bias=True)
(1509): ReLU()
(1510): Linear(in_features=288, out_features=312, bias=True)
(1511): ReLU()
(1512): Linear(in_features=312, out_features=288, bias=True)
(1513): ReLU()
(1514): Linear(in_features=288, out_features=304, bias=True)
(1515): ReLU()
(1516): Linear(in_features=304, out_features=288, bias=True)
(1517): ReLU()
(1518): Linear(in_features=288, out_features=256, bias=True)
(1519): ReLU()
(1520): Linear(in_features=256, out_features=288, bias=True)
(1521): ReLU()
(1522): Linear(in_features=288, out_features=256, bias=True)
(1523): ReLU()
(1524): Linear(in_features=256, out_features=319, bias=True)
(1525): ReLU()
(1526): Linear(in_features=319, out_features=288, bias=True)
(1527): ReLU()
(1528): Linear(in_features=288, out_features=318, bias=True)
(1529): ReLU()
(1530): Linear(in_features=318, out_features=288, bias=True)
(1531): ReLU()
(1532): Linear(in_features=288, out_features=316, bias=True)
(1533): ReLU()
(1534): Linear(in_features=316, out_features=288, bias=True)
(1535): ReLU()
(1536): Linear(in_features=288, out_features=312, bias=True)
(1537): ReLU()
(1538): Linear(in_features=312, out_features=288, bias=True)
(1539): ReLU()
(1540): Linear(in_features=288, out_features=304, bias=True)
(1541): ReLU()
(1542): Linear(in_features=304, out_features=288, bias=True)
(1543): ReLU()
(1544): Linear(in_features=288, out_features=256, bias=True)
(1545): ReLU()
(1546): Linear(in_features=256, out_features=336, bias=True)
(1547): ReLU()
(1548): Linear(in_features=336, out_features=296, bias=True)
(1549): ReLU()
(1550): Linear(in_features=296, out_features=340, bias=True)
(1551): ReLU()
(1552): Linear(in_features=340, out_features=332, bias=True)
(1553): ReLU()
(1554): Linear(in_features=332, out_features=375, bias=True)
(1555): ReLU()
(1556): Linear(in_features=375, out_features=399, bias=True)
(1557): ReLU()
(1558): Linear(in_features=399, out_features=410, bias=True)
(1559): ReLU()
(1560): Linear(in_features=410, out_features=402, bias=True)
(1561): ReLU()
(1562): Linear(in_features=402, out_features=412, bias=True)
(1563): ReLU()
(1564): Linear(in_features=412, out_features=404, bias=True)
(1565): ReLU()
(1566): Linear(in_features=404, out_features=412, bias=True)
(1567): ReLU()
(1568): Linear(in_features=412, out_features=404, bias=True)
(1569): ReLU()
(1570): Linear(in_features=404, out_features=408, bias=True)
(1571): ReLU()
(1572): Linear(in_features=408, out_features=400, bias=True)
(1573): ReLU()
(1574): Linear(in_features=400, out_features=352, bias=True)
(1575): ReLU()
(1576): Linear(in_features=352, out_features=288, bias=True)
(1577): ReLU()
(1578): Linear(in_features=288, out_features=288, bias=True)
(1579): ReLU()
(1580): Linear(in_features=288, out_features=256, bias=True)
(1581): ReLU()
(1582): Linear(in_features=256, out_features=319, bias=True)
(1583): ReLU()
(1584): Linear(in_features=319, out_features=288, bias=True)
(1585): ReLU()
(1586): Linear(in_features=288, out_features=318, bias=True)
(1587): ReLU()
(1588): Linear(in_features=318, out_features=288, bias=True)
(1589): ReLU()
(1590): Linear(in_features=288, out_features=316, bias=True)
(1591): ReLU()
(1592): Linear(in_features=316, out_features=288, bias=True)
(1593): ReLU()
(1594): Linear(in_features=288, out_features=312, bias=True)
(1595): ReLU()
(1596): Linear(in_features=312, out_features=288, bias=True)
(1597): ReLU()
(1598): Linear(in_features=288, out_features=304, bias=True)
(1599): ReLU()
(1600): Linear(in_features=304, out_features=288, bias=True)
(1601): ReLU()
(1602): Linear(in_features=288, out_features=256, bias=True)
(1603): ReLU()
(1604): Linear(in_features=256, out_features=288, bias=True)
(1605): ReLU()
(1606): Linear(in_features=288, out_features=256, bias=True)
(1607): ReLU()
(1608): Linear(in_features=256, out_features=319, bias=True)
(1609): ReLU()
(1610): Linear(in_features=319, out_features=288, bias=True)
(1611): ReLU()
(1612): Linear(in_features=288, out_features=318, bias=True)
(1613): ReLU()
(1614): Linear(in_features=318, out_features=288, bias=True)
(1615): ReLU()
(1616): Linear(in_features=288, out_features=316, bias=True)
(1617): ReLU()
(1618): Linear(in_features=316, out_features=288, bias=True)
(1619): ReLU()
(1620): Linear(in_features=288, out_features=312, bias=True)
(1621): ReLU()
(1622): Linear(in_features=312, out_features=288, bias=True)
(1623): ReLU()
(1624): Linear(in_features=288, out_features=304, bias=True)
(1625): ReLU()
(1626): Linear(in_features=304, out_features=288, bias=True)
(1627): ReLU()
(1628): Linear(in_features=288, out_features=256, bias=True)
(1629): ReLU()
(1630): Linear(in_features=256, out_features=336, bias=True)
(1631): ReLU()
(1632): Linear(in_features=336, out_features=296, bias=True)
(1633): ReLU()
(1634): Linear(in_features=296, out_features=340, bias=True)
(1635): ReLU()
(1636): Linear(in_features=340, out_features=332, bias=True)
(1637): ReLU()
(1638): Linear(in_features=332, out_features=375, bias=True)
(1639): ReLU()
(1640): Linear(in_features=375, out_features=399, bias=True)
(1641): ReLU()
(1642): Linear(in_features=399, out_features=410, bias=True)
(1643): ReLU()
(1644): Linear(in_features=410, out_features=402, bias=True)
(1645): ReLU()
(1646): Linear(in_features=402, out_features=412, bias=True)
(1647): ReLU()
(1648): Linear(in_features=412, out_features=404, bias=True)
(1649): ReLU()
(1650): Linear(in_features=404, out_features=412, bias=True)
(1651): ReLU()
(1652): Linear(in_features=412, out_features=404, bias=True)
(1653): ReLU()
(1654): Linear(in_features=404, out_features=408, bias=True)
(1655): ReLU()
(1656): Linear(in_features=408, out_features=400, bias=True)
(1657): ReLU()
(1658): Linear(in_features=400, out_features=352, bias=True)
(1659): ReLU()
(1660): Linear(in_features=352, out_features=288, bias=True)
(1661): ReLU()
(1662): Linear(in_features=288, out_features=288, bias=True)
(1663): ReLU()
(1664): Linear(in_features=288, out_features=256, bias=True)
(1665): ReLU()
(1666): Linear(in_features=256, out_features=319, bias=True)
(1667): ReLU()
(1668): Linear(in_features=319, out_features=288, bias=True)
(1669): ReLU()
(1670): Linear(in_features=288, out_features=318, bias=True)
(1671): ReLU()
(1672): Linear(in_features=318, out_features=288, bias=True)
(1673): ReLU()
(1674): Linear(in_features=288, out_features=316, bias=True)
(1675): ReLU()
(1676): Linear(in_features=316, out_features=288, bias=True)
(1677): ReLU()
(1678): Linear(in_features=288, out_features=312, bias=True)
(1679): ReLU()
(1680): Linear(in_features=312, out_features=288, bias=True)
(1681): ReLU()
(1682): Linear(in_features=288, out_features=304, bias=True)
(1683): ReLU()
(1684): Linear(in_features=304, out_features=288, bias=True)
(1685): ReLU()
(1686): Linear(in_features=288, out_features=256, bias=True)
(1687): ReLU()
(1688): Linear(in_features=256, out_features=288, bias=True)
(1689): ReLU()
(1690): Linear(in_features=288, out_features=256, bias=True)
(1691): ReLU()
(1692): Linear(in_features=256, out_features=319, bias=True)
(1693): ReLU()
(1694): Linear(in_features=319, out_features=288, bias=True)
(1695): ReLU()
(1696): Linear(in_features=288, out_features=318, bias=True)
(1697): ReLU()
(1698): Linear(in_features=318, out_features=288, bias=True)
(1699): ReLU()
(1700): Linear(in_features=288, out_features=316, bias=True)
(1701): ReLU()
(1702): Linear(in_features=316, out_features=288, bias=True)
(1703): ReLU()
(1704): Linear(in_features=288, out_features=312, bias=True)
(1705): ReLU()
(1706): Linear(in_features=312, out_features=288, bias=True)
(1707): ReLU()
(1708): Linear(in_features=288, out_features=304, bias=True)
(1709): ReLU()
(1710): Linear(in_features=304, out_features=288, bias=True)
(1711): ReLU()
(1712): Linear(in_features=288, out_features=256, bias=True)
(1713): ReLU()
(1714): Linear(in_features=256, out_features=336, bias=True)
(1715): ReLU()
(1716): Linear(in_features=336, out_features=296, bias=True)
(1717): ReLU()
(1718): Linear(in_features=296, out_features=340, bias=True)
(1719): ReLU()
(1720): Linear(in_features=340, out_features=332, bias=True)
(1721): ReLU()
(1722): Linear(in_features=332, out_features=375, bias=True)
(1723): ReLU()
(1724): Linear(in_features=375, out_features=399, bias=True)
(1725): ReLU()
(1726): Linear(in_features=399, out_features=410, bias=True)
(1727): ReLU()
(1728): Linear(in_features=410, out_features=402, bias=True)
(1729): ReLU()
(1730): Linear(in_features=402, out_features=412, bias=True)
(1731): ReLU()
(1732): Linear(in_features=412, out_features=404, bias=True)
(1733): ReLU()
(1734): Linear(in_features=404, out_features=412, bias=True)
(1735): ReLU()
(1736): Linear(in_features=412, out_features=404, bias=True)
(1737): ReLU()
(1738): Linear(in_features=404, out_features=408, bias=True)
(1739): ReLU()
(1740): Linear(in_features=408, out_features=400, bias=True)
(1741): ReLU()
(1742): Linear(in_features=400, out_features=352, bias=True)
(1743): ReLU()
(1744): Linear(in_features=352, out_features=288, bias=True)
(1745): ReLU()
(1746): Linear(in_features=288, out_features=288, bias=True)
(1747): ReLU()
(1748): Linear(in_features=288, out_features=256, bias=True)
(1749): ReLU()
(1750): Linear(in_features=256, out_features=319, bias=True)
(1751): ReLU()
(1752): Linear(in_features=319, out_features=288, bias=True)
(1753): ReLU()
(1754): Linear(in_features=288, out_features=318, bias=True)
(1755): ReLU()
(1756): Linear(in_features=318, out_features=288, bias=True)
(1757): ReLU()
(1758): Linear(in_features=288, out_features=316, bias=True)
(1759): ReLU()
(1760): Linear(in_features=316, out_features=288, bias=True)
(1761): ReLU()
(1762): Linear(in_features=288, out_features=312, bias=True)
(1763): ReLU()
(1764): Linear(in_features=312, out_features=288, bias=True)
(1765): ReLU()
(1766): Linear(in_features=288, out_features=304, bias=True)
(1767): ReLU()
(1768): Linear(in_features=304, out_features=288, bias=True)
(1769): ReLU()
(1770): Linear(in_features=288, out_features=256, bias=True)
(1771): ReLU()
(1772): Linear(in_features=256, out_features=288, bias=True)
(1773): ReLU()
(1774): Linear(in_features=288, out_features=256, bias=True)
(1775): ReLU()
(1776): Linear(in_features=256, out_features=319, bias=True)
(1777): ReLU()
(1778): Linear(in_features=319, out_features=288, bias=True)
(1779): ReLU()
(1780): Linear(in_features=288, out_features=318, bias=True)
(1781): ReLU()
(1782): Linear(in_features=318, out_features=288, bias=True)
(1783): ReLU()
(1784): Linear(in_features=288, out_features=316, bias=True)
(1785): ReLU()
(1786): Linear(in_features=316, out_features=288, bias=True)
(1787): ReLU()
(1788): Linear(in_features=288, out_features=312, bias=True)
(1789): ReLU()
(1790): Linear(in_features=312, out_features=288, bias=True)
(1791): ReLU()
(1792): Linear(in_features=288, out_features=304, bias=True)
(1793): ReLU()
(1794): Linear(in_features=304, out_features=288, bias=True)
(1795): ReLU()
(1796): Linear(in_features=288, out_features=256, bias=True)
(1797): ReLU()
(1798): Linear(in_features=256, out_features=336, bias=True)
(1799): ReLU()
(1800): Linear(in_features=336, out_features=296, bias=True)
(1801): ReLU()
(1802): Linear(in_features=296, out_features=340, bias=True)
(1803): ReLU()
(1804): Linear(in_features=340, out_features=332, bias=True)
(1805): ReLU()
(1806): Linear(in_features=332, out_features=375, bias=True)
(1807): ReLU()
(1808): Linear(in_features=375, out_features=399, bias=True)
(1809): ReLU()
(1810): Linear(in_features=399, out_features=410, bias=True)
(1811): ReLU()
(1812): Linear(in_features=410, out_features=402, bias=True)
(1813): ReLU()
(1814): Linear(in_features=402, out_features=412, bias=True)
(1815): ReLU()
(1816): Linear(in_features=412, out_features=404, bias=True)
(1817): ReLU()
(1818): Linear(in_features=404, out_features=412, bias=True)
(1819): ReLU()
(1820): Linear(in_features=412, out_features=404, bias=True)
(1821): ReLU()
(1822): Linear(in_features=404, out_features=408, bias=True)
(1823): ReLU()
(1824): Linear(in_features=408, out_features=400, bias=True)
(1825): ReLU()
(1826): Linear(in_features=400, out_features=352, bias=True)
(1827): ReLU()
(1828): Linear(in_features=352, out_features=288, bias=True)
(1829): ReLU()
(1830): Linear(in_features=288, out_features=288, bias=True)
(1831): ReLU()
(1832): Linear(in_features=288, out_features=256, bias=True)
(1833): ReLU()
(1834): Linear(in_features=256, out_features=319, bias=True)
(1835): ReLU()
(1836): Linear(in_features=319, out_features=288, bias=True)
(1837): ReLU()
(1838): Linear(in_features=288, out_features=318, bias=True)
(1839): ReLU()
(1840): Linear(in_features=318, out_features=288, bias=True)
(1841): ReLU()
(1842): Linear(in_features=288, out_features=316, bias=True)
(1843): ReLU()
(1844): Linear(in_features=316, out_features=288, bias=True)
(1845): ReLU()
(1846): Linear(in_features=288, out_features=312, bias=True)
(1847): ReLU()
(1848): Linear(in_features=312, out_features=288, bias=True)
(1849): ReLU()
(1850): Linear(in_features=288, out_features=304, bias=True)
(1851): ReLU()
(1852): Linear(in_features=304, out_features=288, bias=True)
(1853): ReLU()
(1854): Linear(in_features=288, out_features=256, bias=True)
(1855): ReLU()
(1856): Linear(in_features=256, out_features=288, bias=True)
(1857): ReLU()
(1858): Linear(in_features=288, out_features=256, bias=True)
(1859): ReLU()
(1860): Linear(in_features=256, out_features=319, bias=True)
(1861): ReLU()
(1862): Linear(in_features=319, out_features=288, bias=True)
(1863): ReLU()
(1864): Linear(in_features=288, out_features=318, bias=True)
(1865): ReLU()
(1866): Linear(in_features=318, out_features=288, bias=True)
(1867): ReLU()
(1868): Linear(in_features=288, out_features=316, bias=True)
(1869): ReLU()
(1870): Linear(in_features=316, out_features=288, bias=True)
(1871): ReLU()
(1872): Linear(in_features=288, out_features=312, bias=True)
(1873): ReLU()
(1874): Linear(in_features=312, out_features=288, bias=True)
(1875): ReLU()
(1876): Linear(in_features=288, out_features=304, bias=True)
(1877): ReLU()
(1878): Linear(in_features=304, out_features=288, bias=True)
(1879): ReLU()
(1880): Linear(in_features=288, out_features=256, bias=True)
(1881): ReLU()
(1882): Linear(in_features=256, out_features=336, bias=True)
(1883): ReLU()
(1884): Linear(in_features=336, out_features=296, bias=True)
(1885): ReLU()
(1886): Linear(in_features=296, out_features=340, bias=True)
(1887): ReLU()
(1888): Linear(in_features=340, out_features=332, bias=True)
(1889): ReLU()
(1890): Linear(in_features=332, out_features=375, bias=True)
(1891): ReLU()
(1892): Linear(in_features=375, out_features=399, bias=True)
(1893): ReLU()
(1894): Linear(in_features=399, out_features=410, bias=True)
(1895): ReLU()
(1896): Linear(in_features=410, out_features=402, bias=True)
(1897): ReLU()
(1898): Linear(in_features=402, out_features=412, bias=True)
(1899): ReLU()
(1900): Linear(in_features=412, out_features=404, bias=True)
(1901): ReLU()
(1902): Linear(in_features=404, out_features=412, bias=True)
(1903): ReLU()
(1904): Linear(in_features=412, out_features=404, bias=True)
(1905): ReLU()
(1906): Linear(in_features=404, out_features=408, bias=True)
(1907): ReLU()
(1908): Linear(in_features=408, out_features=400, bias=True)
(1909): ReLU()
(1910): Linear(in_features=400, out_features=352, bias=True)
(1911): ReLU()
(1912): Linear(in_features=352, out_features=288, bias=True)
(1913): ReLU()
(1914): Linear(in_features=288, out_features=288, bias=True)
(1915): ReLU()
(1916): Linear(in_features=288, out_features=256, bias=True)
(1917): ReLU()
(1918): Linear(in_features=256, out_features=319, bias=True)
(1919): ReLU()
(1920): Linear(in_features=319, out_features=288, bias=True)
(1921): ReLU()
(1922): Linear(in_features=288, out_features=318, bias=True)
(1923): ReLU()
(1924): Linear(in_features=318, out_features=288, bias=True)
(1925): ReLU()
(1926): Linear(in_features=288, out_features=316, bias=True)
(1927): ReLU()
(1928): Linear(in_features=316, out_features=288, bias=True)
(1929): ReLU()
(1930): Linear(in_features=288, out_features=312, bias=True)
(1931): ReLU()
(1932): Linear(in_features=312, out_features=288, bias=True)
(1933): ReLU()
(1934): Linear(in_features=288, out_features=304, bias=True)
(1935): ReLU()
(1936): Linear(in_features=304, out_features=288, bias=True)
(1937): ReLU()
(1938): Linear(in_features=288, out_features=256, bias=True)
(1939): ReLU()
(1940): Linear(in_features=256, out_features=288, bias=True)
(1941): ReLU()
(1942): Linear(in_features=288, out_features=256, bias=True)
(1943): ReLU()
(1944): Linear(in_features=256, out_features=319, bias=True)
(1945): ReLU()
(1946): Linear(in_features=319, out_features=288, bias=True)
(1947): ReLU()
(1948): Linear(in_features=288, out_features=318, bias=True)
(1949): ReLU()
(1950): Linear(in_features=318, out_features=288, bias=True)
(1951): ReLU()
(1952): Linear(in_features=288, out_features=316, bias=True)
(1953): ReLU()
(1954): Linear(in_features=316, out_features=288, bias=True)
(1955): ReLU()
(1956): Linear(in_features=288, out_features=312, bias=True)
(1957): ReLU()
(1958): Linear(in_features=312, out_features=288, bias=True)
(1959): ReLU()
(1960): Linear(in_features=288, out_features=304, bias=True)
(1961): ReLU()
(1962): Linear(in_features=304, out_features=288, bias=True)
(1963): ReLU()
(1964): Linear(in_features=288, out_features=256, bias=True)
(1965): ReLU()
(1966): Linear(in_features=256, out_features=336, bias=True)
(1967): ReLU()
(1968): Linear(in_features=336, out_features=296, bias=True)
(1969): ReLU()
(1970): Linear(in_features=296, out_features=340, bias=True)
(1971): ReLU()
(1972): Linear(in_features=340, out_features=332, bias=True)
(1973): ReLU()
(1974): Linear(in_features=332, out_features=375, bias=True)
(1975): ReLU()
(1976): Linear(in_features=375, out_features=399, bias=True)
(1977): ReLU()
(1978): Linear(in_features=399, out_features=410, bias=True)
(1979): ReLU()
(1980): Linear(in_features=410, out_features=402, bias=True)
(1981): ReLU()
(1982): Linear(in_features=402, out_features=412, bias=True)
(1983): ReLU()
(1984): Linear(in_features=412, out_features=404, bias=True)
(1985): ReLU()
(1986): Linear(in_features=404, out_features=412, bias=True)
(1987): ReLU()
(1988): Linear(in_features=412, out_features=404, bias=True)
(1989): ReLU()
(1990): Linear(in_features=404, out_features=408, bias=True)
(1991): ReLU()
(1992): Linear(in_features=408, out_features=400, bias=True)
(1993): ReLU()
(1994): Linear(in_features=400, out_features=352, bias=True)
(1995): ReLU()
(1996): Linear(in_features=352, out_features=288, bias=True)
(1997): ReLU()
(1998): Linear(in_features=288, out_features=288, bias=True)
(1999): ReLU()
(2000): Linear(in_features=288, out_features=256, bias=True)
(2001): ReLU()
(2002): Linear(in_features=256, out_features=319, bias=True)
(2003): ReLU()
(2004): Linear(in_features=319, out_features=288, bias=True)
(2005): ReLU()
(2006): Linear(in_features=288, out_features=318, bias=True)
(2007): ReLU()
(2008): Linear(in_features=318, out_features=288, bias=True)
(2009): ReLU()
(2010): Linear(in_features=288, out_features=316, bias=True)
(2011): ReLU()
(2012): Linear(in_features=316, out_features=288, bias=True)
(2013): ReLU()
(2014): Linear(in_features=288, out_features=312, bias=True)
(2015): ReLU()
(2016): Linear(in_features=312, out_features=288, bias=True)
(2017): ReLU()
(2018): Linear(in_features=288, out_features=304, bias=True)
(2019): ReLU()
(2020): Linear(in_features=304, out_features=288, bias=True)
(2021): ReLU()
(2022): Linear(in_features=288, out_features=256, bias=True)
(2023): ReLU()
(2024): Linear(in_features=256, out_features=288, bias=True)
(2025): ReLU()
(2026): Linear(in_features=288, out_features=256, bias=True)
(2027): ReLU()
(2028): Linear(in_features=256, out_features=319, bias=True)
(2029): ReLU()
(2030): Linear(in_features=319, out_features=288, bias=True)
(2031): ReLU()
(2032): Linear(in_features=288, out_features=318, bias=True)
(2033): ReLU()
(2034): Linear(in_features=318, out_features=288, bias=True)
(2035): ReLU()
(2036): Linear(in_features=288, out_features=316, bias=True)
(2037): ReLU()
(2038): Linear(in_features=316, out_features=288, bias=True)
(2039): ReLU()
(2040): Linear(in_features=288, out_features=312, bias=True)
(2041): ReLU()
(2042): Linear(in_features=312, out_features=288, bias=True)
(2043): ReLU()
(2044): Linear(in_features=288, out_features=304, bias=True)
(2045): ReLU()
(2046): Linear(in_features=304, out_features=288, bias=True)
(2047): ReLU()
(2048): Linear(in_features=288, out_features=256, bias=True)
(2049): ReLU()
(2050): Linear(in_features=256, out_features=336, bias=True)
(2051): ReLU()
(2052): Linear(in_features=336, out_features=296, bias=True)
(2053): ReLU()
(2054): Linear(in_features=296, out_features=340, bias=True)
(2055): ReLU()
(2056): Linear(in_features=340, out_features=332, bias=True)
(2057): ReLU()
(2058): Linear(in_features=332, out_features=375, bias=True)
(2059): ReLU()
(2060): Linear(in_features=375, out_features=399, bias=True)
(2061): ReLU()
(2062): Linear(in_features=399, out_features=410, bias=True)
(2063): ReLU()
(2064): Linear(in_features=410, out_features=402, bias=True)
(2065): ReLU()
(2066): Linear(in_features=402, out_features=412, bias=True)
(2067): ReLU()
(2068): Linear(in_features=412, out_features=404, bias=True)
(2069): ReLU()
(2070): Linear(in_features=404, out_features=412, bias=True)
(2071): ReLU()
(2072): Linear(in_features=412, out_features=404, bias=True)
(2073): ReLU()
(2074): Linear(in_features=404, out_features=408, bias=True)
(2075): ReLU()
(2076): Linear(in_features=408, out_features=400, bias=True)
(2077): ReLU()
(2078): Linear(in_features=400, out_features=352, bias=True)
(2079): ReLU()
(2080): Linear(in_features=352, out_features=288, bias=True)
(2081): ReLU()
(2082): Linear(in_features=288, out_features=288, bias=True)
(2083): ReLU()
(2084): Linear(in_features=288, out_features=256, bias=True)
(2085): ReLU()
(2086): Linear(in_features=256, out_features=319, bias=True)
(2087): ReLU()
(2088): Linear(in_features=319, out_features=288, bias=True)
(2089): ReLU()
(2090): Linear(in_features=288, out_features=318, bias=True)
(2091): ReLU()
(2092): Linear(in_features=318, out_features=288, bias=True)
(2093): ReLU()
(2094): Linear(in_features=288, out_features=316, bias=True)
(2095): ReLU()
(2096): Linear(in_features=316, out_features=288, bias=True)
(2097): ReLU()
(2098): Linear(in_features=288, out_features=312, bias=True)
(2099): ReLU()
(2100): Linear(in_features=312, out_features=288, bias=True)
(2101): ReLU()
(2102): Linear(in_features=288, out_features=304, bias=True)
(2103): ReLU()
(2104): Linear(in_features=304, out_features=288, bias=True)
(2105): ReLU()
(2106): Linear(in_features=288, out_features=256, bias=True)
(2107): ReLU()
(2108): Linear(in_features=256, out_features=288, bias=True)
(2109): ReLU()
(2110): Linear(in_features=288, out_features=256, bias=True)
(2111): ReLU()
(2112): Linear(in_features=256, out_features=319, bias=True)
(2113): ReLU()
(2114): Linear(in_features=319, out_features=288, bias=True)
(2115): ReLU()
(2116): Linear(in_features=288, out_features=318, bias=True)
(2117): ReLU()
(2118): Linear(in_features=318, out_features=288, bias=True)
(2119): ReLU()
(2120): Linear(in_features=288, out_features=316, bias=True)
(2121): ReLU()
(2122): Linear(in_features=316, out_features=288, bias=True)
(2123): ReLU()
(2124): Linear(in_features=288, out_features=312, bias=True)
(2125): ReLU()
(2126): Linear(in_features=312, out_features=288, bias=True)
(2127): ReLU()
(2128): Linear(in_features=288, out_features=304, bias=True)
(2129): ReLU()
(2130): Linear(in_features=304, out_features=288, bias=True)
(2131): ReLU()
(2132): Linear(in_features=288, out_features=256, bias=True)
(2133): ReLU()
(2134): Linear(in_features=256, out_features=336, bias=True)
(2135): ReLU()
(2136): Linear(in_features=336, out_features=296, bias=True)
(2137): ReLU()
(2138): Linear(in_features=296, out_features=340, bias=True)
(2139): ReLU()
(2140): Linear(in_features=340, out_features=332, bias=True)
(2141): ReLU()
(2142): Linear(in_features=332, out_features=375, bias=True)
(2143): ReLU()
(2144): Linear(in_features=375, out_features=399, bias=True)
(2145): ReLU()
(2146): Linear(in_features=399, out_features=410, bias=True)
(2147): ReLU()
(2148): Linear(in_features=410, out_features=402, bias=True)
(2149): ReLU()
(2150): Linear(in_features=402, out_features=412, bias=True)
(2151): ReLU()
(2152): Linear(in_features=412, out_features=404, bias=True)
(2153): ReLU()
(2154): Linear(in_features=404, out_features=412, bias=True)
(2155): ReLU()
(2156): Linear(in_features=412, out_features=404, bias=True)
(2157): ReLU()
(2158): Linear(in_features=404, out_features=408, bias=True)
(2159): ReLU()
(2160): Linear(in_features=408, out_features=400, bias=True)
(2161): ReLU()
(2162): Linear(in_features=400, out_features=352, bias=True)
(2163): ReLU()
(2164): Linear(in_features=352, out_features=288, bias=True)
(2165): ReLU()
(2166): Linear(in_features=288, out_features=288, bias=True)
(2167): ReLU()
(2168): Linear(in_features=288, out_features=256, bias=True)
(2169): ReLU()
(2170): Linear(in_features=256, out_features=319, bias=True)
(2171): ReLU()
(2172): Linear(in_features=319, out_features=288, bias=True)
(2173): ReLU()
(2174): Linear(in_features=288, out_features=318, bias=True)
(2175): ReLU()
(2176): Linear(in_features=318, out_features=288, bias=True)
(2177): ReLU()
(2178): Linear(in_features=288, out_features=316, bias=True)
(2179): ReLU()
(2180): Linear(in_features=316, out_features=288, bias=True)
(2181): ReLU()
(2182): Linear(in_features=288, out_features=312, bias=True)
(2183): ReLU()
(2184): Linear(in_features=312, out_features=288, bias=True)
(2185): ReLU()
(2186): Linear(in_features=288, out_features=304, bias=True)
(2187): ReLU()
(2188): Linear(in_features=304, out_features=288, bias=True)
(2189): ReLU()
(2190): Linear(in_features=288, out_features=256, bias=True)
(2191): ReLU()
(2192): Linear(in_features=256, out_features=288, bias=True)
(2193): ReLU()
(2194): Linear(in_features=288, out_features=256, bias=True)
(2195): ReLU()
(2196): Linear(in_features=256, out_features=319, bias=True)
(2197): ReLU()
(2198): Linear(in_features=319, out_features=288, bias=True)
(2199): ReLU()
(2200): Linear(in_features=288, out_features=318, bias=True)
(2201): ReLU()
(2202): Linear(in_features=318, out_features=288, bias=True)
(2203): ReLU()
(2204): Linear(in_features=288, out_features=316, bias=True)
(2205): ReLU()
(2206): Linear(in_features=316, out_features=288, bias=True)
(2207): ReLU()
(2208): Linear(in_features=288, out_features=312, bias=True)
(2209): ReLU()
(2210): Linear(in_features=312, out_features=288, bias=True)
(2211): ReLU()
(2212): Linear(in_features=288, out_features=304, bias=True)
(2213): ReLU()
(2214): Linear(in_features=304, out_features=288, bias=True)
(2215): ReLU()
(2216): Linear(in_features=288, out_features=256, bias=True)
(2217): ReLU()
(2218): Linear(in_features=256, out_features=336, bias=True)
(2219): ReLU()
(2220): Linear(in_features=336, out_features=296, bias=True)
(2221): ReLU()
(2222): Linear(in_features=296, out_features=340, bias=True)
(2223): ReLU()
(2224): Linear(in_features=340, out_features=332, bias=True)
(2225): ReLU()
(2226): Linear(in_features=332, out_features=375, bias=True)
(2227): ReLU()
(2228): Linear(in_features=375, out_features=399, bias=True)
(2229): ReLU()
(2230): Linear(in_features=399, out_features=410, bias=True)
(2231): ReLU()
(2232): Linear(in_features=410, out_features=402, bias=True)
(2233): ReLU()
(2234): Linear(in_features=402, out_features=412, bias=True)
(2235): ReLU()
(2236): Linear(in_features=412, out_features=404, bias=True)
(2237): ReLU()
(2238): Linear(in_features=404, out_features=412, bias=True)
(2239): ReLU()
(2240): Linear(in_features=412, out_features=404, bias=True)
(2241): ReLU()
(2242): Linear(in_features=404, out_features=408, bias=True)
(2243): ReLU()
(2244): Linear(in_features=408, out_features=400, bias=True)
(2245): ReLU()
(2246): Linear(in_features=400, out_features=352, bias=True)
(2247): ReLU()
(2248): Linear(in_features=352, out_features=288, bias=True)
(2249): ReLU()
(2250): Linear(in_features=288, out_features=288, bias=True)
(2251): ReLU()
(2252): Linear(in_features=288, out_features=256, bias=True)
(2253): ReLU()
(2254): Linear(in_features=256, out_features=319, bias=True)
(2255): ReLU()
(2256): Linear(in_features=319, out_features=288, bias=True)
(2257): ReLU()
(2258): Linear(in_features=288, out_features=318, bias=True)
(2259): ReLU()
(2260): Linear(in_features=318, out_features=288, bias=True)
(2261): ReLU()
(2262): Linear(in_features=288, out_features=316, bias=True)
(2263): ReLU()
(2264): Linear(in_features=316, out_features=288, bias=True)
(2265): ReLU()
(2266): Linear(in_features=288, out_features=312, bias=True)
(2267): ReLU()
(2268): Linear(in_features=312, out_features=288, bias=True)
(2269): ReLU()
(2270): Linear(in_features=288, out_features=304, bias=True)
(2271): ReLU()
(2272): Linear(in_features=304, out_features=288, bias=True)
(2273): ReLU()
(2274): Linear(in_features=288, out_features=256, bias=True)
(2275): ReLU()
(2276): Linear(in_features=256, out_features=288, bias=True)
(2277): ReLU()
(2278): Linear(in_features=288, out_features=256, bias=True)
(2279): ReLU()
(2280): Linear(in_features=256, out_features=319, bias=True)
(2281): ReLU()
(2282): Linear(in_features=319, out_features=288, bias=True)
(2283): ReLU()
(2284): Linear(in_features=288, out_features=318, bias=True)
(2285): ReLU()
(2286): Linear(in_features=318, out_features=288, bias=True)
(2287): ReLU()
(2288): Linear(in_features=288, out_features=316, bias=True)
(2289): ReLU()
(2290): Linear(in_features=316, out_features=288, bias=True)
(2291): ReLU()
(2292): Linear(in_features=288, out_features=312, bias=True)
(2293): ReLU()
(2294): Linear(in_features=312, out_features=288, bias=True)
(2295): ReLU()
(2296): Linear(in_features=288, out_features=304, bias=True)
(2297): ReLU()
(2298): Linear(in_features=304, out_features=288, bias=True)
(2299): ReLU()
(2300): Linear(in_features=288, out_features=256, bias=True)
(2301): ReLU()
(2302): Linear(in_features=256, out_features=336, bias=True)
(2303): ReLU()
(2304): Linear(in_features=336, out_features=296, bias=True)
(2305): ReLU()
(2306): Linear(in_features=296, out_features=340, bias=True)
(2307): ReLU()
(2308): Linear(in_features=340, out_features=332, bias=True)
(2309): ReLU()
(2310): Linear(in_features=332, out_features=375, bias=True)
(2311): ReLU()
(2312): Linear(in_features=375, out_features=399, bias=True)
(2313): ReLU()
(2314): Linear(in_features=399, out_features=410, bias=True)
(2315): ReLU()
(2316): Linear(in_features=410, out_features=402, bias=True)
(2317): ReLU()
(2318): Linear(in_features=402, out_features=412, bias=True)
(2319): ReLU()
(2320): Linear(in_features=412, out_features=404, bias=True)
(2321): ReLU()
(2322): Linear(in_features=404, out_features=412, bias=True)
(2323): ReLU()
(2324): Linear(in_features=412, out_features=404, bias=True)
(2325): ReLU()
(2326): Linear(in_features=404, out_features=408, bias=True)
(2327): ReLU()
(2328): Linear(in_features=408, out_features=400, bias=True)
(2329): ReLU()
(2330): Linear(in_features=400, out_features=352, bias=True)
(2331): ReLU()
(2332): Linear(in_features=352, out_features=288, bias=True)
(2333): ReLU()
(2334): Linear(in_features=288, out_features=288, bias=True)
(2335): ReLU()
(2336): Linear(in_features=288, out_features=256, bias=True)
(2337): ReLU()
(2338): Linear(in_features=256, out_features=319, bias=True)
(2339): ReLU()
(2340): Linear(in_features=319, out_features=288, bias=True)
(2341): ReLU()
(2342): Linear(in_features=288, out_features=318, bias=True)
(2343): ReLU()
(2344): Linear(in_features=318, out_features=288, bias=True)
(2345): ReLU()
(2346): Linear(in_features=288, out_features=316, bias=True)
(2347): ReLU()
(2348): Linear(in_features=316, out_features=288, bias=True)
(2349): ReLU()
(2350): Linear(in_features=288, out_features=312, bias=True)
(2351): ReLU()
(2352): Linear(in_features=312, out_features=288, bias=True)
(2353): ReLU()
(2354): Linear(in_features=288, out_features=304, bias=True)
(2355): ReLU()
(2356): Linear(in_features=304, out_features=288, bias=True)
(2357): ReLU()
(2358): Linear(in_features=288, out_features=256, bias=True)
(2359): ReLU()
(2360): Linear(in_features=256, out_features=288, bias=True)
(2361): ReLU()
(2362): Linear(in_features=288, out_features=256, bias=True)
(2363): ReLU()
(2364): Linear(in_features=256, out_features=319, bias=True)
(2365): ReLU()
(2366): Linear(in_features=319, out_features=288, bias=True)
(2367): ReLU()
(2368): Linear(in_features=288, out_features=318, bias=True)
(2369): ReLU()
(2370): Linear(in_features=318, out_features=288, bias=True)
(2371): ReLU()
(2372): Linear(in_features=288, out_features=316, bias=True)
(2373): ReLU()
(2374): Linear(in_features=316, out_features=288, bias=True)
(2375): ReLU()
(2376): Linear(in_features=288, out_features=312, bias=True)
(2377): ReLU()
(2378): Linear(in_features=312, out_features=288, bias=True)
(2379): ReLU()
(2380): Linear(in_features=288, out_features=304, bias=True)
(2381): ReLU()
(2382): Linear(in_features=304, out_features=288, bias=True)
(2383): ReLU()
(2384): Linear(in_features=288, out_features=256, bias=True)
(2385): ReLU()
(2386): Linear(in_features=256, out_features=336, bias=True)
(2387): ReLU()
(2388): Linear(in_features=336, out_features=296, bias=True)
(2389): ReLU()
(2390): Linear(in_features=296, out_features=340, bias=True)
(2391): ReLU()
(2392): Linear(in_features=340, out_features=332, bias=True)
(2393): ReLU()
(2394): Linear(in_features=332, out_features=375, bias=True)
(2395): ReLU()
(2396): Linear(in_features=375, out_features=399, bias=True)
(2397): ReLU()
(2398): Linear(in_features=399, out_features=410, bias=True)
(2399): ReLU()
(2400): Linear(in_features=410, out_features=402, bias=True)
(2401): ReLU()
(2402): Linear(in_features=402, out_features=412, bias=True)
(2403): ReLU()
(2404): Linear(in_features=412, out_features=404, bias=True)
(2405): ReLU()
(2406): Linear(in_features=404, out_features=412, bias=True)
(2407): ReLU()
(2408): Linear(in_features=412, out_features=404, bias=True)
(2409): ReLU()
(2410): Linear(in_features=404, out_features=408, bias=True)
(2411): ReLU()
(2412): Linear(in_features=408, out_features=400, bias=True)
(2413): ReLU()
(2414): Linear(in_features=400, out_features=352, bias=True)
(2415): ReLU()
(2416): Linear(in_features=352, out_features=288, bias=True)
(2417): ReLU()
(2418): Linear(in_features=288, out_features=288, bias=True)
(2419): ReLU()
(2420): Linear(in_features=288, out_features=256, bias=True)
(2421): ReLU()
(2422): Linear(in_features=256, out_features=319, bias=True)
(2423): ReLU()
(2424): Linear(in_features=319, out_features=288, bias=True)
(2425): ReLU()
(2426): Linear(in_features=288, out_features=318, bias=True)
(2427): ReLU()
(2428): Linear(in_features=318, out_features=288, bias=True)
(2429): ReLU()
(2430): Linear(in_features=288, out_features=316, bias=True)
(2431): ReLU()
(2432): Linear(in_features=316, out_features=288, bias=True)
(2433): ReLU()
(2434): Linear(in_features=288, out_features=312, bias=True)
(2435): ReLU()
(2436): Linear(in_features=312, out_features=288, bias=True)
(2437): ReLU()
(2438): Linear(in_features=288, out_features=304, bias=True)
(2439): ReLU()
(2440): Linear(in_features=304, out_features=288, bias=True)
(2441): ReLU()
(2442): Linear(in_features=288, out_features=256, bias=True)
(2443): ReLU()
(2444): Linear(in_features=256, out_features=288, bias=True)
(2445): ReLU()
(2446): Linear(in_features=288, out_features=256, bias=True)
(2447): ReLU()
(2448): Linear(in_features=256, out_features=319, bias=True)
(2449): ReLU()
(2450): Linear(in_features=319, out_features=288, bias=True)
(2451): ReLU()
(2452): Linear(in_features=288, out_features=318, bias=True)
(2453): ReLU()
(2454): Linear(in_features=318, out_features=288, bias=True)
(2455): ReLU()
(2456): Linear(in_features=288, out_features=316, bias=True)
(2457): ReLU()
(2458): Linear(in_features=316, out_features=288, bias=True)
(2459): ReLU()
(2460): Linear(in_features=288, out_features=312, bias=True)
(2461): ReLU()
(2462): Linear(in_features=312, out_features=288, bias=True)
(2463): ReLU()
(2464): Linear(in_features=288, out_features=304, bias=True)
(2465): ReLU()
(2466): Linear(in_features=304, out_features=288, bias=True)
(2467): ReLU()
(2468): Linear(in_features=288, out_features=256, bias=True)
(2469): ReLU()
(2470): Linear(in_features=256, out_features=336, bias=True)
(2471): ReLU()
(2472): Linear(in_features=336, out_features=296, bias=True)
(2473): ReLU()
(2474): Linear(in_features=296, out_features=340, bias=True)
(2475): ReLU()
(2476): Linear(in_features=340, out_features=332, bias=True)
(2477): ReLU()
(2478): Linear(in_features=332, out_features=375, bias=True)
(2479): ReLU()
(2480): Linear(in_features=375, out_features=399, bias=True)
(2481): ReLU()
(2482): Linear(in_features=399, out_features=410, bias=True)
(2483): ReLU()
(2484): Linear(in_features=410, out_features=402, bias=True)
(2485): ReLU()
(2486): Linear(in_features=402, out_features=412, bias=True)
(2487): ReLU()
(2488): Linear(in_features=412, out_features=404, bias=True)
(2489): ReLU()
(2490): Linear(in_features=404, out_features=412, bias=True)
(2491): ReLU()
(2492): Linear(in_features=412, out_features=404, bias=True)
(2493): ReLU()
(2494): Linear(in_features=404, out_features=408, bias=True)
(2495): ReLU()
(2496): Linear(in_features=408, out_features=400, bias=True)
(2497): ReLU()
(2498): Linear(in_features=400, out_features=352, bias=True)
(2499): ReLU()
(2500): Linear(in_features=352, out_features=288, bias=True)
(2501): ReLU()
(2502): Linear(in_features=288, out_features=288, bias=True)
(2503): ReLU()
(2504): Linear(in_features=288, out_features=256, bias=True)
(2505): ReLU()
(2506): Linear(in_features=256, out_features=319, bias=True)
(2507): ReLU()
(2508): Linear(in_features=319, out_features=288, bias=True)
(2509): ReLU()
(2510): Linear(in_features=288, out_features=318, bias=True)
(2511): ReLU()
(2512): Linear(in_features=318, out_features=288, bias=True)
(2513): ReLU()
(2514): Linear(in_features=288, out_features=316, bias=True)
(2515): ReLU()
(2516): Linear(in_features=316, out_features=288, bias=True)
(2517): ReLU()
(2518): Linear(in_features=288, out_features=312, bias=True)
(2519): ReLU()
(2520): Linear(in_features=312, out_features=288, bias=True)
(2521): ReLU()
(2522): Linear(in_features=288, out_features=304, bias=True)
(2523): ReLU()
(2524): Linear(in_features=304, out_features=288, bias=True)
(2525): ReLU()
(2526): Linear(in_features=288, out_features=256, bias=True)
(2527): ReLU()
(2528): Linear(in_features=256, out_features=288, bias=True)
(2529): ReLU()
(2530): Linear(in_features=288, out_features=256, bias=True)
(2531): ReLU()
(2532): Linear(in_features=256, out_features=319, bias=True)
(2533): ReLU()
(2534): Linear(in_features=319, out_features=288, bias=True)
(2535): ReLU()
(2536): Linear(in_features=288, out_features=318, bias=True)
(2537): ReLU()
(2538): Linear(in_features=318, out_features=288, bias=True)
(2539): ReLU()
(2540): Linear(in_features=288, out_features=316, bias=True)
(2541): ReLU()
(2542): Linear(in_features=316, out_features=288, bias=True)
(2543): ReLU()
(2544): Linear(in_features=288, out_features=312, bias=True)
(2545): ReLU()
(2546): Linear(in_features=312, out_features=288, bias=True)
(2547): ReLU()
(2548): Linear(in_features=288, out_features=304, bias=True)
(2549): ReLU()
(2550): Linear(in_features=304, out_features=288, bias=True)
(2551): ReLU()
(2552): Linear(in_features=288, out_features=256, bias=True)
(2553): ReLU()
(2554): Linear(in_features=256, out_features=336, bias=True)
(2555): ReLU()
(2556): Linear(in_features=336, out_features=296, bias=True)
(2557): ReLU()
(2558): Linear(in_features=296, out_features=340, bias=True)
(2559): ReLU()
(2560): Linear(in_features=340, out_features=332, bias=True)
(2561): ReLU()
(2562): Linear(in_features=332, out_features=375, bias=True)
(2563): ReLU()
(2564): Linear(in_features=375, out_features=399, bias=True)
(2565): ReLU()
(2566): Linear(in_features=399, out_features=410, bias=True)
(2567): ReLU()
(2568): Linear(in_features=410, out_features=402, bias=True)
(2569): ReLU()
(2570): Linear(in_features=402, out_features=412, bias=True)
(2571): ReLU()
(2572): Linear(in_features=412, out_features=404, bias=True)
(2573): ReLU()
(2574): Linear(in_features=404, out_features=412, bias=True)
(2575): ReLU()
(2576): Linear(in_features=412, out_features=404, bias=True)
(2577): ReLU()
(2578): Linear(in_features=404, out_features=408, bias=True)
(2579): ReLU()
(2580): Linear(in_features=408, out_features=400, bias=True)
(2581): ReLU()
(2582): Linear(in_features=400, out_features=352, bias=True)
(2583): ReLU()
(2584): Linear(in_features=352, out_features=288, bias=True)
(2585): ReLU()
(2586): Linear(in_features=288, out_features=288, bias=True)
(2587): ReLU()
(2588): Linear(in_features=288, out_features=256, bias=True)
(2589): ReLU()
(2590): Linear(in_features=256, out_features=319, bias=True)
(2591): ReLU()
(2592): Linear(in_features=319, out_features=288, bias=True)
(2593): ReLU()
(2594): Linear(in_features=288, out_features=318, bias=True)
(2595): ReLU()
(2596): Linear(in_features=318, out_features=288, bias=True)
(2597): ReLU()
(2598): Linear(in_features=288, out_features=316, bias=True)
(2599): ReLU()
(2600): Linear(in_features=316, out_features=288, bias=True)
(2601): ReLU()
(2602): Linear(in_features=288, out_features=312, bias=True)
(2603): ReLU()
(2604): Linear(in_features=312, out_features=288, bias=True)
(2605): ReLU()
(2606): Linear(in_features=288, out_features=304, bias=True)
(2607): ReLU()
(2608): Linear(in_features=304, out_features=288, bias=True)
(2609): ReLU()
(2610): Linear(in_features=288, out_features=256, bias=True)
(2611): ReLU()
(2612): Linear(in_features=256, out_features=288, bias=True)
(2613): ReLU()
(2614): Linear(in_features=288, out_features=256, bias=True)
(2615): ReLU()
(2616): Linear(in_features=256, out_features=319, bias=True)
(2617): ReLU()
(2618): Linear(in_features=319, out_features=288, bias=True)
(2619): ReLU()
(2620): Linear(in_features=288, out_features=318, bias=True)
(2621): ReLU()
(2622): Linear(in_features=318, out_features=288, bias=True)
(2623): ReLU()
(2624): Linear(in_features=288, out_features=316, bias=True)
(2625): ReLU()
(2626): Linear(in_features=316, out_features=288, bias=True)
(2627): ReLU()
(2628): Linear(in_features=288, out_features=312, bias=True)
(2629): ReLU()
(2630): Linear(in_features=312, out_features=288, bias=True)
(2631): ReLU()
(2632): Linear(in_features=288, out_features=304, bias=True)
(2633): ReLU()
(2634): Linear(in_features=304, out_features=288, bias=True)
(2635): ReLU()
(2636): Linear(in_features=288, out_features=256, bias=True)
(2637): ReLU()
(2638): Linear(in_features=256, out_features=336, bias=True)
(2639): ReLU()
(2640): Linear(in_features=336, out_features=296, bias=True)
(2641): ReLU()
(2642): Linear(in_features=296, out_features=340, bias=True)
(2643): ReLU()
(2644): Linear(in_features=340, out_features=332, bias=True)
(2645): ReLU()
(2646): Linear(in_features=332, out_features=375, bias=True)
(2647): ReLU()
(2648): Linear(in_features=375, out_features=399, bias=True)
(2649): ReLU()
(2650): Linear(in_features=399, out_features=410, bias=True)
(2651): ReLU()
(2652): Linear(in_features=410, out_features=402, bias=True)
(2653): ReLU()
(2654): Linear(in_features=402, out_features=412, bias=True)
(2655): ReLU()
(2656): Linear(in_features=412, out_features=404, bias=True)
(2657): ReLU()
(2658): Linear(in_features=404, out_features=412, bias=True)
(2659): ReLU()
(2660): Linear(in_features=412, out_features=404, bias=True)
(2661): ReLU()
(2662): Linear(in_features=404, out_features=408, bias=True)
(2663): ReLU()
(2664): Linear(in_features=408, out_features=400, bias=True)
(2665): ReLU()
(2666): Linear(in_features=400, out_features=352, bias=True)
(2667): ReLU()
(2668): Linear(in_features=352, out_features=288, bias=True)
(2669): ReLU()
(2670): Linear(in_features=288, out_features=288, bias=True)
(2671): ReLU()
(2672): Linear(in_features=288, out_features=256, bias=True)
(2673): ReLU()
(2674): Linear(in_features=256, out_features=319, bias=True)
(2675): ReLU()
(2676): Linear(in_features=319, out_features=288, bias=True)
(2677): ReLU()
(2678): Linear(in_features=288, out_features=318, bias=True)
(2679): ReLU()
(2680): Linear(in_features=318, out_features=288, bias=True)
(2681): ReLU()
(2682): Linear(in_features=288, out_features=316, bias=True)
(2683): ReLU()
(2684): Linear(in_features=316, out_features=288, bias=True)
(2685): ReLU()
(2686): Linear(in_features=288, out_features=312, bias=True)
(2687): ReLU()
(2688): Linear(in_features=312, out_features=288, bias=True)
(2689): ReLU()
(2690): Linear(in_features=288, out_features=304, bias=True)
(2691): ReLU()
(2692): Linear(in_features=304, out_features=288, bias=True)
(2693): ReLU()
(2694): Linear(in_features=288, out_features=256, bias=True)
(2695): ReLU()
(2696): Linear(in_features=256, out_features=288, bias=True)
(2697): ReLU()
(2698): Linear(in_features=288, out_features=256, bias=True)
(2699): ReLU()
(2700): Linear(in_features=256, out_features=319, bias=True)
(2701): ReLU()
(2702): Linear(in_features=319, out_features=288, bias=True)
(2703): ReLU()
(2704): Linear(in_features=288, out_features=318, bias=True)
(2705): ReLU()
(2706): Linear(in_features=318, out_features=288, bias=True)
(2707): ReLU()
(2708): Linear(in_features=288, out_features=316, bias=True)
(2709): ReLU()
(2710): Linear(in_features=316, out_features=288, bias=True)
(2711): ReLU()
(2712): Linear(in_features=288, out_features=312, bias=True)
(2713): ReLU()
(2714): Linear(in_features=312, out_features=288, bias=True)
(2715): ReLU()
(2716): Linear(in_features=288, out_features=304, bias=True)
(2717): ReLU()
(2718): Linear(in_features=304, out_features=288, bias=True)
(2719): ReLU()
(2720): Linear(in_features=288, out_features=256, bias=True)
(2721): ReLU()
(2722): Linear(in_features=256, out_features=368, bias=True)
(2723): ReLU()
(2724): Linear(in_features=368, out_features=328, bias=True)
(2725): ReLU()
(2726): Linear(in_features=328, out_features=340, bias=True)
(2727): ReLU()
(2728): Linear(in_features=340, out_features=332, bias=True)
(2729): ReLU()
(2730): Linear(in_features=332, out_features=375, bias=True)
(2731): ReLU()
(2732): Linear(in_features=375, out_features=399, bias=True)
(2733): ReLU()
(2734): Linear(in_features=399, out_features=410, bias=True)
(2735): ReLU()
(2736): Linear(in_features=410, out_features=402, bias=True)
(2737): ReLU()
(2738): Linear(in_features=402, out_features=412, bias=True)
(2739): ReLU()
(2740): Linear(in_features=412, out_features=404, bias=True)
(2741): ReLU()
(2742): Linear(in_features=404, out_features=412, bias=True)
(2743): ReLU()
(2744): Linear(in_features=412, out_features=404, bias=True)
(2745): ReLU()
(2746): Linear(in_features=404, out_features=408, bias=True)
(2747): ReLU()
(2748): Linear(in_features=408, out_features=400, bias=True)
(2749): ReLU()
(2750): Linear(in_features=400, out_features=352, bias=True)
(2751): ReLU()
(2752): Linear(in_features=352, out_features=288, bias=True)
(2753): ReLU()
(2754): Linear(in_features=288, out_features=288, bias=True)
(2755): ReLU()
(2756): Linear(in_features=288, out_features=256, bias=True)
(2757): ReLU()
(2758): Linear(in_features=256, out_features=319, bias=True)
(2759): ReLU()
(2760): Linear(in_features=319, out_features=288, bias=True)
(2761): ReLU()
(2762): Linear(in_features=288, out_features=318, bias=True)
(2763): ReLU()
(2764): Linear(in_features=318, out_features=288, bias=True)
(2765): ReLU()
(2766): Linear(in_features=288, out_features=316, bias=True)
(2767): ReLU()
(2768): Linear(in_features=316, out_features=288, bias=True)
(2769): ReLU()
(2770): Linear(in_features=288, out_features=312, bias=True)
(2771): ReLU()
(2772): Linear(in_features=312, out_features=288, bias=True)
(2773): ReLU()
(2774): Linear(in_features=288, out_features=304, bias=True)
(2775): ReLU()
(2776): Linear(in_features=304, out_features=288, bias=True)
(2777): ReLU()
(2778): Linear(in_features=288, out_features=256, bias=True)
(2779): ReLU()
(2780): Linear(in_features=256, out_features=288, bias=True)
(2781): ReLU()
(2782): Linear(in_features=288, out_features=256, bias=True)
(2783): ReLU()
(2784): Linear(in_features=256, out_features=319, bias=True)
(2785): ReLU()
(2786): Linear(in_features=319, out_features=288, bias=True)
(2787): ReLU()
(2788): Linear(in_features=288, out_features=318, bias=True)
(2789): ReLU()
(2790): Linear(in_features=318, out_features=288, bias=True)
(2791): ReLU()
(2792): Linear(in_features=288, out_features=316, bias=True)
(2793): ReLU()
(2794): Linear(in_features=316, out_features=288, bias=True)
(2795): ReLU()
(2796): Linear(in_features=288, out_features=312, bias=True)
(2797): ReLU()
(2798): Linear(in_features=312, out_features=288, bias=True)
(2799): ReLU()
(2800): Linear(in_features=288, out_features=304, bias=True)
(2801): ReLU()
(2802): Linear(in_features=304, out_features=288, bias=True)
(2803): ReLU()
(2804): Linear(in_features=288, out_features=256, bias=True)
(2805): ReLU()
(2806): Linear(in_features=256, out_features=368, bias=True)
(2807): ReLU()
(2808): Linear(in_features=368, out_features=328, bias=True)
(2809): ReLU()
(2810): Linear(in_features=328, out_features=340, bias=True)
(2811): ReLU()
(2812): Linear(in_features=340, out_features=332, bias=True)
(2813): ReLU()
(2814): Linear(in_features=332, out_features=375, bias=True)
(2815): ReLU()
(2816): Linear(in_features=375, out_features=399, bias=True)
(2817): ReLU()
(2818): Linear(in_features=399, out_features=410, bias=True)
(2819): ReLU()
(2820): Linear(in_features=410, out_features=402, bias=True)
(2821): ReLU()
(2822): Linear(in_features=402, out_features=412, bias=True)
(2823): ReLU()
(2824): Linear(in_features=412, out_features=404, bias=True)
(2825): ReLU()
(2826): Linear(in_features=404, out_features=412, bias=True)
(2827): ReLU()
(2828): Linear(in_features=412, out_features=404, bias=True)
(2829): ReLU()
(2830): Linear(in_features=404, out_features=408, bias=True)
(2831): ReLU()
(2832): Linear(in_features=408, out_features=400, bias=True)
(2833): ReLU()
(2834): Linear(in_features=400, out_features=352, bias=True)
(2835): ReLU()
(2836): Linear(in_features=352, out_features=288, bias=True)
(2837): ReLU()
(2838): Linear(in_features=288, out_features=288, bias=True)
(2839): ReLU()
(2840): Linear(in_features=288, out_features=256, bias=True)
(2841): ReLU()
(2842): Linear(in_features=256, out_features=319, bias=True)
(2843): ReLU()
(2844): Linear(in_features=319, out_features=288, bias=True)
(2845): ReLU()
(2846): Linear(in_features=288, out_features=318, bias=True)
(2847): ReLU()
(2848): Linear(in_features=318, out_features=288, bias=True)
(2849): ReLU()
(2850): Linear(in_features=288, out_features=316, bias=True)
(2851): ReLU()
(2852): Linear(in_features=316, out_features=288, bias=True)
(2853): ReLU()
(2854): Linear(in_features=288, out_features=312, bias=True)
(2855): ReLU()
(2856): Linear(in_features=312, out_features=288, bias=True)
(2857): ReLU()
(2858): Linear(in_features=288, out_features=304, bias=True)
(2859): ReLU()
(2860): Linear(in_features=304, out_features=288, bias=True)
(2861): ReLU()
(2862): Linear(in_features=288, out_features=256, bias=True)
(2863): ReLU()
(2864): Linear(in_features=256, out_features=288, bias=True)
(2865): ReLU()
(2866): Linear(in_features=288, out_features=256, bias=True)
(2867): ReLU()
(2868): Linear(in_features=256, out_features=319, bias=True)
(2869): ReLU()
(2870): Linear(in_features=319, out_features=288, bias=True)
(2871): ReLU()
(2872): Linear(in_features=288, out_features=318, bias=True)
(2873): ReLU()
(2874): Linear(in_features=318, out_features=288, bias=True)
(2875): ReLU()
(2876): Linear(in_features=288, out_features=316, bias=True)
(2877): ReLU()
(2878): Linear(in_features=316, out_features=288, bias=True)
(2879): ReLU()
(2880): Linear(in_features=288, out_features=312, bias=True)
(2881): ReLU()
(2882): Linear(in_features=312, out_features=288, bias=True)
(2883): ReLU()
(2884): Linear(in_features=288, out_features=304, bias=True)
(2885): ReLU()
(2886): Linear(in_features=304, out_features=288, bias=True)
(2887): ReLU()
(2888): Linear(in_features=288, out_features=256, bias=True)
(2889): ReLU()
(2890): Linear(in_features=256, out_features=368, bias=True)
(2891): ReLU()
(2892): Linear(in_features=368, out_features=328, bias=True)
(2893): ReLU()
(2894): Linear(in_features=328, out_features=340, bias=True)
(2895): ReLU()
(2896): Linear(in_features=340, out_features=332, bias=True)
(2897): ReLU()
(2898): Linear(in_features=332, out_features=375, bias=True)
(2899): ReLU()
(2900): Linear(in_features=375, out_features=399, bias=True)
(2901): ReLU()
(2902): Linear(in_features=399, out_features=410, bias=True)
(2903): ReLU()
(2904): Linear(in_features=410, out_features=402, bias=True)
(2905): ReLU()
(2906): Linear(in_features=402, out_features=412, bias=True)
(2907): ReLU()
(2908): Linear(in_features=412, out_features=404, bias=True)
(2909): ReLU()
(2910): Linear(in_features=404, out_features=412, bias=True)
(2911): ReLU()
(2912): Linear(in_features=412, out_features=404, bias=True)
(2913): ReLU()
(2914): Linear(in_features=404, out_features=408, bias=True)
(2915): ReLU()
(2916): Linear(in_features=408, out_features=400, bias=True)
(2917): ReLU()
(2918): Linear(in_features=400, out_features=352, bias=True)
(2919): ReLU()
(2920): Linear(in_features=352, out_features=288, bias=True)
(2921): ReLU()
(2922): Linear(in_features=288, out_features=288, bias=True)
(2923): ReLU()
(2924): Linear(in_features=288, out_features=256, bias=True)
(2925): ReLU()
(2926): Linear(in_features=256, out_features=319, bias=True)
(2927): ReLU()
(2928): Linear(in_features=319, out_features=288, bias=True)
(2929): ReLU()
(2930): Linear(in_features=288, out_features=318, bias=True)
(2931): ReLU()
(2932): Linear(in_features=318, out_features=288, bias=True)
(2933): ReLU()
(2934): Linear(in_features=288, out_features=316, bias=True)
(2935): ReLU()
(2936): Linear(in_features=316, out_features=288, bias=True)
(2937): ReLU()
(2938): Linear(in_features=288, out_features=312, bias=True)
(2939): ReLU()
(2940): Linear(in_features=312, out_features=288, bias=True)
(2941): ReLU()
(2942): Linear(in_features=288, out_features=304, bias=True)
(2943): ReLU()
(2944): Linear(in_features=304, out_features=288, bias=True)
(2945): ReLU()
(2946): Linear(in_features=288, out_features=256, bias=True)
(2947): ReLU()
(2948): Linear(in_features=256, out_features=288, bias=True)
(2949): ReLU()
(2950): Linear(in_features=288, out_features=256, bias=True)
(2951): ReLU()
(2952): Linear(in_features=256, out_features=319, bias=True)
(2953): ReLU()
(2954): Linear(in_features=319, out_features=288, bias=True)
(2955): ReLU()
(2956): Linear(in_features=288, out_features=318, bias=True)
(2957): ReLU()
(2958): Linear(in_features=318, out_features=288, bias=True)
(2959): ReLU()
(2960): Linear(in_features=288, out_features=316, bias=True)
(2961): ReLU()
(2962): Linear(in_features=316, out_features=288, bias=True)
(2963): ReLU()
(2964): Linear(in_features=288, out_features=312, bias=True)
(2965): ReLU()
(2966): Linear(in_features=312, out_features=288, bias=True)
(2967): ReLU()
(2968): Linear(in_features=288, out_features=304, bias=True)
(2969): ReLU()
(2970): Linear(in_features=304, out_features=288, bias=True)
(2971): ReLU()
(2972): Linear(in_features=288, out_features=256, bias=True)
(2973): ReLU()
(2974): Linear(in_features=256, out_features=368, bias=True)
(2975): ReLU()
(2976): Linear(in_features=368, out_features=328, bias=True)
(2977): ReLU()
(2978): Linear(in_features=328, out_features=340, bias=True)
(2979): ReLU()
(2980): Linear(in_features=340, out_features=332, bias=True)
(2981): ReLU()
(2982): Linear(in_features=332, out_features=375, bias=True)
(2983): ReLU()
(2984): Linear(in_features=375, out_features=399, bias=True)
(2985): ReLU()
(2986): Linear(in_features=399, out_features=410, bias=True)
(2987): ReLU()
(2988): Linear(in_features=410, out_features=402, bias=True)
(2989): ReLU()
(2990): Linear(in_features=402, out_features=412, bias=True)
(2991): ReLU()
(2992): Linear(in_features=412, out_features=404, bias=True)
(2993): ReLU()
(2994): Linear(in_features=404, out_features=412, bias=True)
(2995): ReLU()
(2996): Linear(in_features=412, out_features=404, bias=True)
(2997): ReLU()
(2998): Linear(in_features=404, out_features=408, bias=True)
(2999): ReLU()
(3000): Linear(in_features=408, out_features=400, bias=True)
(3001): ReLU()
(3002): Linear(in_features=400, out_features=352, bias=True)
(3003): ReLU()
(3004): Linear(in_features=352, out_features=288, bias=True)
(3005): ReLU()
(3006): Linear(in_features=288, out_features=288, bias=True)
(3007): ReLU()
(3008): Linear(in_features=288, out_features=256, bias=True)
(3009): ReLU()
(3010): Linear(in_features=256, out_features=319, bias=True)
(3011): ReLU()
(3012): Linear(in_features=319, out_features=288, bias=True)
(3013): ReLU()
(3014): Linear(in_features=288, out_features=318, bias=True)
(3015): ReLU()
(3016): Linear(in_features=318, out_features=288, bias=True)
(3017): ReLU()
(3018): Linear(in_features=288, out_features=316, bias=True)
(3019): ReLU()
(3020): Linear(in_features=316, out_features=288, bias=True)
(3021): ReLU()
(3022): Linear(in_features=288, out_features=312, bias=True)
(3023): ReLU()
(3024): Linear(in_features=312, out_features=288, bias=True)
(3025): ReLU()
(3026): Linear(in_features=288, out_features=304, bias=True)
(3027): ReLU()
(3028): Linear(in_features=304, out_features=288, bias=True)
(3029): ReLU()
(3030): Linear(in_features=288, out_features=256, bias=True)
(3031): ReLU()
(3032): Linear(in_features=256, out_features=288, bias=True)
(3033): ReLU()
(3034): Linear(in_features=288, out_features=256, bias=True)
(3035): ReLU()
(3036): Linear(in_features=256, out_features=319, bias=True)
(3037): ReLU()
(3038): Linear(in_features=319, out_features=288, bias=True)
(3039): ReLU()
(3040): Linear(in_features=288, out_features=318, bias=True)
(3041): ReLU()
(3042): Linear(in_features=318, out_features=288, bias=True)
(3043): ReLU()
(3044): Linear(in_features=288, out_features=316, bias=True)
(3045): ReLU()
(3046): Linear(in_features=316, out_features=288, bias=True)
(3047): ReLU()
(3048): Linear(in_features=288, out_features=312, bias=True)
(3049): ReLU()
(3050): Linear(in_features=312, out_features=288, bias=True)
(3051): ReLU()
(3052): Linear(in_features=288, out_features=304, bias=True)
(3053): ReLU()
(3054): Linear(in_features=304, out_features=288, bias=True)
(3055): ReLU()
(3056): Linear(in_features=288, out_features=256, bias=True)
(3057): ReLU()
(3058): Linear(in_features=256, out_features=368, bias=True)
(3059): ReLU()
(3060): Linear(in_features=368, out_features=328, bias=True)
(3061): ReLU()
(3062): Linear(in_features=328, out_features=340, bias=True)
(3063): ReLU()
(3064): Linear(in_features=340, out_features=332, bias=True)
(3065): ReLU()
(3066): Linear(in_features=332, out_features=375, bias=True)
(3067): ReLU()
(3068): Linear(in_features=375, out_features=399, bias=True)
(3069): ReLU()
(3070): Linear(in_features=399, out_features=410, bias=True)
(3071): ReLU()
(3072): Linear(in_features=410, out_features=402, bias=True)
(3073): ReLU()
(3074): Linear(in_features=402, out_features=412, bias=True)
(3075): ReLU()
(3076): Linear(in_features=412, out_features=404, bias=True)
(3077): ReLU()
(3078): Linear(in_features=404, out_features=412, bias=True)
(3079): ReLU()
(3080): Linear(in_features=412, out_features=404, bias=True)
(3081): ReLU()
(3082): Linear(in_features=404, out_features=408, bias=True)
(3083): ReLU()
(3084): Linear(in_features=408, out_features=400, bias=True)
(3085): ReLU()
(3086): Linear(in_features=400, out_features=352, bias=True)
(3087): ReLU()
(3088): Linear(in_features=352, out_features=288, bias=True)
(3089): ReLU()
(3090): Linear(in_features=288, out_features=288, bias=True)
(3091): ReLU()
(3092): Linear(in_features=288, out_features=256, bias=True)
(3093): ReLU()
(3094): Linear(in_features=256, out_features=319, bias=True)
(3095): ReLU()
(3096): Linear(in_features=319, out_features=288, bias=True)
(3097): ReLU()
(3098): Linear(in_features=288, out_features=318, bias=True)
(3099): ReLU()
(3100): Linear(in_features=318, out_features=288, bias=True)
(3101): ReLU()
(3102): Linear(in_features=288, out_features=316, bias=True)
(3103): ReLU()
(3104): Linear(in_features=316, out_features=288, bias=True)
(3105): ReLU()
(3106): Linear(in_features=288, out_features=312, bias=True)
(3107): ReLU()
(3108): Linear(in_features=312, out_features=288, bias=True)
(3109): ReLU()
(3110): Linear(in_features=288, out_features=304, bias=True)
(3111): ReLU()
(3112): Linear(in_features=304, out_features=288, bias=True)
(3113): ReLU()
(3114): Linear(in_features=288, out_features=256, bias=True)
(3115): ReLU()
(3116): Linear(in_features=256, out_features=288, bias=True)
(3117): ReLU()
(3118): Linear(in_features=288, out_features=256, bias=True)
(3119): ReLU()
(3120): Linear(in_features=256, out_features=319, bias=True)
(3121): ReLU()
(3122): Linear(in_features=319, out_features=288, bias=True)
(3123): ReLU()
(3124): Linear(in_features=288, out_features=318, bias=True)
(3125): ReLU()
(3126): Linear(in_features=318, out_features=288, bias=True)
(3127): ReLU()
(3128): Linear(in_features=288, out_features=316, bias=True)
(3129): ReLU()
(3130): Linear(in_features=316, out_features=288, bias=True)
(3131): ReLU()
(3132): Linear(in_features=288, out_features=312, bias=True)
(3133): ReLU()
(3134): Linear(in_features=312, out_features=288, bias=True)
(3135): ReLU()
(3136): Linear(in_features=288, out_features=304, bias=True)
(3137): ReLU()
(3138): Linear(in_features=304, out_features=288, bias=True)
(3139): ReLU()
(3140): Linear(in_features=288, out_features=256, bias=True)
(3141): ReLU()
(3142): Linear(in_features=256, out_features=368, bias=True)
(3143): ReLU()
(3144): Linear(in_features=368, out_features=328, bias=True)
(3145): ReLU()
(3146): Linear(in_features=328, out_features=340, bias=True)
(3147): ReLU()
(3148): Linear(in_features=340, out_features=332, bias=True)
(3149): ReLU()
(3150): Linear(in_features=332, out_features=375, bias=True)
(3151): ReLU()
(3152): Linear(in_features=375, out_features=399, bias=True)
(3153): ReLU()
(3154): Linear(in_features=399, out_features=410, bias=True)
(3155): ReLU()
(3156): Linear(in_features=410, out_features=402, bias=True)
(3157): ReLU()
(3158): Linear(in_features=402, out_features=412, bias=True)
(3159): ReLU()
(3160): Linear(in_features=412, out_features=404, bias=True)
(3161): ReLU()
(3162): Linear(in_features=404, out_features=412, bias=True)
(3163): ReLU()
(3164): Linear(in_features=412, out_features=404, bias=True)
(3165): ReLU()
(3166): Linear(in_features=404, out_features=408, bias=True)
(3167): ReLU()
(3168): Linear(in_features=408, out_features=400, bias=True)
(3169): ReLU()
(3170): Linear(in_features=400, out_features=352, bias=True)
(3171): ReLU()
(3172): Linear(in_features=352, out_features=288, bias=True)
(3173): ReLU()
(3174): Linear(in_features=288, out_features=288, bias=True)
(3175): ReLU()
(3176): Linear(in_features=288, out_features=256, bias=True)
(3177): ReLU()
(3178): Linear(in_features=256, out_features=319, bias=True)
(3179): ReLU()
(3180): Linear(in_features=319, out_features=288, bias=True)
(3181): ReLU()
(3182): Linear(in_features=288, out_features=318, bias=True)
(3183): ReLU()
(3184): Linear(in_features=318, out_features=288, bias=True)
(3185): ReLU()
(3186): Linear(in_features=288, out_features=316, bias=True)
(3187): ReLU()
(3188): Linear(in_features=316, out_features=288, bias=True)
(3189): ReLU()
(3190): Linear(in_features=288, out_features=312, bias=True)
(3191): ReLU()
(3192): Linear(in_features=312, out_features=288, bias=True)
(3193): ReLU()
(3194): Linear(in_features=288, out_features=304, bias=True)
(3195): ReLU()
(3196): Linear(in_features=304, out_features=288, bias=True)
(3197): ReLU()
(3198): Linear(in_features=288, out_features=256, bias=True)
(3199): ReLU()
(3200): Linear(in_features=256, out_features=288, bias=True)
(3201): ReLU()
(3202): Linear(in_features=288, out_features=256, bias=True)
(3203): ReLU()
(3204): Linear(in_features=256, out_features=319, bias=True)
(3205): ReLU()
(3206): Linear(in_features=319, out_features=288, bias=True)
(3207): ReLU()
(3208): Linear(in_features=288, out_features=318, bias=True)
(3209): ReLU()
(3210): Linear(in_features=318, out_features=288, bias=True)
(3211): ReLU()
(3212): Linear(in_features=288, out_features=316, bias=True)
(3213): ReLU()
(3214): Linear(in_features=316, out_features=288, bias=True)
(3215): ReLU()
(3216): Linear(in_features=288, out_features=312, bias=True)
(3217): ReLU()
(3218): Linear(in_features=312, out_features=288, bias=True)
(3219): ReLU()
(3220): Linear(in_features=288, out_features=304, bias=True)
(3221): ReLU()
(3222): Linear(in_features=304, out_features=288, bias=True)
(3223): ReLU()
(3224): Linear(in_features=288, out_features=256, bias=True)
(3225): ReLU()
(3226): Linear(in_features=256, out_features=368, bias=True)
(3227): ReLU()
(3228): Linear(in_features=368, out_features=328, bias=True)
(3229): ReLU()
(3230): Linear(in_features=328, out_features=340, bias=True)
(3231): ReLU()
(3232): Linear(in_features=340, out_features=332, bias=True)
(3233): ReLU()
(3234): Linear(in_features=332, out_features=375, bias=True)
(3235): ReLU()
(3236): Linear(in_features=375, out_features=399, bias=True)
(3237): ReLU()
(3238): Linear(in_features=399, out_features=410, bias=True)
(3239): ReLU()
(3240): Linear(in_features=410, out_features=402, bias=True)
(3241): ReLU()
(3242): Linear(in_features=402, out_features=412, bias=True)
(3243): ReLU()
(3244): Linear(in_features=412, out_features=404, bias=True)
(3245): ReLU()
(3246): Linear(in_features=404, out_features=412, bias=True)
(3247): ReLU()
(3248): Linear(in_features=412, out_features=404, bias=True)
(3249): ReLU()
(3250): Linear(in_features=404, out_features=408, bias=True)
(3251): ReLU()
(3252): Linear(in_features=408, out_features=400, bias=True)
(3253): ReLU()
(3254): Linear(in_features=400, out_features=352, bias=True)
(3255): ReLU()
(3256): Linear(in_features=352, out_features=288, bias=True)
(3257): ReLU()
(3258): Linear(in_features=288, out_features=288, bias=True)
(3259): ReLU()
(3260): Linear(in_features=288, out_features=256, bias=True)
(3261): ReLU()
(3262): Linear(in_features=256, out_features=319, bias=True)
(3263): ReLU()
(3264): Linear(in_features=319, out_features=288, bias=True)
(3265): ReLU()
(3266): Linear(in_features=288, out_features=318, bias=True)
(3267): ReLU()
(3268): Linear(in_features=318, out_features=288, bias=True)
(3269): ReLU()
(3270): Linear(in_features=288, out_features=316, bias=True)
(3271): ReLU()
(3272): Linear(in_features=316, out_features=288, bias=True)
(3273): ReLU()
(3274): Linear(in_features=288, out_features=312, bias=True)
(3275): ReLU()
(3276): Linear(in_features=312, out_features=288, bias=True)
(3277): ReLU()
(3278): Linear(in_features=288, out_features=304, bias=True)
(3279): ReLU()
(3280): Linear(in_features=304, out_features=288, bias=True)
(3281): ReLU()
(3282): Linear(in_features=288, out_features=256, bias=True)
(3283): ReLU()
(3284): Linear(in_features=256, out_features=288, bias=True)
(3285): ReLU()
(3286): Linear(in_features=288, out_features=256, bias=True)
(3287): ReLU()
(3288): Linear(in_features=256, out_features=319, bias=True)
(3289): ReLU()
(3290): Linear(in_features=319, out_features=288, bias=True)
(3291): ReLU()
(3292): Linear(in_features=288, out_features=318, bias=True)
(3293): ReLU()
(3294): Linear(in_features=318, out_features=288, bias=True)
(3295): ReLU()
(3296): Linear(in_features=288, out_features=316, bias=True)
(3297): ReLU()
(3298): Linear(in_features=316, out_features=288, bias=True)
(3299): ReLU()
(3300): Linear(in_features=288, out_features=312, bias=True)
(3301): ReLU()
(3302): Linear(in_features=312, out_features=288, bias=True)
(3303): ReLU()
(3304): Linear(in_features=288, out_features=304, bias=True)
(3305): ReLU()
(3306): Linear(in_features=304, out_features=288, bias=True)
(3307): ReLU()
(3308): Linear(in_features=288, out_features=256, bias=True)
(3309): ReLU()
(3310): Linear(in_features=256, out_features=368, bias=True)
(3311): ReLU()
(3312): Linear(in_features=368, out_features=328, bias=True)
(3313): ReLU()
(3314): Linear(in_features=328, out_features=340, bias=True)
(3315): ReLU()
(3316): Linear(in_features=340, out_features=332, bias=True)
(3317): ReLU()
(3318): Linear(in_features=332, out_features=375, bias=True)
(3319): ReLU()
(3320): Linear(in_features=375, out_features=399, bias=True)
(3321): ReLU()
(3322): Linear(in_features=399, out_features=410, bias=True)
(3323): ReLU()
(3324): Linear(in_features=410, out_features=402, bias=True)
(3325): ReLU()
(3326): Linear(in_features=402, out_features=412, bias=True)
(3327): ReLU()
(3328): Linear(in_features=412, out_features=404, bias=True)
(3329): ReLU()
(3330): Linear(in_features=404, out_features=412, bias=True)
(3331): ReLU()
(3332): Linear(in_features=412, out_features=404, bias=True)
(3333): ReLU()
(3334): Linear(in_features=404, out_features=408, bias=True)
(3335): ReLU()
(3336): Linear(in_features=408, out_features=400, bias=True)
(3337): ReLU()
(3338): Linear(in_features=400, out_features=352, bias=True)
(3339): ReLU()
(3340): Linear(in_features=352, out_features=288, bias=True)
(3341): ReLU()
(3342): Linear(in_features=288, out_features=288, bias=True)
(3343): ReLU()
(3344): Linear(in_features=288, out_features=256, bias=True)
(3345): ReLU()
(3346): Linear(in_features=256, out_features=319, bias=True)
(3347): ReLU()
(3348): Linear(in_features=319, out_features=288, bias=True)
(3349): ReLU()
(3350): Linear(in_features=288, out_features=318, bias=True)
(3351): ReLU()
(3352): Linear(in_features=318, out_features=288, bias=True)
(3353): ReLU()
(3354): Linear(in_features=288, out_features=316, bias=True)
(3355): ReLU()
(3356): Linear(in_features=316, out_features=288, bias=True)
(3357): ReLU()
(3358): Linear(in_features=288, out_features=312, bias=True)
(3359): ReLU()
(3360): Linear(in_features=312, out_features=288, bias=True)
(3361): ReLU()
(3362): Linear(in_features=288, out_features=304, bias=True)
(3363): ReLU()
(3364): Linear(in_features=304, out_features=288, bias=True)
(3365): ReLU()
(3366): Linear(in_features=288, out_features=256, bias=True)
(3367): ReLU()
(3368): Linear(in_features=256, out_features=288, bias=True)
(3369): ReLU()
(3370): Linear(in_features=288, out_features=256, bias=True)
(3371): ReLU()
(3372): Linear(in_features=256, out_features=319, bias=True)
(3373): ReLU()
(3374): Linear(in_features=319, out_features=288, bias=True)
(3375): ReLU()
(3376): Linear(in_features=288, out_features=318, bias=True)
(3377): ReLU()
(3378): Linear(in_features=318, out_features=288, bias=True)
(3379): ReLU()
(3380): Linear(in_features=288, out_features=316, bias=True)
(3381): ReLU()
(3382): Linear(in_features=316, out_features=288, bias=True)
(3383): ReLU()
(3384): Linear(in_features=288, out_features=312, bias=True)
(3385): ReLU()
(3386): Linear(in_features=312, out_features=288, bias=True)
(3387): ReLU()
(3388): Linear(in_features=288, out_features=304, bias=True)
(3389): ReLU()
(3390): Linear(in_features=304, out_features=288, bias=True)
(3391): ReLU()
(3392): Linear(in_features=288, out_features=256, bias=True)
(3393): ReLU()
(3394): Linear(in_features=256, out_features=368, bias=True)
(3395): ReLU()
(3396): Linear(in_features=368, out_features=328, bias=True)
(3397): ReLU()
(3398): Linear(in_features=328, out_features=340, bias=True)
(3399): ReLU()
(3400): Linear(in_features=340, out_features=332, bias=True)
(3401): ReLU()
(3402): Linear(in_features=332, out_features=375, bias=True)
(3403): ReLU()
(3404): Linear(in_features=375, out_features=399, bias=True)
(3405): ReLU()
(3406): Linear(in_features=399, out_features=410, bias=True)
(3407): ReLU()
(3408): Linear(in_features=410, out_features=402, bias=True)
(3409): ReLU()
(3410): Linear(in_features=402, out_features=412, bias=True)
(3411): ReLU()
(3412): Linear(in_features=412, out_features=404, bias=True)
(3413): ReLU()
(3414): Linear(in_features=404, out_features=412, bias=True)
(3415): ReLU()
(3416): Linear(in_features=412, out_features=404, bias=True)
(3417): ReLU()
(3418): Linear(in_features=404, out_features=408, bias=True)
(3419): ReLU()
(3420): Linear(in_features=408, out_features=400, bias=True)
(3421): ReLU()
(3422): Linear(in_features=400, out_features=352, bias=True)
(3423): ReLU()
(3424): Linear(in_features=352, out_features=288, bias=True)
(3425): ReLU()
(3426): Linear(in_features=288, out_features=288, bias=True)
(3427): ReLU()
(3428): Linear(in_features=288, out_features=256, bias=True)
(3429): ReLU()
(3430): Linear(in_features=256, out_features=319, bias=True)
(3431): ReLU()
(3432): Linear(in_features=319, out_features=288, bias=True)
(3433): ReLU()
(3434): Linear(in_features=288, out_features=318, bias=True)
(3435): ReLU()
(3436): Linear(in_features=318, out_features=288, bias=True)
(3437): ReLU()
(3438): Linear(in_features=288, out_features=316, bias=True)
(3439): ReLU()
(3440): Linear(in_features=316, out_features=288, bias=True)
(3441): ReLU()
(3442): Linear(in_features=288, out_features=312, bias=True)
(3443): ReLU()
(3444): Linear(in_features=312, out_features=288, bias=True)
(3445): ReLU()
(3446): Linear(in_features=288, out_features=304, bias=True)
(3447): ReLU()
(3448): Linear(in_features=304, out_features=288, bias=True)
(3449): ReLU()
(3450): Linear(in_features=288, out_features=256, bias=True)
(3451): ReLU()
(3452): Linear(in_features=256, out_features=288, bias=True)
(3453): ReLU()
(3454): Linear(in_features=288, out_features=256, bias=True)
(3455): ReLU()
(3456): Linear(in_features=256, out_features=319, bias=True)
(3457): ReLU()
(3458): Linear(in_features=319, out_features=288, bias=True)
(3459): ReLU()
(3460): Linear(in_features=288, out_features=318, bias=True)
(3461): ReLU()
(3462): Linear(in_features=318, out_features=288, bias=True)
(3463): ReLU()
(3464): Linear(in_features=288, out_features=316, bias=True)
(3465): ReLU()
(3466): Linear(in_features=316, out_features=288, bias=True)
(3467): ReLU()
(3468): Linear(in_features=288, out_features=312, bias=True)
(3469): ReLU()
(3470): Linear(in_features=312, out_features=288, bias=True)
(3471): ReLU()
(3472): Linear(in_features=288, out_features=304, bias=True)
(3473): ReLU()
(3474): Linear(in_features=304, out_features=288, bias=True)
(3475): ReLU()
(3476): Linear(in_features=288, out_features=256, bias=True)
(3477): ReLU()
(3478): Linear(in_features=256, out_features=368, bias=True)
(3479): ReLU()
(3480): Linear(in_features=368, out_features=328, bias=True)
(3481): ReLU()
(3482): Linear(in_features=328, out_features=340, bias=True)
(3483): ReLU()
(3484): Linear(in_features=340, out_features=332, bias=True)
(3485): ReLU()
(3486): Linear(in_features=332, out_features=375, bias=True)
(3487): ReLU()
(3488): Linear(in_features=375, out_features=399, bias=True)
(3489): ReLU()
(3490): Linear(in_features=399, out_features=410, bias=True)
(3491): ReLU()
(3492): Linear(in_features=410, out_features=402, bias=True)
(3493): ReLU()
(3494): Linear(in_features=402, out_features=412, bias=True)
(3495): ReLU()
(3496): Linear(in_features=412, out_features=404, bias=True)
(3497): ReLU()
(3498): Linear(in_features=404, out_features=412, bias=True)
(3499): ReLU()
(3500): Linear(in_features=412, out_features=404, bias=True)
(3501): ReLU()
(3502): Linear(in_features=404, out_features=408, bias=True)
(3503): ReLU()
(3504): Linear(in_features=408, out_features=400, bias=True)
(3505): ReLU()
(3506): Linear(in_features=400, out_features=352, bias=True)
(3507): ReLU()
(3508): Linear(in_features=352, out_features=288, bias=True)
(3509): ReLU()
(3510): Linear(in_features=288, out_features=288, bias=True)
(3511): ReLU()
(3512): Linear(in_features=288, out_features=256, bias=True)
(3513): ReLU()
(3514): Linear(in_features=256, out_features=319, bias=True)
(3515): ReLU()
(3516): Linear(in_features=319, out_features=288, bias=True)
(3517): ReLU()
(3518): Linear(in_features=288, out_features=318, bias=True)
(3519): ReLU()
(3520): Linear(in_features=318, out_features=288, bias=True)
(3521): ReLU()
(3522): Linear(in_features=288, out_features=316, bias=True)
(3523): ReLU()
(3524): Linear(in_features=316, out_features=288, bias=True)
(3525): ReLU()
(3526): Linear(in_features=288, out_features=312, bias=True)
(3527): ReLU()
(3528): Linear(in_features=312, out_features=288, bias=True)
(3529): ReLU()
(3530): Linear(in_features=288, out_features=304, bias=True)
(3531): ReLU()
(3532): Linear(in_features=304, out_features=288, bias=True)
(3533): ReLU()
(3534): Linear(in_features=288, out_features=256, bias=True)
(3535): ReLU()
(3536): Linear(in_features=256, out_features=288, bias=True)
(3537): ReLU()
(3538): Linear(in_features=288, out_features=256, bias=True)
(3539): ReLU()
(3540): Linear(in_features=256, out_features=319, bias=True)
(3541): ReLU()
(3542): Linear(in_features=319, out_features=288, bias=True)
(3543): ReLU()
(3544): Linear(in_features=288, out_features=318, bias=True)
(3545): ReLU()
(3546): Linear(in_features=318, out_features=288, bias=True)
(3547): ReLU()
(3548): Linear(in_features=288, out_features=316, bias=True)
(3549): ReLU()
(3550): Linear(in_features=316, out_features=288, bias=True)
(3551): ReLU()
(3552): Linear(in_features=288, out_features=312, bias=True)
(3553): ReLU()
(3554): Linear(in_features=312, out_features=288, bias=True)
(3555): ReLU()
(3556): Linear(in_features=288, out_features=304, bias=True)
(3557): ReLU()
(3558): Linear(in_features=304, out_features=288, bias=True)
(3559): ReLU()
(3560): Linear(in_features=288, out_features=256, bias=True)
(3561): ReLU()
(3562): Linear(in_features=256, out_features=368, bias=True)
(3563): ReLU()
(3564): Linear(in_features=368, out_features=328, bias=True)
(3565): ReLU()
(3566): Linear(in_features=328, out_features=340, bias=True)
(3567): ReLU()
(3568): Linear(in_features=340, out_features=332, bias=True)
(3569): ReLU()
(3570): Linear(in_features=332, out_features=375, bias=True)
(3571): ReLU()
(3572): Linear(in_features=375, out_features=399, bias=True)
(3573): ReLU()
(3574): Linear(in_features=399, out_features=410, bias=True)
(3575): ReLU()
(3576): Linear(in_features=410, out_features=402, bias=True)
(3577): ReLU()
(3578): Linear(in_features=402, out_features=412, bias=True)
(3579): ReLU()
(3580): Linear(in_features=412, out_features=404, bias=True)
(3581): ReLU()
(3582): Linear(in_features=404, out_features=412, bias=True)
(3583): ReLU()
(3584): Linear(in_features=412, out_features=404, bias=True)
(3585): ReLU()
(3586): Linear(in_features=404, out_features=408, bias=True)
(3587): ReLU()
(3588): Linear(in_features=408, out_features=400, bias=True)
(3589): ReLU()
(3590): Linear(in_features=400, out_features=352, bias=True)
(3591): ReLU()
(3592): Linear(in_features=352, out_features=288, bias=True)
(3593): ReLU()
(3594): Linear(in_features=288, out_features=288, bias=True)
(3595): ReLU()
(3596): Linear(in_features=288, out_features=256, bias=True)
(3597): ReLU()
(3598): Linear(in_features=256, out_features=319, bias=True)
(3599): ReLU()
(3600): Linear(in_features=319, out_features=288, bias=True)
(3601): ReLU()
(3602): Linear(in_features=288, out_features=318, bias=True)
(3603): ReLU()
(3604): Linear(in_features=318, out_features=288, bias=True)
(3605): ReLU()
(3606): Linear(in_features=288, out_features=316, bias=True)
(3607): ReLU()
(3608): Linear(in_features=316, out_features=288, bias=True)
(3609): ReLU()
(3610): Linear(in_features=288, out_features=312, bias=True)
(3611): ReLU()
(3612): Linear(in_features=312, out_features=288, bias=True)
(3613): ReLU()
(3614): Linear(in_features=288, out_features=304, bias=True)
(3615): ReLU()
(3616): Linear(in_features=304, out_features=288, bias=True)
(3617): ReLU()
(3618): Linear(in_features=288, out_features=256, bias=True)
(3619): ReLU()
(3620): Linear(in_features=256, out_features=288, bias=True)
(3621): ReLU()
(3622): Linear(in_features=288, out_features=256, bias=True)
(3623): ReLU()
(3624): Linear(in_features=256, out_features=319, bias=True)
(3625): ReLU()
(3626): Linear(in_features=319, out_features=288, bias=True)
(3627): ReLU()
(3628): Linear(in_features=288, out_features=318, bias=True)
(3629): ReLU()
(3630): Linear(in_features=318, out_features=288, bias=True)
(3631): ReLU()
(3632): Linear(in_features=288, out_features=316, bias=True)
(3633): ReLU()
(3634): Linear(in_features=316, out_features=288, bias=True)
(3635): ReLU()
(3636): Linear(in_features=288, out_features=312, bias=True)
(3637): ReLU()
(3638): Linear(in_features=312, out_features=288, bias=True)
(3639): ReLU()
(3640): Linear(in_features=288, out_features=304, bias=True)
(3641): ReLU()
(3642): Linear(in_features=304, out_features=288, bias=True)
(3643): ReLU()
(3644): Linear(in_features=288, out_features=256, bias=True)
(3645): ReLU()
(3646): Linear(in_features=256, out_features=368, bias=True)
(3647): ReLU()
(3648): Linear(in_features=368, out_features=328, bias=True)
(3649): ReLU()
(3650): Linear(in_features=328, out_features=340, bias=True)
(3651): ReLU()
(3652): Linear(in_features=340, out_features=332, bias=True)
(3653): ReLU()
(3654): Linear(in_features=332, out_features=375, bias=True)
(3655): ReLU()
(3656): Linear(in_features=375, out_features=399, bias=True)
(3657): ReLU()
(3658): Linear(in_features=399, out_features=410, bias=True)
(3659): ReLU()
(3660): Linear(in_features=410, out_features=402, bias=True)
(3661): ReLU()
(3662): Linear(in_features=402, out_features=412, bias=True)
(3663): ReLU()
(3664): Linear(in_features=412, out_features=404, bias=True)
(3665): ReLU()
(3666): Linear(in_features=404, out_features=412, bias=True)
(3667): ReLU()
(3668): Linear(in_features=412, out_features=404, bias=True)
(3669): ReLU()
(3670): Linear(in_features=404, out_features=408, bias=True)
(3671): ReLU()
(3672): Linear(in_features=408, out_features=400, bias=True)
(3673): ReLU()
(3674): Linear(in_features=400, out_features=352, bias=True)
(3675): ReLU()
(3676): Linear(in_features=352, out_features=288, bias=True)
(3677): ReLU()
(3678): Linear(in_features=288, out_features=288, bias=True)
(3679): ReLU()
(3680): Linear(in_features=288, out_features=256, bias=True)
(3681): ReLU()
(3682): Linear(in_features=256, out_features=319, bias=True)
(3683): ReLU()
(3684): Linear(in_features=319, out_features=288, bias=True)
(3685): ReLU()
(3686): Linear(in_features=288, out_features=318, bias=True)
(3687): ReLU()
(3688): Linear(in_features=318, out_features=288, bias=True)
(3689): ReLU()
(3690): Linear(in_features=288, out_features=316, bias=True)
(3691): ReLU()
(3692): Linear(in_features=316, out_features=288, bias=True)
(3693): ReLU()
(3694): Linear(in_features=288, out_features=312, bias=True)
(3695): ReLU()
(3696): Linear(in_features=312, out_features=288, bias=True)
(3697): ReLU()
(3698): Linear(in_features=288, out_features=304, bias=True)
(3699): ReLU()
(3700): Linear(in_features=304, out_features=288, bias=True)
(3701): ReLU()
(3702): Linear(in_features=288, out_features=256, bias=True)
(3703): ReLU()
(3704): Linear(in_features=256, out_features=288, bias=True)
(3705): ReLU()
(3706): Linear(in_features=288, out_features=256, bias=True)
(3707): ReLU()
(3708): Linear(in_features=256, out_features=319, bias=True)
(3709): ReLU()
(3710): Linear(in_features=319, out_features=288, bias=True)
(3711): ReLU()
(3712): Linear(in_features=288, out_features=318, bias=True)
(3713): ReLU()
(3714): Linear(in_features=318, out_features=288, bias=True)
(3715): ReLU()
(3716): Linear(in_features=288, out_features=316, bias=True)
(3717): ReLU()
(3718): Linear(in_features=316, out_features=288, bias=True)
(3719): ReLU()
(3720): Linear(in_features=288, out_features=312, bias=True)
(3721): ReLU()
(3722): Linear(in_features=312, out_features=288, bias=True)
(3723): ReLU()
(3724): Linear(in_features=288, out_features=304, bias=True)
(3725): ReLU()
(3726): Linear(in_features=304, out_features=288, bias=True)
(3727): ReLU()
(3728): Linear(in_features=288, out_features=256, bias=True)
(3729): ReLU()
(3730): Linear(in_features=256, out_features=368, bias=True)
(3731): ReLU()
(3732): Linear(in_features=368, out_features=328, bias=True)
(3733): ReLU()
(3734): Linear(in_features=328, out_features=340, bias=True)
(3735): ReLU()
(3736): Linear(in_features=340, out_features=332, bias=True)
(3737): ReLU()
(3738): Linear(in_features=332, out_features=375, bias=True)
(3739): ReLU()
(3740): Linear(in_features=375, out_features=399, bias=True)
(3741): ReLU()
(3742): Linear(in_features=399, out_features=410, bias=True)
(3743): ReLU()
(3744): Linear(in_features=410, out_features=402, bias=True)
(3745): ReLU()
(3746): Linear(in_features=402, out_features=412, bias=True)
(3747): ReLU()
(3748): Linear(in_features=412, out_features=404, bias=True)
(3749): ReLU()
(3750): Linear(in_features=404, out_features=412, bias=True)
(3751): ReLU()
(3752): Linear(in_features=412, out_features=404, bias=True)
(3753): ReLU()
(3754): Linear(in_features=404, out_features=408, bias=True)
(3755): ReLU()
(3756): Linear(in_features=408, out_features=400, bias=True)
(3757): ReLU()
(3758): Linear(in_features=400, out_features=352, bias=True)
(3759): ReLU()
(3760): Linear(in_features=352, out_features=288, bias=True)
(3761): ReLU()
(3762): Linear(in_features=288, out_features=288, bias=True)
(3763): ReLU()
(3764): Linear(in_features=288, out_features=256, bias=True)
(3765): ReLU()
(3766): Linear(in_features=256, out_features=319, bias=True)
(3767): ReLU()
(3768): Linear(in_features=319, out_features=288, bias=True)
(3769): ReLU()
(3770): Linear(in_features=288, out_features=318, bias=True)
(3771): ReLU()
(3772): Linear(in_features=318, out_features=288, bias=True)
(3773): ReLU()
(3774): Linear(in_features=288, out_features=316, bias=True)
(3775): ReLU()
(3776): Linear(in_features=316, out_features=288, bias=True)
(3777): ReLU()
(3778): Linear(in_features=288, out_features=312, bias=True)
(3779): ReLU()
(3780): Linear(in_features=312, out_features=288, bias=True)
(3781): ReLU()
(3782): Linear(in_features=288, out_features=304, bias=True)
(3783): ReLU()
(3784): Linear(in_features=304, out_features=288, bias=True)
(3785): ReLU()
(3786): Linear(in_features=288, out_features=256, bias=True)
(3787): ReLU()
(3788): Linear(in_features=256, out_features=288, bias=True)
(3789): ReLU()
(3790): Linear(in_features=288, out_features=256, bias=True)
(3791): ReLU()
(3792): Linear(in_features=256, out_features=319, bias=True)
(3793): ReLU()
(3794): Linear(in_features=319, out_features=288, bias=True)
(3795): ReLU()
(3796): Linear(in_features=288, out_features=318, bias=True)
(3797): ReLU()
(3798): Linear(in_features=318, out_features=288, bias=True)
(3799): ReLU()
(3800): Linear(in_features=288, out_features=316, bias=True)
(3801): ReLU()
(3802): Linear(in_features=316, out_features=288, bias=True)
(3803): ReLU()
(3804): Linear(in_features=288, out_features=312, bias=True)
(3805): ReLU()
(3806): Linear(in_features=312, out_features=288, bias=True)
(3807): ReLU()
(3808): Linear(in_features=288, out_features=304, bias=True)
(3809): ReLU()
(3810): Linear(in_features=304, out_features=288, bias=True)
(3811): ReLU()
(3812): Linear(in_features=288, out_features=256, bias=True)
(3813): ReLU()
(3814): Linear(in_features=256, out_features=368, bias=True)
(3815): ReLU()
(3816): Linear(in_features=368, out_features=328, bias=True)
(3817): ReLU()
(3818): Linear(in_features=328, out_features=340, bias=True)
(3819): ReLU()
(3820): Linear(in_features=340, out_features=332, bias=True)
(3821): ReLU()
(3822): Linear(in_features=332, out_features=375, bias=True)
(3823): ReLU()
(3824): Linear(in_features=375, out_features=399, bias=True)
(3825): ReLU()
(3826): Linear(in_features=399, out_features=410, bias=True)
(3827): ReLU()
(3828): Linear(in_features=410, out_features=402, bias=True)
(3829): ReLU()
(3830): Linear(in_features=402, out_features=412, bias=True)
(3831): ReLU()
(3832): Linear(in_features=412, out_features=404, bias=True)
(3833): ReLU()
(3834): Linear(in_features=404, out_features=412, bias=True)
(3835): ReLU()
(3836): Linear(in_features=412, out_features=404, bias=True)
(3837): ReLU()
(3838): Linear(in_features=404, out_features=408, bias=True)
(3839): ReLU()
(3840): Linear(in_features=408, out_features=400, bias=True)
(3841): ReLU()
(3842): Linear(in_features=400, out_features=352, bias=True)
(3843): ReLU()
(3844): Linear(in_features=352, out_features=288, bias=True)
(3845): ReLU()
(3846): Linear(in_features=288, out_features=288, bias=True)
(3847): ReLU()
(3848): Linear(in_features=288, out_features=256, bias=True)
(3849): ReLU()
(3850): Linear(in_features=256, out_features=319, bias=True)
(3851): ReLU()
(3852): Linear(in_features=319, out_features=288, bias=True)
(3853): ReLU()
(3854): Linear(in_features=288, out_features=318, bias=True)
(3855): ReLU()
(3856): Linear(in_features=318, out_features=288, bias=True)
(3857): ReLU()
(3858): Linear(in_features=288, out_features=316, bias=True)
(3859): ReLU()
(3860): Linear(in_features=316, out_features=288, bias=True)
(3861): ReLU()
(3862): Linear(in_features=288, out_features=312, bias=True)
(3863): ReLU()
(3864): Linear(in_features=312, out_features=288, bias=True)
(3865): ReLU()
(3866): Linear(in_features=288, out_features=304, bias=True)
(3867): ReLU()
(3868): Linear(in_features=304, out_features=288, bias=True)
(3869): ReLU()
(3870): Linear(in_features=288, out_features=256, bias=True)
(3871): ReLU()
(3872): Linear(in_features=256, out_features=288, bias=True)
(3873): ReLU()
(3874): Linear(in_features=288, out_features=256, bias=True)
(3875): ReLU()
(3876): Linear(in_features=256, out_features=319, bias=True)
(3877): ReLU()
(3878): Linear(in_features=319, out_features=288, bias=True)
(3879): ReLU()
(3880): Linear(in_features=288, out_features=318, bias=True)
(3881): ReLU()
(3882): Linear(in_features=318, out_features=288, bias=True)
(3883): ReLU()
(3884): Linear(in_features=288, out_features=316, bias=True)
(3885): ReLU()
(3886): Linear(in_features=316, out_features=288, bias=True)
(3887): ReLU()
(3888): Linear(in_features=288, out_features=312, bias=True)
(3889): ReLU()
(3890): Linear(in_features=312, out_features=288, bias=True)
(3891): ReLU()
(3892): Linear(in_features=288, out_features=304, bias=True)
(3893): ReLU()
(3894): Linear(in_features=304, out_features=288, bias=True)
(3895): ReLU()
(3896): Linear(in_features=288, out_features=256, bias=True)
(3897): ReLU()
(3898): Linear(in_features=256, out_features=368, bias=True)
(3899): ReLU()
(3900): Linear(in_features=368, out_features=328, bias=True)
(3901): ReLU()
(3902): Linear(in_features=328, out_features=340, bias=True)
(3903): ReLU()
(3904): Linear(in_features=340, out_features=332, bias=True)
(3905): ReLU()
(3906): Linear(in_features=332, out_features=375, bias=True)
(3907): ReLU()
(3908): Linear(in_features=375, out_features=399, bias=True)
(3909): ReLU()
(3910): Linear(in_features=399, out_features=410, bias=True)
(3911): ReLU()
(3912): Linear(in_features=410, out_features=402, bias=True)
(3913): ReLU()
(3914): Linear(in_features=402, out_features=412, bias=True)
(3915): ReLU()
(3916): Linear(in_features=412, out_features=404, bias=True)
(3917): ReLU()
(3918): Linear(in_features=404, out_features=412, bias=True)
(3919): ReLU()
(3920): Linear(in_features=412, out_features=404, bias=True)
(3921): ReLU()
(3922): Linear(in_features=404, out_features=408, bias=True)
(3923): ReLU()
(3924): Linear(in_features=408, out_features=400, bias=True)
(3925): ReLU()
(3926): Linear(in_features=400, out_features=352, bias=True)
(3927): ReLU()
(3928): Linear(in_features=352, out_features=288, bias=True)
(3929): ReLU()
(3930): Linear(in_features=288, out_features=288, bias=True)
(3931): ReLU()
(3932): Linear(in_features=288, out_features=256, bias=True)
(3933): ReLU()
(3934): Linear(in_features=256, out_features=319, bias=True)
(3935): ReLU()
(3936): Linear(in_features=319, out_features=288, bias=True)
(3937): ReLU()
(3938): Linear(in_features=288, out_features=318, bias=True)
(3939): ReLU()
(3940): Linear(in_features=318, out_features=288, bias=True)
(3941): ReLU()
(3942): Linear(in_features=288, out_features=316, bias=True)
(3943): ReLU()
(3944): Linear(in_features=316, out_features=288, bias=True)
(3945): ReLU()
(3946): Linear(in_features=288, out_features=312, bias=True)
(3947): ReLU()
(3948): Linear(in_features=312, out_features=288, bias=True)
(3949): ReLU()
(3950): Linear(in_features=288, out_features=304, bias=True)
(3951): ReLU()
(3952): Linear(in_features=304, out_features=288, bias=True)
(3953): ReLU()
(3954): Linear(in_features=288, out_features=256, bias=True)
(3955): ReLU()
(3956): Linear(in_features=256, out_features=288, bias=True)
(3957): ReLU()
(3958): Linear(in_features=288, out_features=256, bias=True)
(3959): ReLU()
(3960): Linear(in_features=256, out_features=319, bias=True)
(3961): ReLU()
(3962): Linear(in_features=319, out_features=288, bias=True)
(3963): ReLU()
(3964): Linear(in_features=288, out_features=318, bias=True)
(3965): ReLU()
(3966): Linear(in_features=318, out_features=288, bias=True)
(3967): ReLU()
(3968): Linear(in_features=288, out_features=316, bias=True)
(3969): ReLU()
(3970): Linear(in_features=316, out_features=288, bias=True)
(3971): ReLU()
(3972): Linear(in_features=288, out_features=312, bias=True)
(3973): ReLU()
(3974): Linear(in_features=312, out_features=288, bias=True)
(3975): ReLU()
(3976): Linear(in_features=288, out_features=304, bias=True)
(3977): ReLU()
(3978): Linear(in_features=304, out_features=288, bias=True)
(3979): ReLU()
(3980): Linear(in_features=288, out_features=256, bias=True)
(3981): ReLU()
(3982): Linear(in_features=256, out_features=368, bias=True)
(3983): ReLU()
(3984): Linear(in_features=368, out_features=328, bias=True)
(3985): ReLU()
(3986): Linear(in_features=328, out_features=340, bias=True)
(3987): ReLU()
(3988): Linear(in_features=340, out_features=332, bias=True)
(3989): ReLU()
(3990): Linear(in_features=332, out_features=375, bias=True)
(3991): ReLU()
(3992): Linear(in_features=375, out_features=399, bias=True)
(3993): ReLU()
(3994): Linear(in_features=399, out_features=410, bias=True)
(3995): ReLU()
(3996): Linear(in_features=410, out_features=402, bias=True)
(3997): ReLU()
(3998): Linear(in_features=402, out_features=412, bias=True)
(3999): ReLU()
(4000): Linear(in_features=412, out_features=404, bias=True)
(4001): ReLU()
(4002): Linear(in_features=404, out_features=412, bias=True)
(4003): ReLU()
(4004): Linear(in_features=412, out_features=404, bias=True)
(4005): ReLU()
(4006): Linear(in_features=404, out_features=408, bias=True)
(4007): ReLU()
(4008): Linear(in_features=408, out_features=400, bias=True)
(4009): ReLU()
(4010): Linear(in_features=400, out_features=352, bias=True)
(4011): ReLU()
(4012): Linear(in_features=352, out_features=288, bias=True)
(4013): ReLU()
(4014): Linear(in_features=288, out_features=288, bias=True)
(4015): ReLU()
(4016): Linear(in_features=288, out_features=256, bias=True)
(4017): ReLU()
(4018): Linear(in_features=256, out_features=319, bias=True)
(4019): ReLU()
(4020): Linear(in_features=319, out_features=288, bias=True)
(4021): ReLU()
(4022): Linear(in_features=288, out_features=318, bias=True)
(4023): ReLU()
(4024): Linear(in_features=318, out_features=288, bias=True)
(4025): ReLU()
(4026): Linear(in_features=288, out_features=316, bias=True)
(4027): ReLU()
(4028): Linear(in_features=316, out_features=288, bias=True)
(4029): ReLU()
(4030): Linear(in_features=288, out_features=312, bias=True)
(4031): ReLU()
(4032): Linear(in_features=312, out_features=288, bias=True)
(4033): ReLU()
(4034): Linear(in_features=288, out_features=304, bias=True)
(4035): ReLU()
(4036): Linear(in_features=304, out_features=288, bias=True)
(4037): ReLU()
(4038): Linear(in_features=288, out_features=256, bias=True)
(4039): ReLU()
(4040): Linear(in_features=256, out_features=288, bias=True)
(4041): ReLU()
(4042): Linear(in_features=288, out_features=256, bias=True)
(4043): ReLU()
(4044): Linear(in_features=256, out_features=319, bias=True)
(4045): ReLU()
(4046): Linear(in_features=319, out_features=288, bias=True)
(4047): ReLU()
(4048): Linear(in_features=288, out_features=318, bias=True)
(4049): ReLU()
(4050): Linear(in_features=318, out_features=288, bias=True)
(4051): ReLU()
(4052): Linear(in_features=288, out_features=316, bias=True)
(4053): ReLU()
(4054): Linear(in_features=316, out_features=288, bias=True)
(4055): ReLU()
(4056): Linear(in_features=288, out_features=312, bias=True)
(4057): ReLU()
(4058): Linear(in_features=312, out_features=288, bias=True)
(4059): ReLU()
(4060): Linear(in_features=288, out_features=304, bias=True)
(4061): ReLU()
(4062): Linear(in_features=304, out_features=288, bias=True)
(4063): ReLU()
(4064): Linear(in_features=288, out_features=256, bias=True)
(4065): ReLU()
(4066): Linear(in_features=256, out_features=336, bias=True)
(4067): ReLU()
(4068): Linear(in_features=336, out_features=328, bias=True)
(4069): ReLU()
(4070): Linear(in_features=328, out_features=340, bias=True)
(4071): ReLU()
(4072): Linear(in_features=340, out_features=332, bias=True)
(4073): ReLU()
(4074): Linear(in_features=332, out_features=375, bias=True)
(4075): ReLU()
(4076): Linear(in_features=375, out_features=399, bias=True)
(4077): ReLU()
(4078): Linear(in_features=399, out_features=410, bias=True)
(4079): ReLU()
(4080): Linear(in_features=410, out_features=402, bias=True)
(4081): ReLU()
(4082): Linear(in_features=402, out_features=412, bias=True)
(4083): ReLU()
(4084): Linear(in_features=412, out_features=404, bias=True)
(4085): ReLU()
(4086): Linear(in_features=404, out_features=412, bias=True)
(4087): ReLU()
(4088): Linear(in_features=412, out_features=404, bias=True)
(4089): ReLU()
(4090): Linear(in_features=404, out_features=408, bias=True)
(4091): ReLU()
(4092): Linear(in_features=408, out_features=400, bias=True)
(4093): ReLU()
(4094): Linear(in_features=400, out_features=352, bias=True)
(4095): ReLU()
(4096): Linear(in_features=352, out_features=288, bias=True)
(4097): ReLU()
(4098): Linear(in_features=288, out_features=288, bias=True)
(4099): ReLU()
(4100): Linear(in_features=288, out_features=256, bias=True)
(4101): ReLU()
(4102): Linear(in_features=256, out_features=319, bias=True)
(4103): ReLU()
(4104): Linear(in_features=319, out_features=288, bias=True)
(4105): ReLU()
(4106): Linear(in_features=288, out_features=318, bias=True)
(4107): ReLU()
(4108): Linear(in_features=318, out_features=288, bias=True)
(4109): ReLU()
(4110): Linear(in_features=288, out_features=316, bias=True)
(4111): ReLU()
(4112): Linear(in_features=316, out_features=288, bias=True)
(4113): ReLU()
(4114): Linear(in_features=288, out_features=312, bias=True)
(4115): ReLU()
(4116): Linear(in_features=312, out_features=288, bias=True)
(4117): ReLU()
(4118): Linear(in_features=288, out_features=304, bias=True)
(4119): ReLU()
(4120): Linear(in_features=304, out_features=288, bias=True)
(4121): ReLU()
(4122): Linear(in_features=288, out_features=256, bias=True)
(4123): ReLU()
(4124): Linear(in_features=256, out_features=288, bias=True)
(4125): ReLU()
(4126): Linear(in_features=288, out_features=256, bias=True)
(4127): ReLU()
(4128): Linear(in_features=256, out_features=319, bias=True)
(4129): ReLU()
(4130): Linear(in_features=319, out_features=288, bias=True)
(4131): ReLU()
(4132): Linear(in_features=288, out_features=318, bias=True)
(4133): ReLU()
(4134): Linear(in_features=318, out_features=288, bias=True)
(4135): ReLU()
(4136): Linear(in_features=288, out_features=316, bias=True)
(4137): ReLU()
(4138): Linear(in_features=316, out_features=288, bias=True)
(4139): ReLU()
(4140): Linear(in_features=288, out_features=312, bias=True)
(4141): ReLU()
(4142): Linear(in_features=312, out_features=288, bias=True)
(4143): ReLU()
(4144): Linear(in_features=288, out_features=304, bias=True)
(4145): ReLU()
(4146): Linear(in_features=304, out_features=288, bias=True)
(4147): ReLU()
(4148): Linear(in_features=288, out_features=256, bias=True)
(4149): ReLU()
(4150): Linear(in_features=256, out_features=336, bias=True)
(4151): ReLU()
(4152): Linear(in_features=336, out_features=328, bias=True)
(4153): ReLU()
(4154): Linear(in_features=328, out_features=340, bias=True)
(4155): ReLU()
(4156): Linear(in_features=340, out_features=332, bias=True)
(4157): ReLU()
(4158): Linear(in_features=332, out_features=375, bias=True)
(4159): ReLU()
(4160): Linear(in_features=375, out_features=399, bias=True)
(4161): ReLU()
(4162): Linear(in_features=399, out_features=410, bias=True)
(4163): ReLU()
(4164): Linear(in_features=410, out_features=402, bias=True)
(4165): ReLU()
(4166): Linear(in_features=402, out_features=412, bias=True)
(4167): ReLU()
(4168): Linear(in_features=412, out_features=404, bias=True)
(4169): ReLU()
(4170): Linear(in_features=404, out_features=412, bias=True)
(4171): ReLU()
(4172): Linear(in_features=412, out_features=404, bias=True)
(4173): ReLU()
(4174): Linear(in_features=404, out_features=408, bias=True)
(4175): ReLU()
(4176): Linear(in_features=408, out_features=400, bias=True)
(4177): ReLU()
(4178): Linear(in_features=400, out_features=352, bias=True)
(4179): ReLU()
(4180): Linear(in_features=352, out_features=288, bias=True)
(4181): ReLU()
(4182): Linear(in_features=288, out_features=288, bias=True)
(4183): ReLU()
(4184): Linear(in_features=288, out_features=256, bias=True)
(4185): ReLU()
(4186): Linear(in_features=256, out_features=319, bias=True)
(4187): ReLU()
(4188): Linear(in_features=319, out_features=288, bias=True)
(4189): ReLU()
(4190): Linear(in_features=288, out_features=318, bias=True)
(4191): ReLU()
(4192): Linear(in_features=318, out_features=288, bias=True)
(4193): ReLU()
(4194): Linear(in_features=288, out_features=316, bias=True)
(4195): ReLU()
(4196): Linear(in_features=316, out_features=288, bias=True)
(4197): ReLU()
(4198): Linear(in_features=288, out_features=312, bias=True)
(4199): ReLU()
(4200): Linear(in_features=312, out_features=288, bias=True)
(4201): ReLU()
(4202): Linear(in_features=288, out_features=304, bias=True)
(4203): ReLU()
(4204): Linear(in_features=304, out_features=288, bias=True)
(4205): ReLU()
(4206): Linear(in_features=288, out_features=256, bias=True)
(4207): ReLU()
(4208): Linear(in_features=256, out_features=288, bias=True)
(4209): ReLU()
(4210): Linear(in_features=288, out_features=256, bias=True)
(4211): ReLU()
(4212): Linear(in_features=256, out_features=319, bias=True)
(4213): ReLU()
(4214): Linear(in_features=319, out_features=288, bias=True)
(4215): ReLU()
(4216): Linear(in_features=288, out_features=318, bias=True)
(4217): ReLU()
(4218): Linear(in_features=318, out_features=288, bias=True)
(4219): ReLU()
(4220): Linear(in_features=288, out_features=316, bias=True)
(4221): ReLU()
(4222): Linear(in_features=316, out_features=288, bias=True)
(4223): ReLU()
(4224): Linear(in_features=288, out_features=312, bias=True)
(4225): ReLU()
(4226): Linear(in_features=312, out_features=288, bias=True)
(4227): ReLU()
(4228): Linear(in_features=288, out_features=304, bias=True)
(4229): ReLU()
(4230): Linear(in_features=304, out_features=288, bias=True)
(4231): ReLU()
(4232): Linear(in_features=288, out_features=256, bias=True)
(4233): ReLU()
(4234): Linear(in_features=256, out_features=336, bias=True)
(4235): ReLU()
(4236): Linear(in_features=336, out_features=328, bias=True)
(4237): ReLU()
(4238): Linear(in_features=328, out_features=340, bias=True)
(4239): ReLU()
(4240): Linear(in_features=340, out_features=332, bias=True)
(4241): ReLU()
(4242): Linear(in_features=332, out_features=375, bias=True)
(4243): ReLU()
(4244): Linear(in_features=375, out_features=399, bias=True)
(4245): ReLU()
(4246): Linear(in_features=399, out_features=410, bias=True)
(4247): ReLU()
(4248): Linear(in_features=410, out_features=402, bias=True)
(4249): ReLU()
(4250): Linear(in_features=402, out_features=412, bias=True)
(4251): ReLU()
(4252): Linear(in_features=412, out_features=404, bias=True)
(4253): ReLU()
(4254): Linear(in_features=404, out_features=412, bias=True)
(4255): ReLU()
(4256): Linear(in_features=412, out_features=404, bias=True)
(4257): ReLU()
(4258): Linear(in_features=404, out_features=408, bias=True)
(4259): ReLU()
(4260): Linear(in_features=408, out_features=400, bias=True)
(4261): ReLU()
(4262): Linear(in_features=400, out_features=352, bias=True)
(4263): ReLU()
(4264): Linear(in_features=352, out_features=288, bias=True)
(4265): ReLU()
(4266): Linear(in_features=288, out_features=288, bias=True)
(4267): ReLU()
(4268): Linear(in_features=288, out_features=256, bias=True)
(4269): ReLU()
(4270): Linear(in_features=256, out_features=319, bias=True)
(4271): ReLU()
(4272): Linear(in_features=319, out_features=288, bias=True)
(4273): ReLU()
(4274): Linear(in_features=288, out_features=318, bias=True)
(4275): ReLU()
(4276): Linear(in_features=318, out_features=288, bias=True)
(4277): ReLU()
(4278): Linear(in_features=288, out_features=316, bias=True)
(4279): ReLU()
(4280): Linear(in_features=316, out_features=288, bias=True)
(4281): ReLU()
(4282): Linear(in_features=288, out_features=312, bias=True)
(4283): ReLU()
(4284): Linear(in_features=312, out_features=288, bias=True)
(4285): ReLU()
(4286): Linear(in_features=288, out_features=304, bias=True)
(4287): ReLU()
(4288): Linear(in_features=304, out_features=288, bias=True)
(4289): ReLU()
(4290): Linear(in_features=288, out_features=256, bias=True)
(4291): ReLU()
(4292): Linear(in_features=256, out_features=288, bias=True)
(4293): ReLU()
(4294): Linear(in_features=288, out_features=256, bias=True)
(4295): ReLU()
(4296): Linear(in_features=256, out_features=319, bias=True)
(4297): ReLU()
(4298): Linear(in_features=319, out_features=288, bias=True)
(4299): ReLU()
(4300): Linear(in_features=288, out_features=318, bias=True)
(4301): ReLU()
(4302): Linear(in_features=318, out_features=288, bias=True)
(4303): ReLU()
(4304): Linear(in_features=288, out_features=316, bias=True)
(4305): ReLU()
(4306): Linear(in_features=316, out_features=288, bias=True)
(4307): ReLU()
(4308): Linear(in_features=288, out_features=312, bias=True)
(4309): ReLU()
(4310): Linear(in_features=312, out_features=288, bias=True)
(4311): ReLU()
(4312): Linear(in_features=288, out_features=304, bias=True)
(4313): ReLU()
(4314): Linear(in_features=304, out_features=288, bias=True)
(4315): ReLU()
(4316): Linear(in_features=288, out_features=256, bias=True)
(4317): ReLU()
(4318): Linear(in_features=256, out_features=336, bias=True)
(4319): ReLU()
(4320): Linear(in_features=336, out_features=328, bias=True)
(4321): ReLU()
(4322): Linear(in_features=328, out_features=340, bias=True)
(4323): ReLU()
(4324): Linear(in_features=340, out_features=332, bias=True)
(4325): ReLU()
(4326): Linear(in_features=332, out_features=375, bias=True)
(4327): ReLU()
(4328): Linear(in_features=375, out_features=399, bias=True)
(4329): ReLU()
(4330): Linear(in_features=399, out_features=410, bias=True)
(4331): ReLU()
(4332): Linear(in_features=410, out_features=402, bias=True)
(4333): ReLU()
(4334): Linear(in_features=402, out_features=412, bias=True)
(4335): ReLU()
(4336): Linear(in_features=412, out_features=404, bias=True)
(4337): ReLU()
(4338): Linear(in_features=404, out_features=412, bias=True)
(4339): ReLU()
(4340): Linear(in_features=412, out_features=404, bias=True)
(4341): ReLU()
(4342): Linear(in_features=404, out_features=408, bias=True)
(4343): ReLU()
(4344): Linear(in_features=408, out_features=400, bias=True)
(4345): ReLU()
(4346): Linear(in_features=400, out_features=352, bias=True)
(4347): ReLU()
(4348): Linear(in_features=352, out_features=288, bias=True)
(4349): ReLU()
(4350): Linear(in_features=288, out_features=288, bias=True)
(4351): ReLU()
(4352): Linear(in_features=288, out_features=256, bias=True)
(4353): ReLU()
(4354): Linear(in_features=256, out_features=319, bias=True)
(4355): ReLU()
(4356): Linear(in_features=319, out_features=288, bias=True)
(4357): ReLU()
(4358): Linear(in_features=288, out_features=318, bias=True)
(4359): ReLU()
(4360): Linear(in_features=318, out_features=288, bias=True)
(4361): ReLU()
(4362): Linear(in_features=288, out_features=316, bias=True)
(4363): ReLU()
(4364): Linear(in_features=316, out_features=288, bias=True)
(4365): ReLU()
(4366): Linear(in_features=288, out_features=312, bias=True)
(4367): ReLU()
(4368): Linear(in_features=312, out_features=288, bias=True)
(4369): ReLU()
(4370): Linear(in_features=288, out_features=304, bias=True)
(4371): ReLU()
(4372): Linear(in_features=304, out_features=288, bias=True)
(4373): ReLU()
(4374): Linear(in_features=288, out_features=256, bias=True)
(4375): ReLU()
(4376): Linear(in_features=256, out_features=288, bias=True)
(4377): ReLU()
(4378): Linear(in_features=288, out_features=256, bias=True)
(4379): ReLU()
(4380): Linear(in_features=256, out_features=319, bias=True)
(4381): ReLU()
(4382): Linear(in_features=319, out_features=288, bias=True)
(4383): ReLU()
(4384): Linear(in_features=288, out_features=318, bias=True)
(4385): ReLU()
(4386): Linear(in_features=318, out_features=288, bias=True)
(4387): ReLU()
(4388): Linear(in_features=288, out_features=316, bias=True)
(4389): ReLU()
(4390): Linear(in_features=316, out_features=288, bias=True)
(4391): ReLU()
(4392): Linear(in_features=288, out_features=312, bias=True)
(4393): ReLU()
(4394): Linear(in_features=312, out_features=288, bias=True)
(4395): ReLU()
(4396): Linear(in_features=288, out_features=304, bias=True)
(4397): ReLU()
(4398): Linear(in_features=304, out_features=288, bias=True)
(4399): ReLU()
(4400): Linear(in_features=288, out_features=256, bias=True)
(4401): ReLU()
(4402): Linear(in_features=256, out_features=336, bias=True)
(4403): ReLU()
(4404): Linear(in_features=336, out_features=328, bias=True)
(4405): ReLU()
(4406): Linear(in_features=328, out_features=340, bias=True)
(4407): ReLU()
(4408): Linear(in_features=340, out_features=332, bias=True)
(4409): ReLU()
(4410): Linear(in_features=332, out_features=375, bias=True)
(4411): ReLU()
(4412): Linear(in_features=375, out_features=399, bias=True)
(4413): ReLU()
(4414): Linear(in_features=399, out_features=410, bias=True)
(4415): ReLU()
(4416): Linear(in_features=410, out_features=402, bias=True)
(4417): ReLU()
(4418): Linear(in_features=402, out_features=412, bias=True)
(4419): ReLU()
(4420): Linear(in_features=412, out_features=404, bias=True)
(4421): ReLU()
(4422): Linear(in_features=404, out_features=412, bias=True)
(4423): ReLU()
(4424): Linear(in_features=412, out_features=404, bias=True)
(4425): ReLU()
(4426): Linear(in_features=404, out_features=408, bias=True)
(4427): ReLU()
(4428): Linear(in_features=408, out_features=400, bias=True)
(4429): ReLU()
(4430): Linear(in_features=400, out_features=352, bias=True)
(4431): ReLU()
(4432): Linear(in_features=352, out_features=288, bias=True)
(4433): ReLU()
(4434): Linear(in_features=288, out_features=288, bias=True)
(4435): ReLU()
(4436): Linear(in_features=288, out_features=256, bias=True)
(4437): ReLU()
(4438): Linear(in_features=256, out_features=319, bias=True)
(4439): ReLU()
(4440): Linear(in_features=319, out_features=288, bias=True)
(4441): ReLU()
(4442): Linear(in_features=288, out_features=318, bias=True)
(4443): ReLU()
(4444): Linear(in_features=318, out_features=288, bias=True)
(4445): ReLU()
(4446): Linear(in_features=288, out_features=316, bias=True)
(4447): ReLU()
(4448): Linear(in_features=316, out_features=288, bias=True)
(4449): ReLU()
(4450): Linear(in_features=288, out_features=312, bias=True)
(4451): ReLU()
(4452): Linear(in_features=312, out_features=288, bias=True)
(4453): ReLU()
(4454): Linear(in_features=288, out_features=304, bias=True)
(4455): ReLU()
(4456): Linear(in_features=304, out_features=288, bias=True)
(4457): ReLU()
(4458): Linear(in_features=288, out_features=256, bias=True)
(4459): ReLU()
(4460): Linear(in_features=256, out_features=288, bias=True)
(4461): ReLU()
(4462): Linear(in_features=288, out_features=256, bias=True)
(4463): ReLU()
(4464): Linear(in_features=256, out_features=319, bias=True)
(4465): ReLU()
(4466): Linear(in_features=319, out_features=288, bias=True)
(4467): ReLU()
(4468): Linear(in_features=288, out_features=318, bias=True)
(4469): ReLU()
(4470): Linear(in_features=318, out_features=288, bias=True)
(4471): ReLU()
(4472): Linear(in_features=288, out_features=316, bias=True)
(4473): ReLU()
(4474): Linear(in_features=316, out_features=288, bias=True)
(4475): ReLU()
(4476): Linear(in_features=288, out_features=312, bias=True)
(4477): ReLU()
(4478): Linear(in_features=312, out_features=288, bias=True)
(4479): ReLU()
(4480): Linear(in_features=288, out_features=304, bias=True)
(4481): ReLU()
(4482): Linear(in_features=304, out_features=288, bias=True)
(4483): ReLU()
(4484): Linear(in_features=288, out_features=256, bias=True)
(4485): ReLU()
(4486): Linear(in_features=256, out_features=336, bias=True)
(4487): ReLU()
(4488): Linear(in_features=336, out_features=328, bias=True)
(4489): ReLU()
(4490): Linear(in_features=328, out_features=340, bias=True)
(4491): ReLU()
(4492): Linear(in_features=340, out_features=332, bias=True)
(4493): ReLU()
(4494): Linear(in_features=332, out_features=375, bias=True)
(4495): ReLU()
(4496): Linear(in_features=375, out_features=399, bias=True)
(4497): ReLU()
(4498): Linear(in_features=399, out_features=410, bias=True)
(4499): ReLU()
(4500): Linear(in_features=410, out_features=402, bias=True)
(4501): ReLU()
(4502): Linear(in_features=402, out_features=412, bias=True)
(4503): ReLU()
(4504): Linear(in_features=412, out_features=404, bias=True)
(4505): ReLU()
(4506): Linear(in_features=404, out_features=412, bias=True)
(4507): ReLU()
(4508): Linear(in_features=412, out_features=404, bias=True)
(4509): ReLU()
(4510): Linear(in_features=404, out_features=408, bias=True)
(4511): ReLU()
(4512): Linear(in_features=408, out_features=400, bias=True)
(4513): ReLU()
(4514): Linear(in_features=400, out_features=352, bias=True)
(4515): ReLU()
(4516): Linear(in_features=352, out_features=288, bias=True)
(4517): ReLU()
(4518): Linear(in_features=288, out_features=288, bias=True)
(4519): ReLU()
(4520): Linear(in_features=288, out_features=256, bias=True)
(4521): ReLU()
(4522): Linear(in_features=256, out_features=319, bias=True)
(4523): ReLU()
(4524): Linear(in_features=319, out_features=288, bias=True)
(4525): ReLU()
(4526): Linear(in_features=288, out_features=318, bias=True)
(4527): ReLU()
(4528): Linear(in_features=318, out_features=288, bias=True)
(4529): ReLU()
(4530): Linear(in_features=288, out_features=316, bias=True)
(4531): ReLU()
(4532): Linear(in_features=316, out_features=288, bias=True)
(4533): ReLU()
(4534): Linear(in_features=288, out_features=312, bias=True)
(4535): ReLU()
(4536): Linear(in_features=312, out_features=288, bias=True)
(4537): ReLU()
(4538): Linear(in_features=288, out_features=304, bias=True)
(4539): ReLU()
(4540): Linear(in_features=304, out_features=288, bias=True)
(4541): ReLU()
(4542): Linear(in_features=288, out_features=256, bias=True)
(4543): ReLU()
(4544): Linear(in_features=256, out_features=288, bias=True)
(4545): ReLU()
(4546): Linear(in_features=288, out_features=256, bias=True)
(4547): ReLU()
(4548): Linear(in_features=256, out_features=319, bias=True)
(4549): ReLU()
(4550): Linear(in_features=319, out_features=288, bias=True)
(4551): ReLU()
(4552): Linear(in_features=288, out_features=318, bias=True)
(4553): ReLU()
(4554): Linear(in_features=318, out_features=288, bias=True)
(4555): ReLU()
(4556): Linear(in_features=288, out_features=316, bias=True)
(4557): ReLU()
(4558): Linear(in_features=316, out_features=288, bias=True)
(4559): ReLU()
(4560): Linear(in_features=288, out_features=312, bias=True)
(4561): ReLU()
(4562): Linear(in_features=312, out_features=288, bias=True)
(4563): ReLU()
(4564): Linear(in_features=288, out_features=304, bias=True)
(4565): ReLU()
(4566): Linear(in_features=304, out_features=288, bias=True)
(4567): ReLU()
(4568): Linear(in_features=288, out_features=256, bias=True)
(4569): ReLU()
(4570): Linear(in_features=256, out_features=336, bias=True)
(4571): ReLU()
(4572): Linear(in_features=336, out_features=328, bias=True)
(4573): ReLU()
(4574): Linear(in_features=328, out_features=340, bias=True)
(4575): ReLU()
(4576): Linear(in_features=340, out_features=332, bias=True)
(4577): ReLU()
(4578): Linear(in_features=332, out_features=375, bias=True)
(4579): ReLU()
(4580): Linear(in_features=375, out_features=399, bias=True)
(4581): ReLU()
(4582): Linear(in_features=399, out_features=410, bias=True)
(4583): ReLU()
(4584): Linear(in_features=410, out_features=402, bias=True)
(4585): ReLU()
(4586): Linear(in_features=402, out_features=412, bias=True)
(4587): ReLU()
(4588): Linear(in_features=412, out_features=404, bias=True)
(4589): ReLU()
(4590): Linear(in_features=404, out_features=412, bias=True)
(4591): ReLU()
(4592): Linear(in_features=412, out_features=404, bias=True)
(4593): ReLU()
(4594): Linear(in_features=404, out_features=408, bias=True)
(4595): ReLU()
(4596): Linear(in_features=408, out_features=400, bias=True)
(4597): ReLU()
(4598): Linear(in_features=400, out_features=352, bias=True)
(4599): ReLU()
(4600): Linear(in_features=352, out_features=288, bias=True)
(4601): ReLU()
(4602): Linear(in_features=288, out_features=288, bias=True)
(4603): ReLU()
(4604): Linear(in_features=288, out_features=256, bias=True)
(4605): ReLU()
(4606): Linear(in_features=256, out_features=319, bias=True)
(4607): ReLU()
(4608): Linear(in_features=319, out_features=288, bias=True)
(4609): ReLU()
(4610): Linear(in_features=288, out_features=318, bias=True)
(4611): ReLU()
(4612): Linear(in_features=318, out_features=288, bias=True)
(4613): ReLU()
(4614): Linear(in_features=288, out_features=316, bias=True)
(4615): ReLU()
(4616): Linear(in_features=316, out_features=288, bias=True)
(4617): ReLU()
(4618): Linear(in_features=288, out_features=312, bias=True)
(4619): ReLU()
(4620): Linear(in_features=312, out_features=288, bias=True)
(4621): ReLU()
(4622): Linear(in_features=288, out_features=304, bias=True)
(4623): ReLU()
(4624): Linear(in_features=304, out_features=288, bias=True)
(4625): ReLU()
(4626): Linear(in_features=288, out_features=256, bias=True)
(4627): ReLU()
(4628): Linear(in_features=256, out_features=288, bias=True)
(4629): ReLU()
(4630): Linear(in_features=288, out_features=256, bias=True)
(4631): ReLU()
(4632): Linear(in_features=256, out_features=319, bias=True)
(4633): ReLU()
(4634): Linear(in_features=319, out_features=288, bias=True)
(4635): ReLU()
(4636): Linear(in_features=288, out_features=318, bias=True)
(4637): ReLU()
(4638): Linear(in_features=318, out_features=288, bias=True)
(4639): ReLU()
(4640): Linear(in_features=288, out_features=316, bias=True)
(4641): ReLU()
(4642): Linear(in_features=316, out_features=288, bias=True)
(4643): ReLU()
(4644): Linear(in_features=288, out_features=312, bias=True)
(4645): ReLU()
(4646): Linear(in_features=312, out_features=288, bias=True)
(4647): ReLU()
(4648): Linear(in_features=288, out_features=304, bias=True)
(4649): ReLU()
(4650): Linear(in_features=304, out_features=288, bias=True)
(4651): ReLU()
(4652): Linear(in_features=288, out_features=256, bias=True)
(4653): ReLU()
(4654): Linear(in_features=256, out_features=336, bias=True)
(4655): ReLU()
(4656): Linear(in_features=336, out_features=328, bias=True)
(4657): ReLU()
(4658): Linear(in_features=328, out_features=340, bias=True)
(4659): ReLU()
(4660): Linear(in_features=340, out_features=332, bias=True)
(4661): ReLU()
(4662): Linear(in_features=332, out_features=375, bias=True)
(4663): ReLU()
(4664): Linear(in_features=375, out_features=399, bias=True)
(4665): ReLU()
(4666): Linear(in_features=399, out_features=410, bias=True)
(4667): ReLU()
(4668): Linear(in_features=410, out_features=402, bias=True)
(4669): ReLU()
(4670): Linear(in_features=402, out_features=412, bias=True)
(4671): ReLU()
(4672): Linear(in_features=412, out_features=404, bias=True)
(4673): ReLU()
(4674): Linear(in_features=404, out_features=412, bias=True)
(4675): ReLU()
(4676): Linear(in_features=412, out_features=404, bias=True)
(4677): ReLU()
(4678): Linear(in_features=404, out_features=408, bias=True)
(4679): ReLU()
(4680): Linear(in_features=408, out_features=400, bias=True)
(4681): ReLU()
(4682): Linear(in_features=400, out_features=352, bias=True)
(4683): ReLU()
(4684): Linear(in_features=352, out_features=288, bias=True)
(4685): ReLU()
(4686): Linear(in_features=288, out_features=288, bias=True)
(4687): ReLU()
(4688): Linear(in_features=288, out_features=256, bias=True)
(4689): ReLU()
(4690): Linear(in_features=256, out_features=319, bias=True)
(4691): ReLU()
(4692): Linear(in_features=319, out_features=288, bias=True)
(4693): ReLU()
(4694): Linear(in_features=288, out_features=318, bias=True)
(4695): ReLU()
(4696): Linear(in_features=318, out_features=288, bias=True)
(4697): ReLU()
(4698): Linear(in_features=288, out_features=316, bias=True)
(4699): ReLU()
(4700): Linear(in_features=316, out_features=288, bias=True)
(4701): ReLU()
(4702): Linear(in_features=288, out_features=312, bias=True)
(4703): ReLU()
(4704): Linear(in_features=312, out_features=288, bias=True)
(4705): ReLU()
(4706): Linear(in_features=288, out_features=304, bias=True)
(4707): ReLU()
(4708): Linear(in_features=304, out_features=288, bias=True)
(4709): ReLU()
(4710): Linear(in_features=288, out_features=256, bias=True)
(4711): ReLU()
(4712): Linear(in_features=256, out_features=288, bias=True)
(4713): ReLU()
(4714): Linear(in_features=288, out_features=256, bias=True)
(4715): ReLU()
(4716): Linear(in_features=256, out_features=319, bias=True)
(4717): ReLU()
(4718): Linear(in_features=319, out_features=288, bias=True)
(4719): ReLU()
(4720): Linear(in_features=288, out_features=318, bias=True)
(4721): ReLU()
(4722): Linear(in_features=318, out_features=288, bias=True)
(4723): ReLU()
(4724): Linear(in_features=288, out_features=316, bias=True)
(4725): ReLU()
(4726): Linear(in_features=316, out_features=288, bias=True)
(4727): ReLU()
(4728): Linear(in_features=288, out_features=312, bias=True)
(4729): ReLU()
(4730): Linear(in_features=312, out_features=288, bias=True)
(4731): ReLU()
(4732): Linear(in_features=288, out_features=304, bias=True)
(4733): ReLU()
(4734): Linear(in_features=304, out_features=288, bias=True)
(4735): ReLU()
(4736): Linear(in_features=288, out_features=256, bias=True)
(4737): ReLU()
(4738): Linear(in_features=256, out_features=336, bias=True)
(4739): ReLU()
(4740): Linear(in_features=336, out_features=328, bias=True)
(4741): ReLU()
(4742): Linear(in_features=328, out_features=340, bias=True)
(4743): ReLU()
(4744): Linear(in_features=340, out_features=332, bias=True)
(4745): ReLU()
(4746): Linear(in_features=332, out_features=375, bias=True)
(4747): ReLU()
(4748): Linear(in_features=375, out_features=399, bias=True)
(4749): ReLU()
(4750): Linear(in_features=399, out_features=410, bias=True)
(4751): ReLU()
(4752): Linear(in_features=410, out_features=402, bias=True)
(4753): ReLU()
(4754): Linear(in_features=402, out_features=412, bias=True)
(4755): ReLU()
(4756): Linear(in_features=412, out_features=404, bias=True)
(4757): ReLU()
(4758): Linear(in_features=404, out_features=412, bias=True)
(4759): ReLU()
(4760): Linear(in_features=412, out_features=404, bias=True)
(4761): ReLU()
(4762): Linear(in_features=404, out_features=408, bias=True)
(4763): ReLU()
(4764): Linear(in_features=408, out_features=400, bias=True)
(4765): ReLU()
(4766): Linear(in_features=400, out_features=352, bias=True)
(4767): ReLU()
(4768): Linear(in_features=352, out_features=288, bias=True)
(4769): ReLU()
(4770): Linear(in_features=288, out_features=288, bias=True)
(4771): ReLU()
(4772): Linear(in_features=288, out_features=256, bias=True)
(4773): ReLU()
(4774): Linear(in_features=256, out_features=319, bias=True)
(4775): ReLU()
(4776): Linear(in_features=319, out_features=288, bias=True)
(4777): ReLU()
(4778): Linear(in_features=288, out_features=318, bias=True)
(4779): ReLU()
(4780): Linear(in_features=318, out_features=288, bias=True)
(4781): ReLU()
(4782): Linear(in_features=288, out_features=316, bias=True)
(4783): ReLU()
(4784): Linear(in_features=316, out_features=288, bias=True)
(4785): ReLU()
(4786): Linear(in_features=288, out_features=312, bias=True)
(4787): ReLU()
(4788): Linear(in_features=312, out_features=288, bias=True)
(4789): ReLU()
(4790): Linear(in_features=288, out_features=304, bias=True)
(4791): ReLU()
(4792): Linear(in_features=304, out_features=288, bias=True)
(4793): ReLU()
(4794): Linear(in_features=288, out_features=256, bias=True)
(4795): ReLU()
(4796): Linear(in_features=256, out_features=288, bias=True)
(4797): ReLU()
(4798): Linear(in_features=288, out_features=256, bias=True)
(4799): ReLU()
(4800): Linear(in_features=256, out_features=319, bias=True)
(4801): ReLU()
(4802): Linear(in_features=319, out_features=288, bias=True)
(4803): ReLU()
(4804): Linear(in_features=288, out_features=318, bias=True)
(4805): ReLU()
(4806): Linear(in_features=318, out_features=288, bias=True)
(4807): ReLU()
(4808): Linear(in_features=288, out_features=316, bias=True)
(4809): ReLU()
(4810): Linear(in_features=316, out_features=288, bias=True)
(4811): ReLU()
(4812): Linear(in_features=288, out_features=312, bias=True)
(4813): ReLU()
(4814): Linear(in_features=312, out_features=288, bias=True)
(4815): ReLU()
(4816): Linear(in_features=288, out_features=304, bias=True)
(4817): ReLU()
(4818): Linear(in_features=304, out_features=288, bias=True)
(4819): ReLU()
(4820): Linear(in_features=288, out_features=256, bias=True)
(4821): ReLU()
(4822): Linear(in_features=256, out_features=336, bias=True)
(4823): ReLU()
(4824): Linear(in_features=336, out_features=328, bias=True)
(4825): ReLU()
(4826): Linear(in_features=328, out_features=340, bias=True)
(4827): ReLU()
(4828): Linear(in_features=340, out_features=332, bias=True)
(4829): ReLU()
(4830): Linear(in_features=332, out_features=375, bias=True)
(4831): ReLU()
(4832): Linear(in_features=375, out_features=399, bias=True)
(4833): ReLU()
(4834): Linear(in_features=399, out_features=410, bias=True)
(4835): ReLU()
(4836): Linear(in_features=410, out_features=402, bias=True)
(4837): ReLU()
(4838): Linear(in_features=402, out_features=412, bias=True)
(4839): ReLU()
(4840): Linear(in_features=412, out_features=404, bias=True)
(4841): ReLU()
(4842): Linear(in_features=404, out_features=412, bias=True)
(4843): ReLU()
(4844): Linear(in_features=412, out_features=404, bias=True)
(4845): ReLU()
(4846): Linear(in_features=404, out_features=408, bias=True)
(4847): ReLU()
(4848): Linear(in_features=408, out_features=400, bias=True)
(4849): ReLU()
(4850): Linear(in_features=400, out_features=352, bias=True)
(4851): ReLU()
(4852): Linear(in_features=352, out_features=288, bias=True)
(4853): ReLU()
(4854): Linear(in_features=288, out_features=288, bias=True)
(4855): ReLU()
(4856): Linear(in_features=288, out_features=256, bias=True)
(4857): ReLU()
(4858): Linear(in_features=256, out_features=319, bias=True)
(4859): ReLU()
(4860): Linear(in_features=319, out_features=288, bias=True)
(4861): ReLU()
(4862): Linear(in_features=288, out_features=318, bias=True)
(4863): ReLU()
(4864): Linear(in_features=318, out_features=288, bias=True)
(4865): ReLU()
(4866): Linear(in_features=288, out_features=316, bias=True)
(4867): ReLU()
(4868): Linear(in_features=316, out_features=288, bias=True)
(4869): ReLU()
(4870): Linear(in_features=288, out_features=312, bias=True)
(4871): ReLU()
(4872): Linear(in_features=312, out_features=288, bias=True)
(4873): ReLU()
(4874): Linear(in_features=288, out_features=304, bias=True)
(4875): ReLU()
(4876): Linear(in_features=304, out_features=288, bias=True)
(4877): ReLU()
(4878): Linear(in_features=288, out_features=256, bias=True)
(4879): ReLU()
(4880): Linear(in_features=256, out_features=288, bias=True)
(4881): ReLU()
(4882): Linear(in_features=288, out_features=256, bias=True)
(4883): ReLU()
(4884): Linear(in_features=256, out_features=319, bias=True)
(4885): ReLU()
(4886): Linear(in_features=319, out_features=288, bias=True)
(4887): ReLU()
(4888): Linear(in_features=288, out_features=318, bias=True)
(4889): ReLU()
(4890): Linear(in_features=318, out_features=288, bias=True)
(4891): ReLU()
(4892): Linear(in_features=288, out_features=316, bias=True)
(4893): ReLU()
(4894): Linear(in_features=316, out_features=288, bias=True)
(4895): ReLU()
(4896): Linear(in_features=288, out_features=312, bias=True)
(4897): ReLU()
(4898): Linear(in_features=312, out_features=288, bias=True)
(4899): ReLU()
(4900): Linear(in_features=288, out_features=304, bias=True)
(4901): ReLU()
(4902): Linear(in_features=304, out_features=288, bias=True)
(4903): ReLU()
(4904): Linear(in_features=288, out_features=256, bias=True)
(4905): ReLU()
(4906): Linear(in_features=256, out_features=336, bias=True)
(4907): ReLU()
(4908): Linear(in_features=336, out_features=328, bias=True)
(4909): ReLU()
(4910): Linear(in_features=328, out_features=340, bias=True)
(4911): ReLU()
(4912): Linear(in_features=340, out_features=332, bias=True)
(4913): ReLU()
(4914): Linear(in_features=332, out_features=375, bias=True)
(4915): ReLU()
(4916): Linear(in_features=375, out_features=399, bias=True)
(4917): ReLU()
(4918): Linear(in_features=399, out_features=410, bias=True)
(4919): ReLU()
(4920): Linear(in_features=410, out_features=402, bias=True)
(4921): ReLU()
(4922): Linear(in_features=402, out_features=412, bias=True)
(4923): ReLU()
(4924): Linear(in_features=412, out_features=404, bias=True)
(4925): ReLU()
(4926): Linear(in_features=404, out_features=412, bias=True)
(4927): ReLU()
(4928): Linear(in_features=412, out_features=404, bias=True)
(4929): ReLU()
(4930): Linear(in_features=404, out_features=408, bias=True)
(4931): ReLU()
(4932): Linear(in_features=408, out_features=400, bias=True)
(4933): ReLU()
(4934): Linear(in_features=400, out_features=352, bias=True)
(4935): ReLU()
(4936): Linear(in_features=352, out_features=288, bias=True)
(4937): ReLU()
(4938): Linear(in_features=288, out_features=288, bias=True)
(4939): ReLU()
(4940): Linear(in_features=288, out_features=256, bias=True)
(4941): ReLU()
(4942): Linear(in_features=256, out_features=319, bias=True)
(4943): ReLU()
(4944): Linear(in_features=319, out_features=288, bias=True)
(4945): ReLU()
(4946): Linear(in_features=288, out_features=318, bias=True)
(4947): ReLU()
(4948): Linear(in_features=318, out_features=288, bias=True)
(4949): ReLU()
(4950): Linear(in_features=288, out_features=316, bias=True)
(4951): ReLU()
(4952): Linear(in_features=316, out_features=288, bias=True)
(4953): ReLU()
(4954): Linear(in_features=288, out_features=312, bias=True)
(4955): ReLU()
(4956): Linear(in_features=312, out_features=288, bias=True)
(4957): ReLU()
(4958): Linear(in_features=288, out_features=304, bias=True)
(4959): ReLU()
(4960): Linear(in_features=304, out_features=288, bias=True)
(4961): ReLU()
(4962): Linear(in_features=288, out_features=256, bias=True)
(4963): ReLU()
(4964): Linear(in_features=256, out_features=288, bias=True)
(4965): ReLU()
(4966): Linear(in_features=288, out_features=256, bias=True)
(4967): ReLU()
(4968): Linear(in_features=256, out_features=319, bias=True)
(4969): ReLU()
(4970): Linear(in_features=319, out_features=288, bias=True)
(4971): ReLU()
(4972): Linear(in_features=288, out_features=318, bias=True)
(4973): ReLU()
(4974): Linear(in_features=318, out_features=288, bias=True)
(4975): ReLU()
(4976): Linear(in_features=288, out_features=316, bias=True)
(4977): ReLU()
(4978): Linear(in_features=316, out_features=288, bias=True)
(4979): ReLU()
(4980): Linear(in_features=288, out_features=312, bias=True)
(4981): ReLU()
(4982): Linear(in_features=312, out_features=288, bias=True)
(4983): ReLU()
(4984): Linear(in_features=288, out_features=304, bias=True)
(4985): ReLU()
(4986): Linear(in_features=304, out_features=288, bias=True)
(4987): ReLU()
(4988): Linear(in_features=288, out_features=256, bias=True)
(4989): ReLU()
(4990): Linear(in_features=256, out_features=336, bias=True)
(4991): ReLU()
(4992): Linear(in_features=336, out_features=328, bias=True)
(4993): ReLU()
(4994): Linear(in_features=328, out_features=340, bias=True)
(4995): ReLU()
(4996): Linear(in_features=340, out_features=332, bias=True)
(4997): ReLU()
(4998): Linear(in_features=332, out_features=375, bias=True)
(4999): ReLU()
(5000): Linear(in_features=375, out_features=399, bias=True)
(5001): ReLU()
(5002): Linear(in_features=399, out_features=410, bias=True)
(5003): ReLU()
(5004): Linear(in_features=410, out_features=402, bias=True)
(5005): ReLU()
(5006): Linear(in_features=402, out_features=412, bias=True)
(5007): ReLU()
(5008): Linear(in_features=412, out_features=404, bias=True)
(5009): ReLU()
(5010): Linear(in_features=404, out_features=412, bias=True)
(5011): ReLU()
(5012): Linear(in_features=412, out_features=404, bias=True)
(5013): ReLU()
(5014): Linear(in_features=404, out_features=408, bias=True)
(5015): ReLU()
(5016): Linear(in_features=408, out_features=400, bias=True)
(5017): ReLU()
(5018): Linear(in_features=400, out_features=352, bias=True)
(5019): ReLU()
(5020): Linear(in_features=352, out_features=288, bias=True)
(5021): ReLU()
(5022): Linear(in_features=288, out_features=288, bias=True)
(5023): ReLU()
(5024): Linear(in_features=288, out_features=256, bias=True)
(5025): ReLU()
(5026): Linear(in_features=256, out_features=319, bias=True)
(5027): ReLU()
(5028): Linear(in_features=319, out_features=288, bias=True)
(5029): ReLU()
(5030): Linear(in_features=288, out_features=318, bias=True)
(5031): ReLU()
(5032): Linear(in_features=318, out_features=288, bias=True)
(5033): ReLU()
(5034): Linear(in_features=288, out_features=316, bias=True)
(5035): ReLU()
(5036): Linear(in_features=316, out_features=288, bias=True)
(5037): ReLU()
(5038): Linear(in_features=288, out_features=312, bias=True)
(5039): ReLU()
(5040): Linear(in_features=312, out_features=288, bias=True)
(5041): ReLU()
(5042): Linear(in_features=288, out_features=304, bias=True)
(5043): ReLU()
(5044): Linear(in_features=304, out_features=288, bias=True)
(5045): ReLU()
(5046): Linear(in_features=288, out_features=256, bias=True)
(5047): ReLU()
(5048): Linear(in_features=256, out_features=288, bias=True)
(5049): ReLU()
(5050): Linear(in_features=288, out_features=256, bias=True)
(5051): ReLU()
(5052): Linear(in_features=256, out_features=319, bias=True)
(5053): ReLU()
(5054): Linear(in_features=319, out_features=288, bias=True)
(5055): ReLU()
(5056): Linear(in_features=288, out_features=318, bias=True)
(5057): ReLU()
(5058): Linear(in_features=318, out_features=288, bias=True)
(5059): ReLU()
(5060): Linear(in_features=288, out_features=316, bias=True)
(5061): ReLU()
(5062): Linear(in_features=316, out_features=288, bias=True)
(5063): ReLU()
(5064): Linear(in_features=288, out_features=312, bias=True)
(5065): ReLU()
(5066): Linear(in_features=312, out_features=288, bias=True)
(5067): ReLU()
(5068): Linear(in_features=288, out_features=304, bias=True)
(5069): ReLU()
(5070): Linear(in_features=304, out_features=288, bias=True)
(5071): ReLU()
(5072): Linear(in_features=288, out_features=256, bias=True)
(5073): ReLU()
(5074): Linear(in_features=256, out_features=336, bias=True)
(5075): ReLU()
(5076): Linear(in_features=336, out_features=328, bias=True)
(5077): ReLU()
(5078): Linear(in_features=328, out_features=340, bias=True)
(5079): ReLU()
(5080): Linear(in_features=340, out_features=332, bias=True)
(5081): ReLU()
(5082): Linear(in_features=332, out_features=375, bias=True)
(5083): ReLU()
(5084): Linear(in_features=375, out_features=399, bias=True)
(5085): ReLU()
(5086): Linear(in_features=399, out_features=410, bias=True)
(5087): ReLU()
(5088): Linear(in_features=410, out_features=402, bias=True)
(5089): ReLU()
(5090): Linear(in_features=402, out_features=412, bias=True)
(5091): ReLU()
(5092): Linear(in_features=412, out_features=404, bias=True)
(5093): ReLU()
(5094): Linear(in_features=404, out_features=412, bias=True)
(5095): ReLU()
(5096): Linear(in_features=412, out_features=404, bias=True)
(5097): ReLU()
(5098): Linear(in_features=404, out_features=408, bias=True)
(5099): ReLU()
(5100): Linear(in_features=408, out_features=400, bias=True)
(5101): ReLU()
(5102): Linear(in_features=400, out_features=352, bias=True)
(5103): ReLU()
(5104): Linear(in_features=352, out_features=288, bias=True)
(5105): ReLU()
(5106): Linear(in_features=288, out_features=288, bias=True)
(5107): ReLU()
(5108): Linear(in_features=288, out_features=256, bias=True)
(5109): ReLU()
(5110): Linear(in_features=256, out_features=319, bias=True)
(5111): ReLU()
(5112): Linear(in_features=319, out_features=288, bias=True)
(5113): ReLU()
(5114): Linear(in_features=288, out_features=318, bias=True)
(5115): ReLU()
(5116): Linear(in_features=318, out_features=288, bias=True)
(5117): ReLU()
(5118): Linear(in_features=288, out_features=316, bias=True)
(5119): ReLU()
(5120): Linear(in_features=316, out_features=288, bias=True)
(5121): ReLU()
(5122): Linear(in_features=288, out_features=312, bias=True)
(5123): ReLU()
(5124): Linear(in_features=312, out_features=288, bias=True)
(5125): ReLU()
(5126): Linear(in_features=288, out_features=304, bias=True)
(5127): ReLU()
(5128): Linear(in_features=304, out_features=288, bias=True)
(5129): ReLU()
(5130): Linear(in_features=288, out_features=256, bias=True)
(5131): ReLU()
(5132): Linear(in_features=256, out_features=288, bias=True)
(5133): ReLU()
(5134): Linear(in_features=288, out_features=256, bias=True)
(5135): ReLU()
(5136): Linear(in_features=256, out_features=319, bias=True)
(5137): ReLU()
(5138): Linear(in_features=319, out_features=288, bias=True)
(5139): ReLU()
(5140): Linear(in_features=288, out_features=318, bias=True)
(5141): ReLU()
(5142): Linear(in_features=318, out_features=288, bias=True)
(5143): ReLU()
(5144): Linear(in_features=288, out_features=316, bias=True)
(5145): ReLU()
(5146): Linear(in_features=316, out_features=288, bias=True)
(5147): ReLU()
(5148): Linear(in_features=288, out_features=312, bias=True)
(5149): ReLU()
(5150): Linear(in_features=312, out_features=288, bias=True)
(5151): ReLU()
(5152): Linear(in_features=288, out_features=304, bias=True)
(5153): ReLU()
(5154): Linear(in_features=304, out_features=288, bias=True)
(5155): ReLU()
(5156): Linear(in_features=288, out_features=256, bias=True)
(5157): ReLU()
(5158): Linear(in_features=256, out_features=336, bias=True)
(5159): ReLU()
(5160): Linear(in_features=336, out_features=328, bias=True)
(5161): ReLU()
(5162): Linear(in_features=328, out_features=340, bias=True)
(5163): ReLU()
(5164): Linear(in_features=340, out_features=332, bias=True)
(5165): ReLU()
(5166): Linear(in_features=332, out_features=375, bias=True)
(5167): ReLU()
(5168): Linear(in_features=375, out_features=399, bias=True)
(5169): ReLU()
(5170): Linear(in_features=399, out_features=410, bias=True)
(5171): ReLU()
(5172): Linear(in_features=410, out_features=402, bias=True)
(5173): ReLU()
(5174): Linear(in_features=402, out_features=412, bias=True)
(5175): ReLU()
(5176): Linear(in_features=412, out_features=404, bias=True)
(5177): ReLU()
(5178): Linear(in_features=404, out_features=412, bias=True)
(5179): ReLU()
(5180): Linear(in_features=412, out_features=404, bias=True)
(5181): ReLU()
(5182): Linear(in_features=404, out_features=408, bias=True)
(5183): ReLU()
(5184): Linear(in_features=408, out_features=400, bias=True)
(5185): ReLU()
(5186): Linear(in_features=400, out_features=352, bias=True)
(5187): ReLU()
(5188): Linear(in_features=352, out_features=288, bias=True)
(5189): ReLU()
(5190): Linear(in_features=288, out_features=288, bias=True)
(5191): ReLU()
(5192): Linear(in_features=288, out_features=256, bias=True)
(5193): ReLU()
(5194): Linear(in_features=256, out_features=319, bias=True)
(5195): ReLU()
(5196): Linear(in_features=319, out_features=288, bias=True)
(5197): ReLU()
(5198): Linear(in_features=288, out_features=318, bias=True)
(5199): ReLU()
(5200): Linear(in_features=318, out_features=288, bias=True)
(5201): ReLU()
(5202): Linear(in_features=288, out_features=316, bias=True)
(5203): ReLU()
(5204): Linear(in_features=316, out_features=288, bias=True)
(5205): ReLU()
(5206): Linear(in_features=288, out_features=312, bias=True)
(5207): ReLU()
(5208): Linear(in_features=312, out_features=288, bias=True)
(5209): ReLU()
(5210): Linear(in_features=288, out_features=304, bias=True)
(5211): ReLU()
(5212): Linear(in_features=304, out_features=288, bias=True)
(5213): ReLU()
(5214): Linear(in_features=288, out_features=256, bias=True)
(5215): ReLU()
(5216): Linear(in_features=256, out_features=288, bias=True)
(5217): ReLU()
(5218): Linear(in_features=288, out_features=256, bias=True)
(5219): ReLU()
(5220): Linear(in_features=256, out_features=319, bias=True)
(5221): ReLU()
(5222): Linear(in_features=319, out_features=288, bias=True)
(5223): ReLU()
(5224): Linear(in_features=288, out_features=318, bias=True)
(5225): ReLU()
(5226): Linear(in_features=318, out_features=288, bias=True)
(5227): ReLU()
(5228): Linear(in_features=288, out_features=316, bias=True)
(5229): ReLU()
(5230): Linear(in_features=316, out_features=288, bias=True)
(5231): ReLU()
(5232): Linear(in_features=288, out_features=312, bias=True)
(5233): ReLU()
(5234): Linear(in_features=312, out_features=288, bias=True)
(5235): ReLU()
(5236): Linear(in_features=288, out_features=304, bias=True)
(5237): ReLU()
(5238): Linear(in_features=304, out_features=288, bias=True)
(5239): ReLU()
(5240): Linear(in_features=288, out_features=256, bias=True)
(5241): ReLU()
(5242): Linear(in_features=256, out_features=336, bias=True)
(5243): ReLU()
(5244): Linear(in_features=336, out_features=328, bias=True)
(5245): ReLU()
(5246): Linear(in_features=328, out_features=340, bias=True)
(5247): ReLU()
(5248): Linear(in_features=340, out_features=332, bias=True)
(5249): ReLU()
(5250): Linear(in_features=332, out_features=375, bias=True)
(5251): ReLU()
(5252): Linear(in_features=375, out_features=399, bias=True)
(5253): ReLU()
(5254): Linear(in_features=399, out_features=410, bias=True)
(5255): ReLU()
(5256): Linear(in_features=410, out_features=402, bias=True)
(5257): ReLU()
(5258): Linear(in_features=402, out_features=412, bias=True)
(5259): ReLU()
(5260): Linear(in_features=412, out_features=404, bias=True)
(5261): ReLU()
(5262): Linear(in_features=404, out_features=412, bias=True)
(5263): ReLU()
(5264): Linear(in_features=412, out_features=404, bias=True)
(5265): ReLU()
(5266): Linear(in_features=404, out_features=408, bias=True)
(5267): ReLU()
(5268): Linear(in_features=408, out_features=400, bias=True)
(5269): ReLU()
(5270): Linear(in_features=400, out_features=352, bias=True)
(5271): ReLU()
(5272): Linear(in_features=352, out_features=288, bias=True)
(5273): ReLU()
(5274): Linear(in_features=288, out_features=288, bias=True)
(5275): ReLU()
(5276): Linear(in_features=288, out_features=256, bias=True)
(5277): ReLU()
(5278): Linear(in_features=256, out_features=319, bias=True)
(5279): ReLU()
(5280): Linear(in_features=319, out_features=288, bias=True)
(5281): ReLU()
(5282): Linear(in_features=288, out_features=318, bias=True)
(5283): ReLU()
(5284): Linear(in_features=318, out_features=288, bias=True)
(5285): ReLU()
(5286): Linear(in_features=288, out_features=316, bias=True)
(5287): ReLU()
(5288): Linear(in_features=316, out_features=288, bias=True)
(5289): ReLU()
(5290): Linear(in_features=288, out_features=312, bias=True)
(5291): ReLU()
(5292): Linear(in_features=312, out_features=288, bias=True)
(5293): ReLU()
(5294): Linear(in_features=288, out_features=304, bias=True)
(5295): ReLU()
(5296): Linear(in_features=304, out_features=288, bias=True)
(5297): ReLU()
(5298): Linear(in_features=288, out_features=256, bias=True)
(5299): ReLU()
(5300): Linear(in_features=256, out_features=288, bias=True)
(5301): ReLU()
(5302): Linear(in_features=288, out_features=256, bias=True)
(5303): ReLU()
(5304): Linear(in_features=256, out_features=319, bias=True)
(5305): ReLU()
(5306): Linear(in_features=319, out_features=288, bias=True)
(5307): ReLU()
(5308): Linear(in_features=288, out_features=318, bias=True)
(5309): ReLU()
(5310): Linear(in_features=318, out_features=288, bias=True)
(5311): ReLU()
(5312): Linear(in_features=288, out_features=316, bias=True)
(5313): ReLU()
(5314): Linear(in_features=316, out_features=288, bias=True)
(5315): ReLU()
(5316): Linear(in_features=288, out_features=312, bias=True)
(5317): ReLU()
(5318): Linear(in_features=312, out_features=288, bias=True)
(5319): ReLU()
(5320): Linear(in_features=288, out_features=304, bias=True)
(5321): ReLU()
(5322): Linear(in_features=304, out_features=288, bias=True)
(5323): ReLU()
(5324): Linear(in_features=288, out_features=256, bias=True)
(5325): ReLU()
(5326): Linear(in_features=256, out_features=256, bias=True)
(5327): ReLU()
(5328): Linear(in_features=256, out_features=256, bias=True)
(5329): ReLU()
(5330): Linear(in_features=256, out_features=256, bias=True)
(5331): ReLU()
(5332): Linear(in_features=256, out_features=256, bias=True)
(5333): ReLU()
(5334): Linear(in_features=256, out_features=287, bias=True)
(5335): ReLU()
(5336): Linear(in_features=287, out_features=319, bias=True)
(5337): ReLU()
(5338): Linear(in_features=319, out_features=318, bias=True)
(5339): ReLU()
(5340): Linear(in_features=318, out_features=318, bias=True)
(5341): ReLU()
(5342): Linear(in_features=318, out_features=316, bias=True)
(5343): ReLU()
(5344): Linear(in_features=316, out_features=316, bias=True)
(5345): ReLU()
(5346): Linear(in_features=316, out_features=312, bias=True)
(5347): ReLU()
(5348): Linear(in_features=312, out_features=312, bias=True)
(5349): ReLU()
(5350): Linear(in_features=312, out_features=304, bias=True)
(5351): ReLU()
(5352): Linear(in_features=304, out_features=304, bias=True)
(5353): ReLU()
(5354): Linear(in_features=304, out_features=256, bias=True)
(5355): ReLU()
(5356): Linear(in_features=256, out_features=192, bias=True)
(5357): ReLU()
(5358): Linear(in_features=192, out_features=192, bias=True)
(5359): ReLU()
(5360): Linear(in_features=192, out_features=160, bias=True)
(5361): ReLU()
(5362): Linear(in_features=160, out_features=223, bias=True)
(5363): ReLU()
(5364): Linear(in_features=223, out_features=192, bias=True)
(5365): ReLU()
(5366): Linear(in_features=192, out_features=222, bias=True)
(5367): ReLU()
(5368): Linear(in_features=222, out_features=192, bias=True)
(5369): ReLU()
(5370): Linear(in_features=192, out_features=220, bias=True)
(5371): ReLU()
(5372): Linear(in_features=220, out_features=192, bias=True)
(5373): ReLU()
(5374): Linear(in_features=192, out_features=216, bias=True)
(5375): ReLU()
(5376): Linear(in_features=216, out_features=192, bias=True)
(5377): ReLU()
(5378): Linear(in_features=192, out_features=208, bias=True)
(5379): ReLU()
(5380): Linear(in_features=208, out_features=192, bias=True)
(5381): ReLU()
(5382): Linear(in_features=192, out_features=160, bias=True)
(5383): ReLU()
(5384): Linear(in_features=160, out_features=192, bias=True)
(5385): ReLU()
(5386): Linear(in_features=192, out_features=160, bias=True)
(5387): ReLU()
(5388): Linear(in_features=160, out_features=223, bias=True)
(5389): ReLU()
(5390): Linear(in_features=223, out_features=192, bias=True)
(5391): ReLU()
(5392): Linear(in_features=192, out_features=222, bias=True)
(5393): ReLU()
(5394): Linear(in_features=222, out_features=192, bias=True)
(5395): ReLU()
(5396): Linear(in_features=192, out_features=220, bias=True)
(5397): ReLU()
(5398): Linear(in_features=220, out_features=192, bias=True)
(5399): ReLU()
(5400): Linear(in_features=192, out_features=216, bias=True)
(5401): ReLU()
(5402): Linear(in_features=216, out_features=192, bias=True)
(5403): ReLU()
(5404): Linear(in_features=192, out_features=208, bias=True)
(5405): ReLU()
(5406): Linear(in_features=208, out_features=192, bias=True)
(5407): ReLU()
(5408): Linear(in_features=192, out_features=160, bias=True)
(5409): ReLU()
(5410): Linear(in_features=160, out_features=320, bias=True)
(5411): ReLU()
(5412): Linear(in_features=320, out_features=320, bias=True)
(5413): ReLU()
(5414): Linear(in_features=320, out_features=382, bias=True)
(5415): ReLU()
(5416): Linear(in_features=382, out_features=446, bias=True)
(5417): ReLU()
(5418): Linear(in_features=446, out_features=444, bias=True)
(5419): ReLU()
(5420): Linear(in_features=444, out_features=444, bias=True)
(5421): ReLU()
(5422): Linear(in_features=444, out_features=440, bias=True)
(5423): ReLU()
(5424): Linear(in_features=440, out_features=440, bias=True)
(5425): ReLU()
(5426): Linear(in_features=440, out_features=432, bias=True)
(5427): ReLU()
(5428): Linear(in_features=432, out_features=432, bias=True)
(5429): ReLU()
(5430): Linear(in_features=432, out_features=416, bias=True)
(5431): ReLU()
(5432): Linear(in_features=416, out_features=416, bias=True)
(5433): ReLU()
(5434): Linear(in_features=416, out_features=320, bias=True)
(5435): ReLU()
(5436): Linear(in_features=320, out_features=192, bias=True)
(5437): ReLU()
(5438): Linear(in_features=192, out_features=48, bias=True)
(5439): ReLU()
(5440): Linear(in_features=48, out_features=1, bias=True)
(5441): ReLU()
)/tmp/ipykernel_1060068/378402040.py:2, with the implication
the authors enjoy using notebooks as much as I do. :)
Assets if you just want to follow the cropped out piece analyzed here.
I’ve provided a safetensors file consisting of just the adder gadget.
Usage:
from functools import cache
from torch import Tensor
from torch.nn import Linear, Module, Parameter, ReLU, Sequential
from typing import Iterable
@cache
def get_adder_network_state() -> dict[str, Tensor]:
from safetensors.torch import load_file
state = load_file('files/adder.safetensors')
return state
@cache
def get_adder_network() -> Sequential:
def layers() -> Iterable[Module]:
state = get_adder_network_state()
for i in range(0, 28, 2):
weights = state[f'{i}.weight']
bias = state[f'{i}.bias']
lin = Linear(
in_features=weights.shape[1],
out_features=weights.shape[0],
)
lin.weight = Parameter(weights, requires_grad=False)
lin.bias = Parameter(bias, requires_grad=False)
yield lin
yield ReLU()
model = Sequential(*layers()).eval()
model.to("cuda")
return modeldisplay(get_adder_network())Sequential(
(0): Linear(in_features=64, out_features=96, bias=True)
(1): ReLU()
(2): Linear(in_features=96, out_features=64, bias=True)
(3): ReLU()
(4): Linear(in_features=64, out_features=127, bias=True)
(5): ReLU()
(6): Linear(in_features=127, out_features=96, bias=True)
(7): ReLU()
(8): Linear(in_features=96, out_features=126, bias=True)
(9): ReLU()
(10): Linear(in_features=126, out_features=96, bias=True)
(11): ReLU()
(12): Linear(in_features=96, out_features=124, bias=True)
(13): ReLU()
(14): Linear(in_features=124, out_features=96, bias=True)
(15): ReLU()
(16): Linear(in_features=96, out_features=120, bias=True)
(17): ReLU()
(18): Linear(in_features=120, out_features=96, bias=True)
(19): ReLU()
(20): Linear(in_features=96, out_features=112, bias=True)
(21): ReLU()
(22): Linear(in_features=112, out_features=96, bias=True)
(23): ReLU()
(24): Linear(in_features=96, out_features=64, bias=True)
(25): ReLU()
(26): Linear(in_features=64, out_features=32, bias=True)
(27): ReLU()
)Detail: Cutting out the adder gadget from the broader network
The adder gadget is much smaller than the full model. We can isolate it by cutting out pieces from a stack of layers which contain it, manually analyzed:
from dataclasses import dataclass
@dataclass(slots=True, frozen=True)
class Cut:
layer: int
start_y: int
start_x: int
width: int
height: int
cuts = [
Cut(38, 160, 160, 64, 96),
Cut(40, 192, 160, 96, 64),
Cut(42, 160, 192, 64, 127),
Cut(44, 223, 160, 127, 96),
Cut(46, 192, 223, 96, 126),
Cut(48, 222, 192, 126, 96),
Cut(50, 192, 222, 96, 124),
Cut(52, 220, 192, 124, 96),
Cut(54, 192, 220, 96, 120),
Cut(56, 216, 192, 120, 96),
Cut(58, 192, 216, 96, 112),
Cut(60, 208, 192, 112, 96),
Cut(62, 192, 208, 96, 64),
Cut(64, 160, 192, 64, 32),
]
@cache
def get_adder_network_by_cropping() -> Sequential:
def layers() -> Iterable[Module]:
state = get_network_state()
for cut in cuts:
source_layer = state[f"{cut.layer}.weight"]
cropped_layer = source_layer[cut.start_y:cut.start_y+cut.height, cut.start_x:cut.start_x+cut.width]
source_bias = state[f"{cut.layer}.bias"]
cropped_bias = source_bias[cut.start_y:cut.start_y+cut.height]
lin = Linear(
in_features=cut.width,
out_features=cut.height,
)
lin.weight = Parameter(cropped_layer, requires_grad=False)
lin.bias = Parameter(cropped_bias, requires_grad=False)
yield lin
yield ReLU()
model = Sequential(*layers()).eval()
model.to("cuda")
return modeldisplay(get_adder_network_by_cropping())Sequential(
(0): Linear(in_features=64, out_features=96, bias=True)
(1): ReLU()
(2): Linear(in_features=96, out_features=64, bias=True)
(3): ReLU()
(4): Linear(in_features=64, out_features=127, bias=True)
(5): ReLU()
(6): Linear(in_features=127, out_features=96, bias=True)
(7): ReLU()
(8): Linear(in_features=96, out_features=126, bias=True)
(9): ReLU()
(10): Linear(in_features=126, out_features=96, bias=True)
(11): ReLU()
(12): Linear(in_features=96, out_features=124, bias=True)
(13): ReLU()
(14): Linear(in_features=124, out_features=96, bias=True)
(15): ReLU()
(16): Linear(in_features=96, out_features=120, bias=True)
(17): ReLU()
(18): Linear(in_features=120, out_features=96, bias=True)
(19): ReLU()
(20): Linear(in_features=96, out_features=112, bias=True)
(21): ReLU()
(22): Linear(in_features=112, out_features=96, bias=True)
(23): ReLU()
(24): Linear(in_features=96, out_features=64, bias=True)
(25): ReLU()
(26): Linear(in_features=64, out_features=32, bias=True)
(27): ReLU()
)from safetensors.torch import save_file
save_file(get_adder_network_by_cropping().state_dict(), 'files/adder.safetensors')Light spoilers for the Jane Street Hugging Face puzzle follow.
Logical gates
Throughout the model, weights and biases are consistently integer
values, and the activations are all ReLU. The model is also
much less intimidating to analyze than the file size would suggest.
While the safetensors file is over 1GB, this is mostly identity matrices
and empty space. Blindly letting sympy simplify equations
as a set of linear equations, the entire system can be characterized in
about 6.5MB of plain text.
Immediately, we see some pretty unusual combinations, and recurring
patterns. Most values are 0, 1, and occasionally 2, and when restricted
to this range, form logical operators. When taking two values, in the
form \(ax_1 + bx_2 + c\), operating on
single bits, after ReLU:
| a | b | c | Output |
|---|---|---|---|
| 1 | 1 | -1 | \(x_1\) AND \(x_2\) |
| -1 | -1 | 1 | \(x_1\) NOR \(x_2\) |
| -1 | 0 | 1 | NOT \(x_1\) |
| 1 | 1 | 0 | HIGH(\(x_1\), \(x_2\)) (overhigh vector intermediate, with “bit” values up to 2, used for computing XOR) |
XOR is composed over two layers: \(x_1\,\text{XOR}\, x_2 = \text{HIGH}(x_1, x_2) + -2 \times (x_1\,\text{AND}\,x_2)\). XOR is often computed alongside AND, but there aren’t any optimizations applied, so you’ll often see a layer computing AND twice, one of which is passed through via identity matrix, and one of which is used for computing XOR.
The happy path
Given well-formed input, the function of the model is pretty simple: Two 32-bit numbers, represented as bit-vectors, are added to form one 32-bit vector output, with overflow wrapping:
import numpy as np
from numpy.typing import NDArray
from torch import Tensor
from dataclasses import dataclass
def pack_le(value: int | np.signedinteger | np.unsignedinteger, width: int = 32) -> list[int]:
return [int((value >> i) & 1) for i in range(width)]
def unpack_le(bits: Iterable[int]) -> int:
return sum([int(bit) << i for i, bit in enumerate(bits)])
def tensor_input(a, b) -> NDArray[np.int16]:
'''
Returns a 64-"bit" tensor input for the adder network
'''
x = np.zeros((64), dtype=np.int16)
x[:32] = pack_le(a, 32)
x[32:] = pack_le(b, 32)
return x
@dataclass(slots=True, frozen=True)
class Case:
a: int
b: int
expected: int
def check_add(cases: Iterable[Case], network: Sequential, verbose=False) -> None:
batch = Tensor(np.array([
tensor_input(c.a, c.b)
for c in cases])
)
batch = batch.to("cuda")
results = network.forward(batch).cpu().numpy().astype(np.int16)
for case, result in zip(cases, results):
expected = case.expected
actual = unpack_le(result)
if verbose:
print(f"Case {case.a} + {case.b} = {expected}{"" if expected == actual else f" -!> {actual}"}")
assert expected == actual, f"Expected {expected} but got {actual} for inputs {case.a} and {case.b}"
adder = get_adder_network()
cases = []
for a_f in range(5):
for b_f in range(5):
a = (5 ** a_f + 4) % 43
v = (7 ** b_f + 11) % 43
cases += [Case(a, v, a + v)]
check_add(cases, adder, verbose=True)Case 5 + 12 = 17
Case 5 + 18 = 23
Case 5 + 17 = 22
Case 5 + 10 = 15
Case 5 + 4 = 9
Case 9 + 12 = 21
Case 9 + 18 = 27
Case 9 + 17 = 26
Case 9 + 10 = 19
Case 9 + 4 = 13
Case 29 + 12 = 41
Case 29 + 18 = 47
Case 29 + 17 = 46
Case 29 + 10 = 39
Case 29 + 4 = 33
Case 0 + 12 = 12
Case 0 + 18 = 18
Case 0 + 17 = 17
Case 0 + 10 = 10
Case 0 + 4 = 4
Case 27 + 12 = 39
Case 27 + 18 = 45
Case 27 + 17 = 44
Case 27 + 10 = 37
Case 27 + 4 = 31So far, so good. But there are specific circumstances in the full puzzle where the input “bits” supplied to the adder aren’t always 0 or 1.
What happens when the preconditions of the adder circuit aren’t met?
The big one
There are two situations where a hidden state used as input can have a value other than 0 or 1. Let \(A = \left<a_0, \dots, a_{31}\right>\) be the first operand, and \(B = \left<b_0, \dots, b_{31}\right>\) be the second operand.
In one scenario, a state can be overfull up to 129. In another situation, a state can be overfull up to 186.
In both cases, they affect the “1-bit” of a byte in the first operand: \(a_0\), \(a_8\), \(a_{16}\), or \(a_{24}\).
Intuitively, we expect the effect to be much the same regardless of which of the four bytes this lands on, through symmetry: The circuit goes through 1-carry, 2-carry, 4-carry, 8-carry, and 16-carry layers. Only in the last two stages would we get the possibility of one byte affecting the next.
Focusing on \(a_0\):
from collections import Counter
import numpy as np
from IPython.display import display
from torch import Tensor
import torch
# Axes: y = first byte of operand A, x = first byte of operand B.
a_vals = np.arange(256, dtype=np.int16)
b_vals = np.arange(256, dtype=np.int16)
overfull_vals = np.arange(187, dtype=np.int16)
# Build base input batch for all (a_byte, b_byte) pairs in row-major order.
aa, bb = np.meshgrid(a_vals, b_vals, indexing='ij')
aa_flat = aa.reshape(-1)
bb_flat = bb.reshape(-1)
base_batch = np.zeros((aa_flat.shape[0], 64), dtype=np.int16)
for bit in range(8):
base_batch[:, bit] = (aa_flat >> bit) & 1
base_batch[:, 32 + bit] = (bb_flat >> bit) & 1
# Overfull is injected at the 1-bit of the first operand's first byte.
a_without_bit1 = aa_flat & ~1
errors_by_frame = []
with torch.no_grad():
for overfull in overfull_vals:
batch = base_batch.copy()
batch[:, 0] = int(overfull)
actual = adder(Tensor(batch).to("cuda")).cpu().numpy().astype(np.int16)
actual_vals = np.array([unpack_le(row) for row in actual], dtype=np.int64)
expected_vals = a_without_bit1.astype(np.int64) + (int(overfull)) + bb_flat.astype(np.int64)
error_grid = (actual_vals - expected_vals).reshape(256, 256)
errors_by_frame.append(error_grid)
errors_by_frame = np.stack(errors_by_frame, axis=0)
nonzero_mask = errors_by_frame != 0
err_vmin = int(errors_by_frame[nonzero_mask].min())
err_vmax = int(errors_by_frame[nonzero_mask].max())
error_counts = Counter(errors_by_frame.flatten())
error_counts_bytes = [Counter(((errors_by_frame >> off) & 0xff).flatten()) for off in range(0, 32, 8)]
display(err_vmin, err_vmax)131229
131673error_counts_bytes = [Counter(((errors_by_frame >> off) & 0xff).flatten()) for off in range(0, 32, 8)]
display(len(error_counts_bytes[0]))
display([str(v)[:100] for v in error_counts_bytes])256
['Counter({np.int64(0): 190976, np.int64(213): 65536, np.int64(212): 65536, np.int64(211): 65536, np.i',
'Counter({np.int64(1): 11498240, np.int64(2): 543744, np.int64(0): 213248})',
'Counter({np.int64(2): 12091392, np.int64(0): 163840})',
'Counter({np.int64(0): 12255232})']We observe structure in the error, which naturally coalesces around bit patterns. The least significant byte of error has all 256 possible values represented; the second least significant byte of error magnitude only has 3.
Can we visualize both absolute error, along with an alternative bit mapping to focus on bit patterns?
from itertools import count
from colorsys import hsv_to_rgb
from IPython.display import HTML
from matplotlib import colormaps
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
from matplotlib.animation import FuncAnimation
from matplotlib.ticker import MultipleLocator
viridis = colormaps.get_cmap('viridis')
white = (1.0, 1.0, 1.0)
def bits_set(value: int) -> int:
for n in count():
if value == 0:
return n
value &= value - 1
raise
def bits_avgpos(value: int) -> float:
pos_total = 0
pos_count = 0
for n in count():
if value == 0:
break
if value & 1:
pos_total += n
pos_count += 1
value >>= 1
return pos_total / pos_count if pos_count > 0 else 0
colors_lin = []
colors_bit = []
ind_to_result = []
most_bits_set = 0
bits_range = 10
for i, err in enumerate(sorted(error_counts.keys())):
ind_to_result.append(err)
if err == 0:
colors_lin.append(white)
colors_bit.append(white)
continue
colors_lin.append(viridis(0.2 + 0.8 * (err - err_vmin) / (err_vmax - err_vmin)))
err_bits_set, err_bits_avgpos = bits_set(err), bits_avgpos(err)
most_bits_set = max(most_bits_set, err_bits_set)
colors_bit.append(hsv_to_rgb(err_bits_avgpos / 16, 1.0, 0.15 + 0.7 * (err_bits_set / bits_range)))
assert bits_range == most_bits_set, f"{bits_range=} != {most_bits_set=}"
norm = mcolors.BoundaryNorm(boundaries=ind_to_result, ncolors=len(error_counts))
cmap_lin = mcolors.ListedColormap(colors_lin, name="error_scale_integer")
cmap_bit = mcolors.ListedColormap(colors_bit, name="error_scale_bitpattern")
fig, (ax_lin, ax_bit) = plt.subplots(1, 2, figsize=(12, 5), sharex=True, sharey=True)
image_lin = ax_lin.imshow(
errors_by_frame[0],
origin='lower',
interpolation='nearest',
cmap=cmap_lin,
norm=norm,
extent=(0, 255, 0, 255),
)
image_bit = ax_bit.imshow(
errors_by_frame[0],
origin='lower',
interpolation='nearest',
cmap=cmap_bit,
norm=norm,
extent=(0, 255, 0, 255),
)
colorbar = fig.colorbar(image_lin, ax=ax_lin)
colorbar.set_label('output error = actual - expected')
for ax in (ax_lin, ax_bit):
ax.xaxis.set_major_locator(MultipleLocator(64))
ax.yaxis.set_major_locator(MultipleLocator(64))
ax.set_xlabel('first byte of operand B')
ax.set_ylabel('first byte of operand A')
title = fig.suptitle('overfull bit a_0 value = 0')
def animate(frame_idx: int):
image_lin.set_data(errors_by_frame[frame_idx])
image_bit.set_data(errors_by_frame[frame_idx])
title.set_text(f'overfull bit a_0 value = {frame_idx}')
return [image_lin, title]
anim = FuncAnimation(fig, animate, frames=errors_by_frame.shape[0], interval=140, blit=False)
anim.save('files/tensors-error-animation-full.webm', writer='ffmpeg')
anim = FuncAnimation(fig, animate, frames=10, interval=140, blit=False)
plt.close(fig)
display(HTML(anim.to_jshtml()))If javascript is enabled, the above interactive chart lets you finely control display of the error patterns for the first ten values of \(a_0\).
Video of all 187 frames:
The 1-bit propagates through the XOR carry circuit of the adder. For different states of the first operand’s 1-bit:
- At 0 or 1, the bit is not overfull, and there is no error.
- At 2, if the second operand is even - that is, the second operand’s 1-bit is unset - it recovers for a correct result, exactly as if both 1-bits had been 1.
When the error condition is met, the lowest two bits of each byte seem to have no impact on the final value. Intuitively, this makes sense: Forcibly triggering carry behaviour in the first two positions should zero out the bits at these positions.
Three’s a crowd
Visually, an overfull \(a_0\) above 3 does not appear to change the result. The linear error shown seems to smoothly decrease, as the expected A smoothly increases.
Verify this:
error_condition_vals = None
with torch.no_grad():
for overfull in range(3, 187):
batch = base_batch.copy()
batch[:, 0] = int(overfull)
actual = adder(Tensor(batch).to("cuda")).cpu().numpy().astype(np.int16)
actual_vals = np.array([unpack_le(row) for row in actual], dtype=np.int64)
if error_condition_vals is not None:
assert np.array_equal(actual_vals, error_condition_vals), f"Actual values changed from previous frame for {overfull=}"
error_condition_vals = actual_vals
assert error_condition_vals is not None
# Also check the odd B, overfull 2 case:
with torch.no_grad():
overfull = 2
batch = base_batch.copy()
batch[:, 0] = int(overfull)
actual = adder(Tensor(batch).to("cuda")).cpu().numpy().astype(np.int16)
actual_vals = np.array([unpack_le(row) for row in actual], dtype=np.int64)
# Only compare cases where B is odd; even B recovers to the correct result
mask = (bb_flat & 1) == 1
assert np.array_equal(actual_vals[mask], error_condition_vals[mask]), "Actual values did not match expected values for overfull 2"So the error condition is \(a_0 > 2 \lor \left(a_0 = 2 \land b_0 \& 1 = 1\right)\). When this condition is triggered, the actual value of \(a_0\) doesn’t matter for the final result.
Create a new animation, still examining the lowest byte of \(A\) and \(B\), but with \(a_0\) fixed at \(3\), and time increasing a square search pattern through higher bytes of \(A\) and \(B\):
from itertools import chain
# Search out in a bouncing quarter spiral pattern
def square_search_pattern(depth_limit: int) -> Iterable[tuple[int, int]]:
for depth in range(depth_limit):
if depth & 1 == 0:
for offset in range(depth + 1):
yield (depth, offset)
for offset in range(1, depth + 1):
yield (depth - offset, depth)
else:
for offset in range(depth + 1):
yield (offset, depth)
for offset in range(1, depth + 1):
yield (depth, depth - offset)
def skipper(seq: Iterable[tuple[int, int]], scale=2, threshold=16, stride=1) -> Iterable[tuple[int, int]]:
# Move on to checking for lower frequency patterns once we've checked for higher frequency patterns
for idx, elm in enumerate(seq):
if idx > threshold:
threshold = int(threshold * scale)
stride = int(stride * scale)
if idx % stride != 0:
continue
yield elm
overfull = 3
result_by_frame = []
labels_by_frame = []
with torch.no_grad():
for a, b in skipper(square_search_pattern(128)):
batch = base_batch.copy()
batch[:, 0] = int(overfull)
for bit in range(8):
batch[:, 8 + bit] = (a >> bit) & 1
batch[:, 32 + 8 + bit] = (b >> bit) & 1
actual = adder(Tensor(batch).to("cuda")).cpu().numpy().astype(np.int32)
actual_vals = np.array([unpack_le(row) for row in actual], dtype=np.int64)
result_vals = actual_vals.reshape(256, 256)
result_by_frame.append(result_vals)
labels_by_frame.append(f'A >> 8 = 0x{a:02x}, B >> 8 = 0x{b:02x}')
result_count = Counter(chain.from_iterable(frame.flatten() for frame in result_by_frame))
ind_to_result = []
colors = []
for result in sorted(result_count.keys()):
ind_to_result.append(result)
colors.append((((result >> 16) & 0xff) / 255, ((result >> 8) & 0xff) / 255, (result & 0xff) / 255))
norm = mcolors.BoundaryNorm(boundaries=ind_to_result, ncolors=len(ind_to_result))
cmap = mcolors.ListedColormap(colors, name="result_scale_integer")
fig, ax = plt.subplots(1, 1, figsize=(6, 5))
ax.imshow(
result_by_frame[0],
origin='lower',
interpolation='nearest',
cmap=cmap,
norm=norm,
extent=(0, 255, 0, 255),
)
colorbar = fig.colorbar(ax.images[0], ax=ax)
colorbar.set_label('output value')
ax.xaxis.set_major_locator(MultipleLocator(64))
ax.yaxis.set_major_locator(MultipleLocator(64))
ax.set_xlabel('first byte of operand B')
ax.set_ylabel('first byte of operand A')
fig.suptitle('overfull bit a_0 value = 3, searching over increasing A >> 8 and B >> 8')
title = ax.set_title(f'A >> 8 = 0x{0:02x}, B >> 8 = 0x{0:02x}', fontfamily='monospace')
def animate(frame_idx: int):
ax.images[0].set_data(result_by_frame[frame_idx])
title.set_text(labels_by_frame[frame_idx])
return [ax.images[0], title]
# Render full video
anim = FuncAnimation(fig, animate, frames=len(result_by_frame), interval=300, blit=False)
plt.close(fig)
anim.save('files/tensors-result-animation.webm', writer='ffmpeg')The broader pattern seems consistent with our expectation: The lowest two bits of each byte have no impact on the result. The grid maintains the same pattern. We can cut analysis to a space \(\frac{1}{16}^{th}\) of its original size, and begin building a function to characterize the overfull adder behaviour.
Give me space
A few more limitations from the larger network make analysis much easier. First, for an overfull condition to be met, all other bits of the overfull byte must be set, practical analysis can be kept to the case where \(\forall a \in \left< a_7,\dots,a_1 \right> : a = 1\). Secondly, in one of the scenarios triggering the overfull byte condition, all other bytes of A are typically 0. (In the other scenario, this is not the case, but that scenario requires more unusual input.)
First, let’s pre-cache the results from the actual network.
# Axes: y = third byte of operand B, x = fourth byte of operand B.
b2_vals = np.arange(0, 256, 4, dtype=np.int64)
b3_vals = np.arange(0, 256, 4, dtype=np.int64)
# Build base input batch for all (b0_byte, b1_byte) pairs in row-major order.
b2, b3 = np.meshgrid(b2_vals, b3_vals, indexing='ij')
b2_flat = b2.reshape(-1)
b3_flat = b3.reshape(-1)
baseu_batch = np.zeros((b2_flat.shape[0], 64), dtype=np.int16)
# A: All set, lowest overfull
baseu_batch[:, 0] = int(overfull)
baseu_batch[:, 1:8] = 1
# B: From b2, b3
for bit in range(8):
baseu_batch[:, 32 + bit + 16] = (b2_flat >> bit) & 1
baseu_batch[:, 32 + bit + 24] = (b3_flat >> bit) & 1
@dataclass(slots=True, frozen=True)
class ActualFrame:
b0: int
b1: int
result_vals: NDArray[np.int64]
actuals = []
with torch.no_grad():
for b0, b1 in skipper(square_search_pattern(256), threshold=64):
batch = baseu_batch.copy()
# B: From b0, b1
for bit in range(0, 8):
batch[:, 32 + bit + 0] = (b0 >> bit) & 1
batch[:, 32 + bit + 8] = (b1 >> bit) & 1
actual = adder(Tensor(batch).to("cuda")).cpu().numpy().astype(np.int16)
actual_vals = np.array([unpack_le(row) for row in actual], dtype=np.int64)
actuals.append(ActualFrame(b0, b1, actual_vals))Then, repeatedly rebuild and plot the error of a characterization function, focusing on the all-bits-set \(A\), as in order for \(a_0\) to be overfull, \(a_1, \dots, a_{31}\) must all be set in the full puzzle.
def adjustment(v: int) -> int:
adj = 0
match v & 0x300:
case 0x000:
adj = 0x000
case 0x100:
adj = -0x100
case 0x200:
adj = -0x200
case 0x300:
adj = 0x100
case _:
raise
match v & 0x30000:
case 0x00000:
adj += 0x00000
case 0x10000:
adj += -0x10000
case 0x20000:
adj += -0x20000
case 0x30000:
adj += 0x10000
case _:
raise
match v & 0x3ff00:
case 0x3ff00:
adj += 0x20000
case 0x1ff00:
adj += -0x20000
return adj
def overfull_add(a, b, verbose=False) -> int:
'''
Assumes:
- Overfull bit is a_0
- a_7...a_1 are all set
- a_31...a_8 are all unset
'''
result = (0x20300 + adjustment(b) + (b & ~0x303)) & 0xFFFF_FFFF
if verbose:
# Look up actual
lb3, lb2, lb1, lb0 = (b >> 24) & 0xFF, (b >> 16) & 0xFF, (b >> 8) & 0xFF, b & 0xFF
frame = next(f for f in actuals if f.b0 == lb0 and f.b1 == lb1)
actual = frame.result_vals.reshape(64, 64)[lb2 // 4, lb3 // 4]
print(f"{a=}, {b=}, {adjustment(b)=}, {hex(adjustment(b))=}, {result=} {actual=}")
return resultPlotting the error of our function against the actual overfull adder result:
from typing import Callable
def check_characterization(characterize_candidate: Callable[[int, int], int]) -> None:
errors = []
labels = []
last_error_frame = None
for actual in actuals:
b_base = (actual.b1 << 8) + actual.b0
a = 0xff
attempt_vals = np.array([characterize_candidate(a, int(b_base + (b3 << 24) + (b2 << 16))) for b2, b3 in zip(b2_flat, b3_flat)], dtype=np.int64)
char_error = actual.result_vals - attempt_vals
errors.append(char_error)
if np.any(char_error != 0):
if last_error_frame is None:
print(f"First error frame: {actual.b0=}, {actual.b1=}")
last_error_frame = (actual.b0, actual.b1)
labels.append(f'B >> 0 = 0x{actual.b0:02x}, B >> 8 = 0x{actual.b1:02x}')
if last_error_frame is not None:
print(f"Last error frame: actual.b0={last_error_frame[0]}, actual.b1={last_error_frame[1]}")
else:
print("No errors found in any frame.")
char_err_count = Counter(chain.from_iterable(error.flatten() for error in errors))
vals = list(char_err_count.keys())
if len(vals) == 1:
vals.append(vals[0] + 1)
char_vmax = max(vals)
char_vmin = min(vals)
ind_to_result = []
colors = []
for result in sorted(vals):
ind_to_result.append(result)
if result == 0:
colors.append(white)
continue
colors.append(viridis(0.2 + 0.8 * (result - char_vmin) / (char_vmax - char_vmin)))
norm = mcolors.BoundaryNorm(boundaries=ind_to_result, ncolors=len(ind_to_result))
cmap = mcolors.ListedColormap(colors, name="char_error_scale_integer")
fig, ax = plt.subplots(1, 1, figsize=(6, 5))
ax.imshow(
errors[0].reshape(64, 64),
origin='lower',
interpolation='nearest',
cmap=cmap,
norm=norm,
extent=(0, 255, 0, 255),
)
colorbar = fig.colorbar(ax.images[0], ax=ax)
colorbar.set_label('characterizing function error = actual - attempt')
ax.xaxis.set_major_locator(MultipleLocator(64))
ax.yaxis.set_major_locator(MultipleLocator(64))
ax.set_xlabel('fourth byte of operand B')
ax.set_ylabel('third byte of operand B')
fig.suptitle(f'attempted characterization: {characterize_candidate.__name__}')
title = ax.set_title(f'B >> 16 = 0x{0:02x}, B >> 24 = 0x{0:02x}', fontfamily='monospace')
def animate(frame_idx: int):
ax.images[0].set_data(errors[frame_idx].reshape(64, 64))
title.set_text(labels[frame_idx])
return [ax.images[0], title]
anim = FuncAnimation(fig, animate, frames=len(errors), interval=300, blit=False)
plt.close(fig)
anim.save('files/tensors-characterize-animation.webm', writer='ffmpeg')
check_characterization(overfull_add)No errors found in any frame.This suffices for practical brute force analysis in the broader puzzle. Generalizing to \(a_8\), \(a_{16}\), and \(a_{24}\), or when other bits in A’s overfull byte are not fully set, is an exercise left to the reader. Or, to my future self in an unlikely potential update.