[Computer-go] CNN with 54% prediction on KGS 6d+ data

Tue Dec 29 01:24:44 PST 2015

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I am fighting with the problem most seem to have with the strong move
predictions at the moment, MCTS is not increasing the players a lot :)

I wonder, if somebody measured the performance of the pure CNN54
against pachi 10k (or 100k), to get a comparison with the darkforest CNN.

It is not too much work, but you probably did it already.

Thanks,

Detlef

Am 21.12.2015 um 12:42 schrieb Hiroshi Yamashita:
> Hi Detlef,
> 
> Thank you for publishing your data and latest oakform code! It was
> very helpful for me.
> 
> I tried your 54% data with Aya.
> 
> Aya with Detlef54% vs Aya with Detlef44%, 10000 playout/move Aya
> with Detlef54%'s winrate is 0.569 (124wins / 218games).
> 
> CGOS BayseElo rating Aya with Detlef44%  (aya786n_Detlef_10k) 3040 
> Aya with Detlef54%  (Aya786m_Det54_10k ) 3036 
> http://www.yss-aya.com/cgos/19x19/bayes.html
> 
> Detlef54% is a bit stronger in selfplay, but they are similar on
> CGOS. Maybe Detlef54%'s prediction is strong, and Aya's playout
> strength is not enough.
> 
> Speed for a position on GTS 450. Detlef54%   21ms Detlef44%   17ms
> 
> Cumulative accuracy from 1000 pro games.
> 
> move rank  Aya    Detlef54%  Mixture 1      40.8      47.6
> 48.0 2      53.5      62.4     62.7 3      60.2      70.7     71.0 
> 4      64.8      75.8     76.1 5      68.1      79.5     79.9 6
> 71.0      82.3     82.6 7      73.2      84.5     84.8 8      75.2
> 86.3     86.6 9      76.9      87.8     88.1 10      78.3      89.0
> 89.3 11      79.6      90.2     90.6 12      80.8      91.2
> 91.4 13      81.9      92.0     92.2 14      82.9      92.7
> 92.9 15      83.8      93.3     93.5 16      84.6      93.9
> 94.1 17      85.4      94.3     94.5 18      86.1      94.8
> 95.0 19      86.8      95.2     95.4 20      87.4      95.5
> 95.7
> 
> Mixture is pretty same as Detlef54%. I changed learning method from
> MM to LFR. Aya's own accuracy is from LFR rank, not MM gamma. So
> comparison is difficult.
> 
> Cumulative accuracy Detlef44% 
> http://computer-go.org/pipermail/computer-go/2015-October/008031.html
>
>  Regards, Hiroshi Yamashita
> 
> 
> ----- Original Message ----- From: "Detlef Schmicker"
> <ds2 at physik.de> To: <computer-go at computer-go.org> Sent: Wednesday,
> December 09, 2015 12:13 AM Subject: [Computer-go] CNN with 54%
> prediction on KGS 6d+ data
> 
> 
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> 
>> Hi,
>> 
>> as somebody ask I will offer my actual CNN for testing.
>> 
>> It has 54% prediction on KGS 6d+ data (which I thought would be
>> state of the art when I started training, but it is not
>> anymore:).
>> 
>> it has: 1 2 3
>>> 4 libs playing color
>> 1 2 3
>>> 4 libs opponent color
>> Empty points last move second last move third last move forth
>> last move
>> 
>> input layers, and it is fully convolutional, so with just editing
>> the golast19.prototxt file you can use it for 13x13 as well, as I
>> did on last sunday. It was used in November tournament as well.
>> 
>> You can find it http://physik.de/CNNlast.tar.gz
>> 
>> 
>> 
>> If you try here some points I like to get discussion:
>> 
>> - - it seems to me, that the playouts get much more important
>> with such a strong move prediction. Often the move prediction
>> seems better the playouts (I use 8000 at the moment against pachi
>> 32000 with about 70% winrate on 19x19, but with an extremely
>> focused progressive widening (a=400, a=20 was usual).
>> 
>> - - live and death becomes worse. My interpretation is, that the
>> strong CNN does not play moves, which obviously do not help to
>> get a group life, but would help the playouts to recognize the
>> group is dead. (http://physik.de/example.sgf top black group was
>> with weaker move prediction read very dead, with good CNN it was
>> 30% alive or so :(
>> 
>> 
>> OK, hope you try it, as you know our engine oakfoam is open
>> source :) We just merged all the CNN stuff into the main branch! 
>> https://bitbucket.org/francoisvn/oakfoam/wiki/Home 
>> http://oakfoam.com
>> 
>> 
>> Do the very best with the CNN
>> 
>> Detlef
> 
> _______________________________________________ Computer-go mailing
> list Computer-go at computer-go.org 
> http://computer-go.org/mailman/listinfo/computer-go
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJWglFcAAoJEInWdHg+Znf42q4P/AnMdgqhps4RSJG3NoLiwEUq
QmT4mQd58WbuxnXRO4xiyIKGTQq13+FOpqVu7RgFPXxQaKS+8Hi1qpGVjg8aE8Zh
bnHb3D+p30hv9lCT8e4xNQ2B1JZsgOlM3MsbeFdQB+vxca3kUcnCf9oMvHo0W8TL
Tl8q7sDbI1bW0Z16lCKfDHdwyiBhDjETPP9j1wlfZgXyqD5JMCqwxcUkOrxlsh96
ZhX5bCnbN5CAPKedTxQVz8GcPwo74TIXCb+UmzklVOBC3pGJ3WrtWmNyHiPwiJ75
qYEzolICvW+wE+RbCfeiGaaL1CY9B5N2GKSCPQdzd0UYUwBrXsUMG3mTJ5Kwg26G
+nIg/KBnWCbgjN9WpHVkAsRewkAGezom7OSp2y1KyrIORcQc3FW8LLWxhzXjBNuj
3VFx9iT6zSiO+5kjUINdejVh4cT19Oao+ZVWZuPyBf9y/dcUn01NE2tCr+xIcqFq
7p+R0y9VA15f/KDufgJHUeeaPCdox6YU4VlxlbQoKdQt/X6iQftxPEDcBe39kxRy
R7SGJ6sMYxJBbsnNFfb547jBpeJRunHaX2dswjZtKleEUSTGXKgs77/ju3kbgC8n
WZuvvs6QcPqPsAyFFBsYbpOelP2NT7jpMX7IdkiGLb5wpblUhtnkV+nTy2ccTG1e
veWukuo97oFFhUBSHQtV
=+mE4
-----END PGP SIGNATURE-----