Neural Computers
Mingchen Zhuge ⋅ Changsheng Zhao ⋅ Haozhe Liu ⋅ Zijian Zhou ⋅ Shuming Liu ⋅ Wenyi Wang ⋅ Ernie Chang ⋅ Gael Le Lan ⋅ Junjie Fei ⋅ Wenxuan Zhang ⋅ Yasheng SUN ⋅ Yunyang Xiong ⋅ Zechun Liu ⋅ zhipeng cai ⋅ Yining Yang ⋅ Yuandong Tian ⋅ Yangyang Shi ⋅ Vikas Chandra ⋅ Jürgen Schmidhuber
Abstract
We train neural networks (NNs) to learn the dynamics of traditional computers (TCs). The NN's inputs include screenshots, instructions (natural language or program commands), and user actions such as keyboard typing and mouse movements/clicks. Without relying on internal program state, we investigate whether a unified neural system can learn, simulate, and operate a computer interface directly from raw I/O data. We present initial steps towards a completely neural computer (CNC): in our first experiments, our NNs learn to emulate TCs in both CLI and GUI settings. We derive practical design considerations for building more and more general CNCs.
Chat is not available.
Successful Page Load