Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Introduction
Background
Related Work
Summary
Hypotheses and Goals
Quantitative Results
Conclusions
References
All Topics
Computer Science

BF++: a language for general-purpose neural program synthesis

Profile image of Vadim LiventsevVadim Liventsev

2021, ArXiv

Abstract

Most state of the art decision systems based on Reinforcement Learning (RL) are data-driven black-box neural models, where it is often difficult to incorporate expert knowledge into the models or let experts review and validate the learned decision mechanisms. Knowledge-insertion and model review are important requirements in many applications involving human health and safety. One way to bridge the gap between data and knowledge driven systems is program synthesis: replacing a neural network that outputs decisions with one that generates decision-making code in some programming language. We propose a new programming language, BF++, designed specifically for neural program synthesis in a Partially Observable Markov Decision Process (POMDP) setting and generate programs for a number of standard OpenAI Gym benchmarks. Source code is available at https://github.com/vadim0x60/cibi

Key takeaways
sparkles

AI

  1. BF++ enables program synthesis for decision-making in Partially Observable Markov Decision Processes (POMDPs).
  2. Incorporating expert knowledge improves the performance of programs generated through BF++.
  3. The language features 22 commands, including non-blocking action operators and a virtual comma for infinite loops.
  4. Experimental results demonstrate functional solutions comparable to deep learning methods in OpenAI Gym benchmarks.
  5. Future work includes developing translation mechanisms to other programming languages and applying BF++ in healthcare.

Related papers

Learning Libraries of Subroutines for Neurally-Guided Bayesian Program Induction
Lucas Morales

2018

Successful approaches to program induction require a hand-engineered domain-specific language (DSL), constraining the space of allowed programs and imparting prior knowledge of the domain. We contribute a program induction algorithm that learns a DSL while jointly training a neural network to efficiently search for programs in the learned DSL. We use our model to synthesize functions on lists, edit text, and solve symbolic regression problems, showing how the model learns a domain-specific library of program components for expressing solutions to problems in the domain.

View PDFchevron_right
Towards Neural-Guided Program Synthesis for Linear Temporal Logic Specifications
Alberto Javier Jaya Camacho

ArXiv, 2019

Synthesizing a program that realizes a logical specification is a classical problem in computer science. We examine a particular type of program synthesis, where the objective is to synthesize a strategy that reacts to a potentially adversarial environment while ensuring that all executions satisfy a Linear Temporal Logic (LTL) specification. Unfortunately, exact methods to solve so-called LTL synthesis via logical inference do not scale. In this work, we cast LTL synthesis as an optimization problem. We employ a neural network to learn a Q-function that is then used to guide search, and to construct programs that are subsequently verified for correctness. Our method is unique in combining search with deep learning to realize LTL synthesis. In our experiments the learned Q-function provides effective guidance for synthesis problems with relatively small specifications.

View PDFchevron_right
A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis
Johnny Ye

arXiv (Cornell University), 2021

We convert the DeepMind Mathematics Dataset into a reinforcement learning environment by interpreting it as a program synthesis problem. Each action taken in the environment adds an operator or an input into a discrete compute graph. Graphs which compute correct answers yield positive reward, enabling the optimization of a policy to construct compute graphs conditioned on problem statements. Baseline models are trained using Double DQN on various subsets of problem types, demonstrating the capability to learn to correctly construct graphs despite the challenges of combinatorial explosion and noisy rewards.

View PDFchevron_right
Neuro-Symbolic Program Synthesis
rishabh singh

2020

Recent years have seen the proposal of a number of neural architectures for the problem of Program Induction. Given a set of input-output examples, these architectures are able to learn mappings that generalize to new test inputs. While achieving impressive results, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network). In this paper, we propose a novel technique, Neuro-Symbolic Program Synthesis, to overcome the above-mentioned problems. Once trained, our approach can automatically construct computer programs in a domain-specific language that are consistent with a set of input-output examples provided at test time. Our method is based on two novel neural modules. The first module, called the cross correlation I/O network, given a set of in...

View PDFchevron_right
Write, Execute, Assess: Program Synthesis with a REPL
Felix A Sosa

2019

We present a neural program synthesis approach integrating components which write, execute, and assess code to navigate the search space of possible programs. We equip the search process with an interpreter or a read-eval-print-loop (REPL), which immediately executes partially written programs, exposing their semantics. The REPL addresses a basic challenge of program synthesis: tiny changes in syntax can lead to huge changes in semantics. We train a pair of models, a policy that proposes the new piece of code to write, and a value function that assesses the prospects of the code written so-far. At test time we can combine these models with a Sequential Monte Carlo algorithm. We apply our approach to two domains: synthesizing text editing programs and inferring 2D and 3D graphics programs.

View PDFchevron_right
Parametrized Hierarchical Procedures for Neural Programming
Ion Gabriel Stoica

2018

Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism. Despite the potential to increase the interpretability and the compositionality of the behavior of artificial agents, it remains difficult to learn from demonstrations neural networks that represent computer programs. The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of specific structures of data, and the extremely limited observability. To address these challenges, we propose to model programs as Parametrized Hierarchical Procedures (PHPs). A PHP is a sequence of conditional operations, using a program counter along with the observation to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller. We develop an algorithm for training PHPs from a set of supervisor demonstrations, only some of w...

View PDFchevron_right
AutoPandas: neural-backed generators for program synthesis
Rohan Bavishi

Proceedings of the ACM on Programming Languages

Developers nowadays have to contend with a growing number of APIs. While in the long-term they are very useful to developers, many modern APIs have an incredibly steep learning curve, due to their hundreds of functions handling many arguments, obscure documentation, and frequently changing semantics. For APIs that perform data transformations, novices can often provide an I/O example demonstrating the desired transformation, but may be stuck on how to translate it to the API. A programming-by-example synthesis engine that takes such I/O examples and directly produces programs in the target API could help such novices. Such an engine presents unique challenges due to the breadth of real-world APIs, and the often-complex constraints over function arguments. We present a generator-based synthesis approach to contend with these problems. This approach uses a program candidate generator, which encodes basic constraints on the space of programs. We introduce neural-backed operators which ...

View PDFchevron_right
Programmable Reinforcement Learning Agents
Stuart J. Russell

We present an expressive agent design language for reinforcement learning that allows the user to constrain the policies considered by the learning process.The language includes standard features such as parameterized subroutines, temporary interrupts, aborts, and memory variables, but also allows for unspecified choices in the agent program. For learning that which isn't specified, we present provably convergent learning algorithms. We demonstrate by example that agent programs written in the language are concise as well as modular. This facilitates state abstraction and the transferability of learned skills.

View PDFchevron_right
HOUDINI: Lifelong Learning as Program Synthesis
Akash Srivastava

2018

We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning. Reusing high-level concepts across domains and learning complex procedures are key challenges in lifelong learning. We show that a program synthesis approach that combines gradient descent with combinatorial search over programs can be a more effective response to these challenges than purely neural methods. Our framework, called HOUDINI, represents neural networks as strongly typed, differentiable functional programs that use symbolic higher-order combinators to compose a library of neural functions. Our learning algorithm consists of: (1) a symbolic program synthesizer that performs a type-directed search over parameterized programs, and decides on the library functions to reuse, and the architectures to combine them, while learning a sequence of tasks; and (2) a neural module that trains these programs using stochastic gradient descent. We evaluate HOU...

View PDFchevron_right
State Abstraction for Programmable Reinforcement Learning Agents
Stuart J. Russell

Safe state abstraction in reinforcement learning allows an agent to ignore aspects of its current state that are irrelevant to its current decision, and therefore speeds up dynamic programming and learning. This paper explores safe state abstraction in hierarchical reinforcement learning, where learned behaviors must conform to a given partial, hierarchical program. Unlike previous approaches to this problem, our methods yield significant state abstraction while maintaining hierarchical optimality, i.e., optimality among all policies consistent with the partial program. We show how to achieve this for a partial programming language that is essentially Lisp augmented with nondeterministic constructs. We demonstrate our methods on two variants of Dietterich's taxi domain, showing how state abstraction and hierarchical optimality result in faster learning of better policies and enable the transfer of learned skills from one problem to another. t ¥ and ¥ are thus simple deterministic functions, determined from the program structure.

View PDFchevron_right

References (44)

  1. David Silver, Julian Schrittwieser, Karen Simonyan, Ioan- nis Antonoglou, Aja Huang, Arthur Guez, Thomas Hu- bert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Na- ture, 550(7676):354-359, oct 2017. ISSN 14764687. doi: 10.1038/nature24270.
  2. OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław De ¸biak, Christy Denni- son, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning. 2019. URL https://arxiv.org/abs/1912.06680.
  3. J. Andrew Bagnell. Reinforcement Learning in Robotics: A Survey. In Springer Tracts in Advanced Robotics, volume 97, pages 9-67. 2014. doi: 10.1007/ 978-3-319-03194-1_2.
  4. Zhenpeng Zhou, Xiaocheng Li, and Richard N Zare. Op- timizing Chemical Reactions with Deep Reinforcement Learning. ACS Central Science, 3(12):1337-1344, 2017. ISSN 23747951. doi: 10.1021/acscentsci.7b00492. URL https://pubs.acs.org/sharingguidelines.
  5. I Arel, C Liu, T Urbanik, and A G Kohls. Reinforcement learning-based multi-agent system for network traffic sig- nal control. 2009. doi: 10.1049/iet-its.2009.0070. URL www.ietdl.org.
  6. Chao Yu, Jiming Liu, and Shamim Nemati. Reinforce- ment learning in healthcare: a survey. arXiv preprint arXiv:1908.08796, 2019.
  7. K J Åström. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174-205, 1965. ISSN 0022-247X. doi: https://doi.org/10.1016/0022-247X(65) 90154-X. URL http://www.sciencedirect.com/science/ article/pii/0022247X6590154X.
  8. Jr Kramer, J David R. Partially Observable Markov Pro- cesses., 1964.
  9. Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction, Second edition in progress, volume 3. 2017. doi: 10.1016/S1364-6613(99)01331-5.
  10. Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
  11. Seyed Sajad Mousavi, Michael Schukat, and Enda Howley. Deep Reinforcement Learning: An Overview. In Lecture Notes in Networks and Systems, volume 16, pages 426- 440. 2018. doi: 10.1007/978-3-319-56991-8_32. URL https://arxiv.org/abs/.
  12. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 5045-5054, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/verma18a. html.
  13. Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program synthesis. Foundations and Trends in Program- ming Languages, 4(1-2):1-119, 2017. ISSN 23251131. doi: 10.1561/2500000010. URL www.nowpublishers. com;.
  14. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Reposito- ries, MSR, pages 476-486. ACM, 2018. doi: https: //doi.org/10.1145/3196398.3196408.
  15. Xiaojun Xu, Chang Liu, and Dawn Song. SQLNet: Gener- ating Structured Queries From Natural Language Without Reinforcement Learning. Technical report.
  16. Neel Kant. Recent Advances in Neural Program Synthesis. 2018. URL http://arxiv.org/abs/1802.02353.
  17. Oleksandr Polozov and Sumit Gulwani. Flashmeta: A framework for inductive program synthesis. In Proceed- ings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, page 107-126, New York, NY, USA, 2015. Association for Computing Ma- chinery. ISBN 9781450336895. doi: 10.1145/2814270. 2814310. URL https://doi.org/10.1145/2814270.2814310.
  18. Ashwin K Vijayakumar, Dhruv Batra, Abhishek Mohta, Prateek Jain, Oleksandr Polozov, and Sumit Gulwani. Neural-guided deductive search for real-time program syn- thesis from examples. In 6th International Conference on Learning Representations, ICLR 2018 -Conference Track Proceedings, 2018. URL https://microsoft.github.io/prose/ impact/.
  19. Richard Shin, Neel Kant, Kavi Gupta, Christopher Ben- der, Brandon Trabucco, Rishabh Singh, and Dawn Song. Synthetic datasets for neural program synthesis. Technical report, 2019.
  20. Wojciech Zaremba, Ilya Sutskever, and Google Brain. Re- inforcement Learning Neural Turing Machines-revised. Technical report. URL https://github.com/ilyasu123/rlntm.
  21. Jason Weston, Sumit Chopra, and Antoine Bordes. Mem- ory networks. In 3rd International Conference on Learning Representations, ICLR 2015 -Conference Track Proceed- ings, oct 2015. URL http://arxiv.org/abs/1410.3916.
  22. Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines. In 4th International Con- ference on Learning Representations, ICLR 2016 -Confer- ence Track Proceedings, 2016.
  23. Alexander L Gaunt, Marc Brockschmidt, Rishabh Singh, Nate Kushman, Pushmeet Kohli, Jonathan Taylor, and Daniel Tarlow. TerpreT: A Probabilistic Programming Language for Program Induction. Technical report.
  24. Daniel A. Abolafia, Mohammad Norouzi, Jonathan Shen, Rui Zhao, and Quoc V. Le. Neural Program Synthesis with Priority Queue Training. 2018. URL http://arxiv.org/abs/ 1801.03526.
  25. U. Muller. Brainfuck -an eight-instruction turing- complete programming language. Available at the inter- net address http://en.wikipedia.org/wiki/Brainfuck, 1993. URL http://en.wikipedia.org/wiki/Brainfuck.
  26. Julie D Allen, Deborah Anderson, Joe Becker, Richard Cook, Mark Davis, Peter Edberg, Michael Everson, As- mus Freytag, Laurentiu Iancu, Richard Ishida, et al. The unicode standard. Mountain view, CA, 2012.
  27. A M Turing. On computable numbers, with an application to the entscheidungsproblem. a correction. Proceedings of the London Mathematical Society, s2-43(1):544-546, 1938. ISSN 1460244X. doi: 10.1112/plms/s2-43.6.544.
  28. Ahmed Touati, Adrien Ali Taiga, and Marc G Bellemare. Zooming for efficient model-free reinforcement learning in metric spaces. arXiv preprint arXiv:2003.04069, 2020.
  29. Mats Linander. control flow in brainfuck | matslina. Avail- able at the internet address http://calmerthanyouare. org/2016/01/14/control-flow-in-brainfuck.html, 2016. URL http://calmerthanyouare.org/2016/01/ 14/control-flow-in-brainfuck.html.
  30. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Re- search, 47:253-279, jun 2013.
  31. Daiki Kimura. Daqn: Deep auto-encoder and q-network. arXiv preprint arXiv:1806.00630, 2018.
  32. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba Openai. OpenAI Gym. Technical report.
  33. A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cyber- netics, SMC-13(5):834-846, Sep. 1983. ISSN 2168-2909. doi: 10.1109/TSMC.1983.6313077.
  34. Andrew William Moore. Efficient memory-based learning for robot control. Technical report, 1990.
  35. Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research, 13:227-303, 2000.
  36. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. 1999.
  37. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Ma- chine learning, 8(3-4):229-256, 1992.
  38. Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
  39. vadim0x60/evestop: Early stopping with exponential vari- ance elmination. https://github.com/vadim0x60/evestop. (Accessed on 01/20/2021).
  40. Wikipedia contributors. Bytecode -Wikipedia, the free encyclopedia, 2020. URL https://en.wikipedia.org/w/index. php?title=Bytecode&oldid=995026385. [Online; accessed 21-January-2021].
  41. Riccardo Poli, William B Langdon, Nicholas F McPhee, and John R Koza. A field guide to genetic programming. Lulu. com, 2008.
  42. P. A. Keane and E. J. Topol. With an eye to AI and au- tonomous diagnosis. NPJ Digit Med, 1:40, 2018. [PubMed Central:PMC6550235] [DOI:10.1038/s41746-018-0048- y] [PubMed:29618526].
  43. Kyle Richardson, Sina Zarrieß, and Jonas Kuhn. The code2text challenge: Text generation in source code li- braries. CoRR, abs/1708.00098, 2017. URL http://arxiv. org/abs/1708.00098.
  44. A. LeClair, S. Jiang, and C. McMillan. A neural model for generating natural language summaries of program subrou- tines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 795-806, 2019. doi: 10.1109/ICSE.2019.00087.

Related papers

BF++: a language for general-purpose program synthesis
Vadim Liventsev

2021

Most state of the art decision systems based on Reinforcement Learning (RL) are data-driven black-box neural models, where it is often difficult to incorporate expert knowledge into the models or let experts review and validate the learned decision mechanisms. Knowledge-insertion and model review are important requirements in many applications involving human health and safety. One way to bridge the gap between data and knowledge driven systems is program synthesis: replacing a neural network that outputs decisions with a symbolic program generated by a neural network or by means of genetic programming. We propose a new programming language, BF++, designed specifically for automatic programming of agents in a Partially Observable Markov Decision Process (POMDP) setting and apply neural program synthesis to solve standard OpenAI Gym benchmarks. Source code is available at https://github.com/vadim0x60/cibi

View PDFchevron_right
Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
Rishabh Singh

ArXiv, 2018

Program synthesis is the task of automatically generating a program consistent with a specification. Recent years have seen proposal of a number of neural approaches for program synthesis, many of which adopt a sequence generation paradigm similar to neural machine translation, in which sequence-to-sequence models are trained to maximize the likelihood of known reference programs. While achieving impressive results, this strategy has two key limitations. First, it ignores Program Aliasing: the fact that many different programs may satisfy a given specification (especially with incomplete specifications such as a few input-output examples). By maximizing the likelihood of only a single reference program, it penalizes many semantically correct programs, which can adversely affect the synthesizer performance. Second, this strategy overlooks the fact that programs have a strict syntax that can be efficiently checked. To address the first limitation, we perform reinforcement learning on ...

View PDFchevron_right
Neural Programming Language
Hava Siegelmann

Analog recurrent neural networks have attracted much attention lately as powerful tools of automatic learning. We formally define a high level language, called NEural Langage, which is rich enough to express any computer algorithm or rulebased system. We show how to compile a NEL program to a network which computes exactly as the original program and requires the same computation time. We suggest this language along with its compiler as the ultimate bridge from symbolic to analog computation, and propose its outcome as an initial network for learning. *

View PDFchevron_right
An inductive synthesis framework for verifiable reinforcement learning
Zikang Xiong

Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2019

Despite the tremendous advances that have been made in the last decade on developing useful machine-learning applications, their wider adoption has been hindered by the lack of strong assurance guarantees that can be made about their behavior. In this paper, we consider how formal verification techniques developed for traditional software systems can be repurposed for verification of reinforcement learningenabled ones, a particularly important class of machine learning systems. Rather than enforcing safety by examining and altering the structure of a complex neural network implementation, our technique uses blackbox methods to synthesizes deterministic programs, simpler, more interpretable, approximations of the network that can nonetheless guarantee desired safety properties are preserved, even when the network is deployed in unanticipated or previously unobserved environments. Our methodology frames the problem of neural network verification in terms of a counterexample and syntax-guided inductive synthesis procedure over these programs. The synthesis procedure searches for both a deterministic program and an inductive invariant over an infinite state transition system that represents a specification of an application's control logic. Additional specifications defining environment-based constraints can also be provided to further refine the search space. Synthesized programs deployed in conjunction with a neural network implementation dynamically enforce safety conditions by monitoring and preventing potentially unsafe actions proposed by neural

View PDFchevron_right
1 Neural-Backed Generators for Program Synthesis
Rohan Bavishi

2019

Developers nowadays have to contend with a growing number of APIs. While in the long-term they are very useful to developers, many modern APIs, with their hundreds of functions handling many arguments, obscure documentation, and frequently changing semantics, have an incredibly steep learning curve. For APIs that perform data transformations, novices can often provide an I/O example demonstrating the desired transformation, but are stuck on how to translate it to the API. Our goal is to build a programming-by-example synthesis engine that takes such I/O examples and directly produces programs in the target API. This presents unique challenges due to the breadth of real-world APIs, and the often-complex constraints over function arguments. We present a generator-based synthesis approach to contend with these problems. This approach uses a program candidate generator, which encodes basic constraints on the space of programs. We introduce neural-backed operators which can be seamlessly integrated into the program generator. To improve the efficiency of the search, we simply use these operators at non-deterministic decision points, instead of relying on domain-specific heuristics. We implement this technique for the Python pandas library in AutoPandas. AutoPandas supports 119 pandas dataframe transformation functions. We evaluate AutoPandas on 26 real-world benchmarks and find it solves 65% of them.

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved